I think the building destruction rate is great. Wars are way less devastating than they should be (even with HiP and building destruction). This at least gives you some insentive to defend your territory against sieges or think about if it's worth it to diminish your own lands in favor of winning a war more quickly.
I honestly agree, I mean the Vikings were a serious threat economically and nearly bankrupted many smaller kingdoms. The Op called them annoying which is exactly what they should be (as long as they aren't too frequent like when TOG was first released), they should take a significant investment to resist and you should feel the burn from when you fail to do so.
The problem is that paradox models economic development with high upfront costs, and fairly short build times. Instead, there should be extremely long build times, but low up-front costs. Wars crippled the treasury, but raids crippled the economy, and right now, there's little distinction.
(In addition to drastically increasing build time and drastically lowering cost for castle towns, e.g., another way to model this would be to increase the size of prosperity modifiers, but also increase the difficult of moving between them. The latter idea, if incorporated into start-up events, would also help model historically OP areas do to population booms, better.)