Pretty historically accurate, though, I'd think. Hardly anything that's represented by a holding in-game would have been completely destroyed by military action, except for possibly a pagan religious holding.
More to do with gameplay balance and the difficulties of having a plausible AI imo. As for historical plausibility - cities can be burned, bishoprics can be plundered and looted, castles can in some cases be broken down to use for building materials or just demolished to demilitarise and centralise a formerly autonomous region. These would be rare occurrences and damaged buildings could be rebuilt or the city relocated, if there was a demand to do so, but sometimes areas were left to decay.
The problem is, in my view, that a "plausible randomness" would be very hard to determine. Could London have been completely abandoned for another English city? Probably not unless under improbably extreme conditions. With the much less populated outer regions of the map, it's a very different story. And then again, should it only happen by military force, by random events with a certain MTTH (50, 100, 150 years?), or by player action in the interface having met certain conditions? All of those leave a lot to be desired and would end up making the natural story and flow of the game to be arbitrarily broken, especially if the AI is affected. There is also the problem that characters having only one barony-level title could suddenly have no title at all, from one day to the next.
As you hint at, holdings would represent more than a single entity in one county as there is a limit of 6 plus the capital. Still, disasters (which ingame would result in the removal of a holding) affecting this scale of land was not unknown of. To mention: War, revolts, famine and climate change, diseases, and emigration (where possible) could radically change the demographic structure of a region.