Should the smaller holdings in a county not just represent the local power over a chunk of land within the County? So either the Baron/Mayor/Bishop has the regional control? So rather than considering it destroying a town, could it not be perceived as a regional change of local autonomy, especially in a post conquest context. I come in and take the power away from the Bishop, and give it to a Baron (who happens to be my man of course).
Obviously this could be offset by modifiers to dissuade this being over used (obviously the argument against the destruction of the holding outright would be the financial penalty). Penalties could include severe tax and unrest modifiers while the new administration establishes itself (could potentially take decades), prestige hits (medieval world was ultra conservative and opposed to almost any change), national malus towards other vassals (if you deprive a bishop of his holding and establish a baron instead, then every other bishop might suddenly become very unhappy with you - same with the Pope in this case.) Finally, the counties would still be subject to standard balancing - one of each, so this could only affect counties with a third baronial level holding.
This way we can still have the desired change without resulting in the notion that an entire city and everyone in it just disappeared one day. They'd all still be there, just under a new administration.
Obviously this could be offset by modifiers to dissuade this being over used (obviously the argument against the destruction of the holding outright would be the financial penalty). Penalties could include severe tax and unrest modifiers while the new administration establishes itself (could potentially take decades), prestige hits (medieval world was ultra conservative and opposed to almost any change), national malus towards other vassals (if you deprive a bishop of his holding and establish a baron instead, then every other bishop might suddenly become very unhappy with you - same with the Pope in this case.) Finally, the counties would still be subject to standard balancing - one of each, so this could only affect counties with a third baronial level holding.
This way we can still have the desired change without resulting in the notion that an entire city and everyone in it just disappeared one day. They'd all still be there, just under a new administration.