Forts are the first to go when a province is sieged, once they die they are burned up entirely and need to be completely rebuilt again.
Nomads have the option to Pillage any settlement that they control, this removes a couple of buildings and nets a small amount of money and (I think) technology points. Keep doing this often enough until there are no buildings left and the settlement itself disappears.
Repeat as necessary for every settlement in the province, every province in the Duchy, every Duchy in the Kingdom and every Kingdom in the EMpire and you're good to go - 100% pasture from sea to shining sea !