Indeed, turning off building should be removed, and deleting them added. Currently, there is zero reason to pay any upkeep at all for military buildings, as you can turn them off whenever you are not training or upgrading units, which is boring and stupid gameplay-wise (as you could as well remove upkeep cost, but then, turning them off would serve zero purpose).
On the other hand, removing a building would allow to save upkeep, and factor some kind of decision making, unlike turning them off which is a no brainer. There needs to be some cost for destroying a building, and some time to, to prevent the player to respec a city on the fly.