Sure, local base manpower giving a small bonus to fort defense makes sense.
But what I feel the game lacks is a connection between the garrison and the manpower system.
It doesn't need to be a 1-1 connection. But at least something.
At the very least when one is out of manpower, the garrison cannot replenish.
That does not mean we take the garrison from the mapower pool, just that when it's at 0 (or close to 0) the garrison stops replenishing or slows down a lot.
Ideally it would actually take from the manpower pool, but yeah that would be a lot more difficult to balance.