I find imagining it as a (abstracted) version of supply lines works. The fort controls whatever river, valley, road or dirt track the local supply lines go through, until you capture the fort it blocks any attempts for your armies to go through. Attempting to transport supplies for your army cross country is too difficult.
Even if this is the case, and there is supply problems, I fail to see how secondary army, that cannot access plains next to a castle province because of imaginary walls, and all imaginary stuff makes any sense. Even in imaginary land.