I remember invading Canada in Hoi 3 while USA was still not in the allies but providing Military access to the Canadians. They couldn't use US territory to go around and enter territory colored in glorious Japanese occupation colors, thus threatening my rear, but they could probably have done so if I left empty unoccupied pockets of Canada on the US border. This also made it very difficult for me to envelop them: Squeezing them against the border caused them to move across it and redeploy home, showing up at the front a few weeks later.
Apparently it will work the same in this game, which makes access very useful on the defense, or as you say to support an ally that you don't share a border with. If France has military access through Belgium it can preposition forces there, ready to rush into defensive lines along Dutch rivers when Germany invades and Netherlands joins the Allies. If you wait until having Belgium in your faction it may be too late to save the useful Dutch divisions, even if Belgium joins immediately and your troops can start moving at once they may be too far away unless you have access.