Caveat: the naval AI and mechanics are a bit abstruse, not fully documented. So some of what I say may be wrong, being based on observation, so if other people's observations are different, give them at least equal weight.
1. What determines whether an AI fleet will engage or withdraw from a battle?
A few points. First and foremost is its mission. If it is on convoy raiding then it is trying to avoid stand up fights with combat fleets, while a fleet on patrol is actively seeking them out. Second is org, when a cap ship is deorged it may lead the entire fleet to withdraw; and when all cap ships are deorged it forces the fleet to withdraw, or so it appears from my experience. But this is not universal, as all-screen fleets may press battle. It seems at least in part to be determined by the initial composition of the fleet as compared to its current state at any given moment.
The third point is not really a factor in why fleets withdraw, but rather a warning: it will at times seem inconsistent. A fleet that should flee may seem to choose not to, while another that shouldn't flee seems to throw in the sponge almost at once. This is often because the game is not showing you all the relevant information. That fleet that should flee but doesn't may be actively trying to but failing to get away. And the one that shouldn't flee but did may in fact have never gotten the chance because the enemy scarpered first. Stuff like that.
2. At what point can superior speed allow a fleet to catch up to the other without it fleeing?
There is no guarantee that speed will enable a fleet to pin down an enemy that wants to flee. Random chance is always a factor, plus other circumstances frequently intervene to help a fleeing enemy escape, e.g. nightfall, inclement weather. That said, any advantage in speed seems to help, but I have not really noticed that exceptionally large speed differentials cause any scaling of that advantage. I suspect there may be one or more calculations in which speed should be factoring but for some reason is not, whether by design, bug or just fluke.
3. Is there any merit to using multiple fleets grouped together (like how Royal Navy starts)? Will they act differently at all?
No. If they are to be on the same mission in the same place, put them in same task force. To do otherwise both costs you extra leaders and risks them being separated and defeated in detail during that time when they are unable to help each other. That said, there definitely can be advantage to having two or more fleets with
different missions operating in the same AO, for example a CTF operating in a large AO in which one or more small submarine wolfpacks are also hunting. That combination would allow one to basically interdict an entire region as far as convoys are concerned while also having a powerful combat group on hand ready to squish any enemy fleet that may come to try clear out the subs.
