I honestly wish they implemented some sort of a silly board mechanic (tiles with units or something) just to make Big Blobs of Death less relevant.
Splitting army in games like Civ makes sense because a small amount of units can hold up a large amount of units. Splitting army in games like Gothic Armada makes sense because of how various AOE abilities, map features and rock-paper-scissors situations play out (there are tons of mechanics, I won't even attempt to list them all).
In Stellaris space is uniform and featureless, except for defense stations which add one point of interest per system, and still optimally defeated by a blob with a larger number written on it. This is just a dead-end approach, from the "fun" point of view, even if it makes the most sense from the "realism" point of view ("realism in this space fantasy setting", you know what I mean).
I'm honestly would be totally fine with an immersion-breaking "well, you can't put two units in one hex" kind of solution if it created reasons to actually think about wars in a strategic way. Not sure if everyone would agree with that, but hopefully people won't argue that doomstacks as the only viable option is a wrong thing.
For a "you cant put two units in one hex" sort of solution that isn't nearly as restrictive or unrealistic as just straight up forbidding multiple fleets to be in one spot, they could add an "inefficient command" penalty where each fleet added past the first applies a multiplicative penalty to the fire rate for all fleets, with one fleet being one command limit worth of ships in a particular battle.
If that penalty were say, 25%, then the first fleet would fire at full efficiency, for a total of 1 fleet worth of fire power. Then the second fleet comes in, and lowers the fire rate of both fleets by 25%, resulting in a total of 1.5 fleets of fire power. Then the third comes in, lowering the fire rate of all fleets to 3/4 of the fire rate the two fleets had, giving around 56% fire rate for each fleet and a total of about 1.7 fleets worth.
Have it be lore justified as the admiralty not being able to handle the excessively "friendly rich" environment, and thus needing to spend extra time coordinating to avoid friendly fire, excessive overkill, and straight up collisions between friendlies. Basically the opposite of how the outnumbered bonus makes it easier for a small force to operate in a "target rich" environment.
This would let outnumbered fleets actually delay the enemy for longer, and give the command limit some more meaning other than requiring a few extra admirals or making the micro of moving a large number of ships around slightly less convenient, without making the outnumbered force able to inflict way more casualties than its supposed to. It could even result in tactics like keeping a reserve fleet ready to join in the battle once the first group of ships gets battered actually useful, since adding a 3rd fleet to a fight at the start wouldn't add as much as maintaining two fleets for longer.
Having disengaged ships still count as being in battle for the inefficient command calculations could also make no retreat and unyielding actually useful and flavorful instead of just stupid since having a higher percentage of ships actually firing would give a substantial bonus to damage output compared to having some disengaged ships lowering the output of the whole fleet. Probably have them count as a fraction of their original value though, otherwise having a single ship refuse to run away for too long would suck.
I'd probably also make command limit somewhat smaller, especially in the late game, as well as boost the overall durability of everything by a fair bit to make it so that time taken in space battles vs time taken in planetary invasions/bombardment would be more balanced and mid battle reinforcement would be more feasible. The formula could also be adjusted to make it more linear, like making it so each extra fleet always adds another .25 fleets of damage, but I feel the geometric method would work better.
EDIT: Should probably mention that the penalty would be scaled if a force was only partially over the command limit, so if an empire had two fleets with a total of 150% of their command limit in a system, their ships would fire at 87.5% normal speed, and 250% of the command limit would mean a fire rate of ~65%.
