Personally I do not see what the big fuss about doomstacks is. Doomstacks make perfect sense. It is space, space is HUGE, you can have a humongous grand armada, mutiply it 10 times and you still occupy a very minimal amount of space.
Doomstacks are (and should be) highlly effective. The best, easiest and most reliable way to defeat an opponent is to show up with a colossal fleet and vaporize everything into space dust. The only downside of doomstacks is inefficiency. Splitting fleets allow to cover more ground... I mean space. Multiple systems can be attached at once. Actually it is a highly effective strategy against massive empires. Attack them from all sides, then start conquering planets everywhere, while the doomstack is playing catch with one of your smaller fleets.
The problem is that the AI is dumb as a rock and cannot make strategic decisions. If you show up with your grand armada, that the AI cannot defeat with another doomstack, it should start attacking your systems with small fleets, conquering stations and landing troops on your planets. Then you will be sure as hell forced to split up the big ball o' doom. If the AI would decide that it has a shot to defeat your grand fleet, it should consolidate its ships into another doomstack to let the epic battle play out. A really smart AI would evaluate the status of its economy and if it is strong, it should be willing to go into epic pyrrhic battles just to screw up your economy.
The only penalty that doomstacks would supposed to get is speed. I think large fleets, regardless of fleet capacity should receive a small penalty to sublight speed and FTL engage speed. But I would even argue against this as well, because players can learn how to exploit this mechanic (keep big numbers of small fleets). Now you can fix this exploit by saying that all ships in a system slow down regardless of fleet size, depending on the number of ships in the system. But then players are also able to abuse that as well, by organizing fleets so all of them would consolidate right before the attack, making a change just another disadvantage the AI should (and fail to) overcome