It doesn't really work like that. If I have 4x as large a fleet as my enemy and it works out so that it will only do the square root of its superiority as damage, it will do twice as much damage as the enemy, and take half of the damage divided over four times the ships; by the time the enemy has taken 100% damage per ship, I will have taken 12.5% damage per ship or ~50% of the enemy's health. If my fleet was only twice as large and I was doing forty percent more damage than the enemy, I'll take 70% of the enemy's health in damage divided into 35% per ship. That is to say: while I'll take more damage with a bigger fleet than I would have without the penalty, I still have no incentive to split my fleet unless it's to try to flank for that pincer bonus you mentioned later, in which case all you've added is more micromanagement.But in small vs large the bigger fleet would still beat the smaller one due to having more guns to bring to bare, but the bigger fleet will be take a disproportionately larger amount of damage.
It's the same principle as back when people were suggesting area of effect weapons to counter doomstacks: if the weapon does enough damage to be worth using to kill doomstacks with, a doomstack will be able to use it more effectively.