Like your ideas, Jyanix, though micromanagement and handing fleet control to AI becomes a bit of a concern. But it at least gives a reason to have your fleets non-doomstacked before war and consequences for doomstacking during war. I don't know that it would be enough to deter doomstacking as the strategically superior move in all cases, though, or that the AI could properly weigh the subtle consequences of showing weakness at the border.
Seems that the problem of the doomstack is correctly diagnosed here - that wars start and end in a single battle. Seems that taking a page from CK2 and EU4's playbooks might solve that.
There needs to be a "Disengagement Doctrine" policy that behaves similar to breaking morale in those games. After all, why in a game with a core premise that the leader of a government can't simply control everything do we have control over a retreat button in battle?
We should set a general policy on how much damage a ship should accept in battle before disengaging by attempting an Emergency FTL. Like 30% HP, 20% HP, 10% HP or "To The Bitter End". When ships hit that point, or when total fleet loss hits that point, they E-FTL to a rally point (Each Fleet should have a mandatory rally point, set to default as their last point of orbit or something but manually changeable). As each ship jumps, it's determined whether its damage prevented its safe retreat just like in any other emergency FTL. So there will still be some losses - but a losing battle will generally result in a reduced fleet regrouping at a predetermined location, where you can then retreat further, repair, etc... So, just like armies retreating in CK2 and EU4 for several provinces and then regrouping to continue the war, fleets would be able to recover and resume the war effort. Could have admiral variables effecting things too - some admirals trait to reduce losses during E-FTLs, a stubborn "hard as nails" trait to cause ships to stay longer than the Disengagement Policy point, or a "lily-livered" trait causing retreat earlier than the policy point, etc...
Other idea on this topic would be to make the occupation of systems and destruction of bases or bombardment of planets much more significant in warscore, so that having multiple fleets occupying multiple enemy systems, while more vulnerable to a mass counterattack, could viably result in enough warscore to win the war without destroying the enemy fleet. The effectiveness of this strategy could also vary depending upon the enemy traits and ethos - the millitaristic ethos, or the some trait or other, could make a species more resistant to occupation, and therefore occupation of systems would have a reduced warscore against them.