How AE/coalitions should work:
A power is becoming a big threat (OE takes out Albania, Serbia, and Wallachia)- this is function of their size (I'd use manpower, but basetax can work in a pinch), their rate of expansion (something like AE), and their past behavior/relations - so the AI's look at each other and begin to damp their mutual hostilities as you get stronger. So the first step is that your warmongering tells Poland that maybe it should ignore the border friction with Hungary and not make a play for Carpathia. So now Poland and Hungary aren't best buds, but Poland starts looking less at the OE as the enemy of my enemy and more as a potential threat so it will let Hungary deal with it unmolested.
This isn't enough (OE takes out Athens, Morea, Naxos, and Corfu), so Poland and Hungary get a free alliance (no slot cost) that is strictly defensive against the OE, Austria begins to lose border friction and other maluses.
Now the OE takes Dalmatia, Croatia, and Moldavia (beating up Hungary & Poland) so now Austria slips into the free alliance with Hungary (thanks to the new border) while Lithuania jumps to a free alliance with Poland. Poland and Hungary form a full coalition that drags all the members in in any defensive war. Joining them are any states who has an average positive relation who think the OE is a major threat - i.e. Trebizond, Georgia, Venice, and Naples; states that have negative relations on average with the coalition (e.g. the Mamelukes, Sheep, and Persia) form their own coalition.
Now we have another war. The OE fights the coalition and alliances. It wins. It now faces a choice, it can make individual peaces - at double (treble or something) warscore AE for territory or it can make a coalition wide peace - that only allows for vassalization, core return (?), cash indemnity, and coalition disruption (e.g. Naples is locked out of anti-OE coalition mechanics for 20 years). The OE chooses to take Hungarian land. So now the coalition grows, but the most threatened states start getting boosts to manpower regeneration (rally the peasants against the destroyer of civilization), tech rate (aggressive OE expansion brings their technology to the victims, victims share ideas and better organizational ideas among themselves) until they equal OE tech, and morale (when fighting on cores of the coalition, protect the fatherland). These mechanisms continue - if you are a big expanding blob the enemies like each other ever more, they become stronger foes both per country, and in aggregate.
So the next war, now Hungary is a OPM with elite soldiers (it is a essentially a garrison state), Poland has been pruned back but is also strong, the coalition also includes Georgia, the Mamelukes (the AI has improved relations & aggressive OE expansion has dampened maluses), Austria, Naples, the Pope, Venice, Genoa, Milan, Tuscany (e.g. all of Italy), Lithuania, and Muscovy. Now the OE loses. Each country faces a choice - make a separate peace (and be evicted from anti-OE coalition mechanics for say 10 years) for full gains ... or hold out until the end. After total collapse of the OE, each state gets to take full gains with full access to coalition mechanics. So say the OE loses and coalition forces overrun the Balkans. Hungary peaces out early for 4 cores. Poland holds on for its cores and release of Moldavia, Naples takes Corfu and Morea. The remnant coalition takes Albania, Ragusa, Dalmatia, Adana, and forces the release of Bosnia, Serbia, Karamon, and Bulgaria as each member makes demands at the end (some just take cash). During a coalition war, after a certain length of war (e.g. 18 months), refusing to make separate peaces results in the offending just taking any of their cores they occupy. Prolonged refusal of strong peace offers (each state has a ticking timer for occupying your land) gives rise to chances that the enemy will just be given your land in spite of your refusal (e.g. hold out through 4 reasonable peace offers and there is a 20% chance the fifth refusal will just hand over occupied territory without ending the war).
Now the coalition is recalced. The OE is much defanged, so the peripheral states (places like Switzerland, Bavaria, and Savoy) stop caring about the OE entirely - they start worrying again about Austria. Places that stayed in the coalition to the end - like say Venice and Austria - recalc the Ottomans. Austria is less threatened and drops the coalition voluntarily. Venice stays in, put loses bonuses to tech, manpower, and morale. The Mamelukes see the OE weakened and immediately drop all the Christian states as they lose modifiers making them like the Christians enough to enter the coalition.
At any point in time, you can try to induce a coalition member out. You give it gold (or provinces) and expend monarch points while assuring it of independence for a set amount of time. You may also delay coalitions by helping out the locals. E.g. the OE returns cores to Hungary from Venice - great Hungary is less worried about the OE menace. You enforce a peace preserving Georgia and guaranteeing it after - great Georgia takes a lot more AE to go coalitionary. High relations from improvement - that also might slow progression down the coalition road.
This is a pretty straight forward system that isn't all or nothing. You have many options to abort - win a war and use WS to reverse the process instead of taking land or vassals, buy people out of coalitions, or actually help them attain their goals - and once in a coalition - the stakes escalate. Yes you can enforce a 100% peace against every coalition member, but yee gods that will be a lot of AE to suck down and the next round will be a bigger badder war where the AIs can each take a 100% peace and holding out till the bitter end can result in massive losses. Coalitions make it easier to rapidly expand when you beat them ... but only by making any remaining opposition stronger.
Coalition warfare should be the most challenging and exciting prospect of an expansionist EUIV game. Yeah you can more efficiently expand by avoiding giving your enemies huge free bonuses ... but if you want to do it the quick & hard way - enjoy the ride.