Chose your side? More like they remained out of the rebellion, that does not mean they chose your side. Importantly that was pretty much how things were back then. Swearing an oath of fealty isn't the same as making an all-in in a poker match.
Two things I can mention without knowing the details of your game.
Firstly, which is quite obvious, is that equalizing vassals as if one vassal should be equal to another one is a plain misconception. You can have 3 vassals whose power is much greater than another 20 combined, and that is not game breaking, that's how the game works.
Secondly, when you declare a war, your vassals are not declaring it with you. That is to say, you are depending on your MAA and the levies from your personal demesne and the vassal taxes. You are not mobilizing your entire realm and the force of all your vassals with you.
When they declare war on you, the same applies.
So why are those 22 vassals having a bigger army than your character? Because the correct comparison is between these 22 who are personally attacking you and YOUR character, not you AND the other vassals who didn't rebel.
On one side you have 22 characters using their personal MMA, own levies and tax levies. On the other you have 1 character doing the same.
Why should your side be bigger than theirs?