Heh, a fleet with lvl 4 weapons will absolutely get destroyed by a fleet with level 1 weapons but 4 times the size. Especially if crystal plating and/or shield recharge modules are involved.
Try a one planet game with the same setup, except 32 AI empires, and you'd find far more of a challenge.
That's a bad argument. Either you are aggressively expanding - in which case you will have no minerals or energy credits for 4 times the size, because you spend hundreds of minerals on colony ships, building upgrades, farms, star ports and removing tile blockers, while most of your energy upkeep will go towards supporting the colonies, star ports and the industry necessary for the minerasl - or you're not, in which ase you're just doing what he does with more focus on military.
Fact of the matter is: his strategy allows him an extreme early game advantage BECAUSE he isn't investing any of his minerals into developing colonies. Colonies take a long time to pay off due to the high starting costs and the time involved in building/clearing/growing pops and upgrading buildings to be more efficient than orbital mining stations.
All the time that you spend investing in your future industrial base, he spends doing research. His planet will be at optimum developement quickly (because he only needs 1 governor to boost his build time and reduce his build costs while it developes, after which he can replace that governor with a research oriented one while at the same time boosting with protocols because he doesn't need influence for starting far away colonies), after which all of his minerals go into orbital stations that are the best very early game bang for buck investments.
Sure, if you had 4 times his ships you might be able to murder them if you get close enough fast enough, but your ships are going to do next to no damage. They'll be outranged, quite probably out-flown (higher level engines means higher evasion) will have higher hull points, armor and shields and do much more damage because they have the sensors and high level weapons tech to actually hit your corvettes while you still trundle around with tier 1 or 2 stuff.
I can understand arguing that his playstyle won't win lategame because he will at some point drown in ships against an 80 planet empire if he never expands past 1 planet, but it has you beat hands down early game and in a pvp game that's when he would crush you. You're like a protoss arguing that the zerg can't beat you with a rush if you have a full supply army backed by 4 bases - yeah, because it would never get to that point as the zerg will rush you when you are still busy buying your second or third HQ, much less any units.