Which requires that you build the Colossus.
What game are you playing? It's certainly not this one if you think it takes that long to get a system productive.
You're right, my bad. Usually I have the minerals (and a conquered FE starbase) to build it right away, so I never pay attention to when the CB is unlocked. The Colossus itself just isn't an integral part of waging wars, so I only use it in a quite limited capacity. My point is, mineral costs and fleet capacity are irrelevant in the lategame, as is the Colossus itself. The true treat is the unlocked CB, which boosts territory gain and your economy (more edicts).
And don't forget that we talk of a difference in size of around 20%. Even with 20% less systems I have better things to do in early to midgame, like developing my core regions instead of some far-away freshly-conquered and hard-to-defend systems that border my rivals. Of course those systems will see some development in time, but only when my minerals are not needed somewhere else. So yes, it might take a few years until those systems become a net profit (not just productive, but a net profit - and that's the point of investing into conquest).