From my experience, rushing too many colonies will cripple your economy and make you lag behind in tech.
In my experience, you can never rush too many colonies. Lagging behind in tech isn't a huge issue because basic corvettes are the most mineral-efficient combat units for long enough for your early colonies to start paying off.
Even if you can't afford to develop the colonies, having your pops growing on multiple worlds will still massively speed up your eventual growth.
And always invade primitives if you're allowed to do that.
This is against the AI, though. Human players can take advantage of colonies to do horrible things to you with the warscore system (like make you cede your homeworld or become a tributary). Technically, AIs can do it too, but they're not as good at it.