If I do settle, I always settle to either grab more Cosmite or something realy valuable later (like a Dwelling or higher Tier Landmark).I do not usually build any new colonies before turn 30, never caused me any problems. I use my initial cosmite to mod units, get the units to be elite and set up lots of forward bases at the locations I eventually intend my colonies to be. New colonies just demand more starting forces because you need to clear out all their sectors, the extra which needs cosmite to mod them, or you will be spending energy and production replacing them when they die. It is quite a while before your new colonies are worth your time to have, but all those extra forces need upgrading and in some cases replacing now.
Bonus points if I can not grow because I lack Happiness buildings.
That way the Colonizer investment quickly pays off.