You can't really do much. This is by design. Eventually, after a bit of unemployment/overcrowding, your population growth will stabilize. If you don't want to deal with the slightly negative side-effects that come from this, I recommend you look into colonizing other planets; violently or by terra-forming

. Otherwise, habitats work as a nice "pressure release valve" for overpopulation problems.
EDIT: So I looked into the defines file:
~/.../Stellaris/common/defines/00_defines.txt
on line 1206, you'll see all the information regarding population mechanics. Specifically, it says that at the point of 1.5 overcrowding, population growth stops. I'm guessing that means that the moment your reach 1.5 times more pops than you have housing for, your growth stops. That can end up being A LOT of people. They would be causing a lot of emigration push, so I recommend you use that to your advantage. Colonize planets, form habitats, make ringworlds, etc.