I tried it... it was fun watching all the slaves rush to my empire. But I had a few annoying problems:
1. Slaves were growing before main species pops even when there were unemployed slaves and many unfilled specialist jobs - not just growing but also being thrown at that planet when raiding. (I think the game mistakenly assumes they can fill specialist jobs)
2. Raided enemy empires were crippled forever, never rebuilding buildings demolished... that made me feel quite sad.
3. Consumer goods were scarce, a mix of low habitability for the mix of races and my science push when I expected to be able to stay in militarised economy for the extra alloys since slaves use less consumer goods.
4. All my planets had to manually focus on growing main species to stop the slaves from ballooning out of all proportions, but that meant a -20% growth penalty. Either that or prevent all slaves reproducing and take the happy/crime/stability hit. That would probably have been more sensible, but I wanted to be able to grow certain powerful slaves when needed.