Another bug you didn't mention about slavers is how slave processing centers will auto-delete if the planet you build them on is not a thrall world. I find that one frustrating as all hell.
Well, after realizing that slave processing plants don't reduce effective slave political power I never bothered to build them, because they're just 5% slave output building that are not really worth a slot unless you have a lot of slaves in one planet. I did see a lot of bug report threads on that issue thought. Hope it gets fixed soon.
Also, even if you were to resettle something like 200 pops, you save only 5000 EC over the span of decades. I personally think Corvee system is quite weak compared to other options you get.
Anyways I guess I'll have to take the 20% penalty afterall. What's annoying about it is that once I get more main species by force-growing them I'll be stuck with -20% growth penalty forever, because if I unlock it the game would favor minorities even more than before and I'll be getting no more extra specialists again. It's a cycle that I really wanted to avoid getting into, but I guess I have no choice.