[SARCASM]Yeah, because it's definetely how it works in real life. That's why prisoners dilemma is constantly solved for double cooperation and we are having world peace instead of the risks of mutual guaranteed annihilation and reasonable resource management instead of global warming[/SARCASM]
The point is that slavery lobbist couldn't care less for your science worlds. They care for their profits instead of your plan to optimize the economy and they use their brains to increase their income and political power instead of making things easier for you. They won't support the idea of reducing the number of slaves, even on a single planet, even if it's beneficial on the grand scale, because it is a step in a direction of limiting their power. So they will keep enslaving pops everywhere, knowing that the goverment will have intention to send these slaves to the mining worlds anyway.
And it logically follows that when you a have a planet with no profitable employ for slaves next to a planet which does, slavers are not just gonna hold their slaves out of employ for the sake of it. The individual incentive is to sell them on or rent them out. It may simply be that the slave owners remain on the initial planet while they outsource the labor.
There's a reason why slavery flourished in places where the agriculture suited it, like the Deep South, and was less common in ranching country like Texas. Even when both places operated under the same empire. Even when the slave owners themselves still had their homes in Texas.
And when there is no domestic market for slaves, slavers would still produce them for the galactic market. But if they really had as much pull as you say, then they should be putting slave pops up for sale themselves, against your will as the player.
The prisoners' dilemma would be in stopping all this trade, not starting it.
And this is all assuming that slavery is privatized rather than state directed to begin with. Slaver "Guilds" could simply refer to government departments (and is unlikely to be a literal association of medieval craftsmen). Stellaris is usually agnostic on precise economic details for a reason.
The issue here is not that there should be no interesting socio-economic inertia to slavery, but that the inertia that exists is 1) inelegant to manage and 2) uninteresting. You can manually resettle your pops back and forth to in order to separate a slave empire into "plantation worlds" and "mansion worlds". In fact, the game already legitimizes that choice by including Thrall Worlds. So the solution you're saying should be difficult to obtain really isn't. It's just a lot of perfunctory weed-pulling when it should be an automated process stemming from policy choice, which is what pop management in 2.2 aims to be.