Can't you deactivate specialist buildings at will? You'd get a little unemployment, but on a newly-conquered planet that's not going to matter because Stability's in the crapper anyway.
Amusingly my first game learning the new system, with the terrible death spirals from conquest wasn't actually because I'd taken a world. It was because I'd abducted slaves.
Story time: I made a weak, decadent race (butterflies) who raided for slaves. I'd played this race in 2.1 (they were weak) and wanted to see if the pop system made them any better.
So I settled half a dozen worlds, I picked the ones that were habitable and decided they'd be my specialist worlds.
I also settled fairly uninhabitable (desert) worlds and planned to dump slaves of the appropriate type on them to mine/farm/etc.
I declared war on each nearby target in turn (starting with my desert friends, then arctic friends etc.), stealing pops and ending the war to go weaken the next guy.
I now had slaves... sounds good... wrong! I was strong enough to beat the AI without slaves with a weak race, I didn't actually need the slaves at all... and there would be consequences...
Now I noticed unemployed pops on my Tech world. So I figure, build more labs. Still more unemployed so I upgrade the labs. Economy doesn't seem to budge (clue number 1). Yet more unemployed... I've been re-settling the slaves that keep being dumped on this world by raiding and I have no clue why they wont automatically go to other worlds that have free mining jobs. I figured they were dumped on the first planet alphabetically or by distance or something...
Eventually I've been resettling so many slaves that get dumped on this world that I worry I'll accidentally close buildings so I start moving all the clerks/farmers/miners of my main race to the tech world from my lower habitability worlds (there aren't that many of my race - clue number 2) and when I've done that I notice my economy has tanked...
Turns out on all my worlds ONLY slaves were growing and had been for the past decade of wars (I had like 6+ slave species, raiding xenophiles) and the reason my tech world was attracting all the slaves from raiding was that it had lots of unfilled specialist jobs. It also had lots of unemployed slaves (and domestic servants working the few mines rather than entertaining). So in moving my few remaining own species across I took the hit for lots of new specialists. I also found out that I'd have to set my own species to force grow on every world at -20% speed if I wanted to man the capitals of any newly conquered worlds. My economy needed a lot of faffing about to get back into shape while all my nearby enemies were sitting on 4+ ruined buildings that they never rebuilt. Game became unfun - the only challenge left was the economy.
So:
I waited to build labs/alloy plants until I was rolling in minerals and had unemployed pops.
I then upgraded the specialist buildings noting I was still earning lots of minerals.
I resettled pops from unsuitable worlds to more suitable worlds.
I tanked my economy because I hadn't hovered over the pop jobs symbol or opened up the specialist tab, thinking that there'd be no possible interaction as only 1 species could do those jobs.
I discovered:
The pop growth system is stupid,
Raiding is OP,
AIs can't recover from raiding,
Micromanage pops or very bad things happen.
Amusingly the next game I tried robots and had entirely different problems... job priority stuff, robots taking jobs when they produced about half the output of my organic workers and vice versa. Gained Psi tech before droids and was frustrated I couldn't focus on engineering. Also had a criminal megacorp set-up and crime started doing strange things, lots of events firing at 0 crime. Felt really buggy so quit. Also felt sad that I couldn't encourage either of the two allied megacorps to set-up on any of my worlds even when I gifted them 10k+ minerals and energy each and they had no other trading partners (lots of purifiers/devouring swarms/megacorps but no normal races).
Then a game where I was buying slaves off the market... I had more of their pops than they did after just a couple of years... crippled them worse than raiding. Their capital was lower population than my colonies. I felt bad for the AI, scrapped that game. Odd note: the slave buying and settlement UI is actually really good, the list of planets, jobs and housing is AMAZING... but why isn't that information available anywhere else or from the start of the game? It'd make a better outliner than the existing outliner.
Then a criminal megacorp... this time I actually looked at what the AI had built on its planets and felt bad for them 4xEnforcer buildings, no upgraded buildings or exploited rare resources etc. Also felt bad about my limited options and the branch closure mechanic. I tried using the market and, curious about the prices generated infinite resources... quit due to exploit and decided to not use the market at all till it gets fixed so I don't accidentally cheat.
Then a normal (well pacifist spiritualist megachurch) megacorp... found out vassals/impose ideology were broken. Reloaded an earlier save and did things differently. Also found out that imposing ideology created a megacorp... for some reason I had hoped they'd pick a random authority and let me install branch offices because they like me for liberating them. Megacorps don't play well with other megacorps... I didn't want to make more of them. Just making subsidiaries worked better... but the AI still felt bad. Yet put up enough of a fight attacking me from 3 different directions (joint wars) with decent fleets despite their oddly poor economy that I felt uncomfortable with raising the difficulty too much more.
Anyway... so several games started but not finished. Also started up as a hive and stopped after 1 war as I'd instantly converted the nearby hive mind to my side and the rest of the game would have provided no challenge whatsoever.
So every game I've had issues with the Job system. Lots of different little issues and some huge glaring maddening issues that people seem to be willfully ignoring. The entire system feels rushed... but the changes in the newest update have helped a great deal and I look forward to more improvements. Like better tooltips, they're not useless now but still need work. I'm looking forward to playing the game when it all gets sorted but for now it kinda gives me a headache trying to work out if I've done something wrong, if it's a bug or a design flaw and how I'd go about fixing it either way. Perhaps I've been harsh but every game so far has been unbalanced with completely new bugs and issues. I'm hoping that one combination of ethics and civics will feel fun... but I haven't found the fun that I had before just yet. Instead I'm repeatedly saying to myself - "well that's obviously not working properly yet. That's an exploit. That's OP. Should that be happening? ...try something else" over and over again. Then coming on the forums to whine about it and vent my frustrations. (and being called a retard for my troubles... which isn't so fun either and could quite easily put a person in a foul mood).