I like the new system, but I find it very confusing too. The constant spikes in ressources are very hard to manage, because like OP says it's realy hard to see where the problem come from. The job system is also very anoying, probably because I don understand realy how it work, I find myself with a food deficit and a planet with 3 farmers jobs empty, just because my pops prefer starving as specialist than working in farm.
Is there a solution for this? Can we force a population to get a job without being autoritarian?
Yes. Don't build the specialist jobs in the first place unless you *know* you can support them. Always make sure your basic districts have plenty enough staff first, and only create new specialist jobs when you can afford to lose a few standard workers, because they will leave their old jobs immediately for the higher tier ones, and you will have a hell of a time trying to convince them to go back if you decide you've made a mistake.
I know we've all been trained by the previous (terrible) tile system to fill every slot and build every building, but in the new system new buildings are not automatically good. Pops are happy to promote upwards, but will get very angry if forced to move down. Imagine if you were sat at a comfy office job one day when the boss walked in, handed you a pickaxe, and told you you were a mineral miner now, over in the deadly explosive plasma hole down the road. Oh, and it comes with 90% less pay. Enjoy!
Personally, I love this new system. The old one required 1000s of clicks, but involved almost no actual decision making or strategy. The new one seems to be the exact opposite. Micromanagement is way down, and yet strategy and thought is way up. I'm sure a massive change like this will have tons of flaws, no doubt it will require balancing and tweaking, but at its core for the first time the economy is an actual fun part of a strategy game, rather than a boring chore that needs to be completed to carry on playing. I couldn't be happier.