It's difficult to reconcile the fact that refineries produce 1 job per building slot. This means you need to create 4 other jobs in order to get to the next refinery building. You can't use T3 buildings with their 8 job per tile since they consume 2 motes, gases, or crystals. Besides, if I'm using T3 buildings on the planet, I'd rather specialize for Alloys, Consumer Goods, or Research, and devote as many building slots as I can to T3 buildings.
An alternative is to build these on a world that's district rich. Since rural worlds only give a negligible 2% bonus to basic resource production, it's not a bad trade off. If it's a dedicated agri, mining, or generator world, building a few refineries shouldn't swap the planet specialization - but then it's not a refinery world.
The last thing I've tried is to pad out jobs with upgraded Commerce buildings, since they provide 11 jobs per building slot and only consume 1 crystal. But the going consensus seems to be that clerks are mediocre.