In all likelihood, the fact that pops do any physical labor in the era portrayed in this game is a bit unrealistic. And it's looking more and more like a lot of our non-physical labor will be automated too. So from that standpoint, it would be more realistic to only play as rogue servitors or other gestalt empires.
Looking at human history, some people (such as Du Bois) have argued that the real "value" of slavery was not any economic benefits, but the sense of superiority it imparts on all of the non-slaves, including even those who never themselves owned a slave. In other words, as long as some people in a given society have a status as sub-human, even a lowly failure of a person can feel like they aren't totally worthless, because they aren't at the bottom of the pile. This is one explanation sometimes given for why large numbers of people who did not own slaves and were even economically harmed by having to compete against free slave labor still fought to preserve the institution of slavery.
But never forget that Stellaris is about simulating empires based on classic science fiction tropes, not an attempt at realistically predicting future society. Because huge numbers of slaves are nearly ubiquitous in science fiction, and are often depicted doing manual labor like mining and farming using very primitive tools, the same thing is very fitting in Stellaris. Having slavery in this game fulfills several common trope fantasies:
1) The horrible, oppressive slaver empire that dominates all.
2) The xenophobic overlords that tolerate aliens only if they are servile.
3) The nihilistic culture that will do anything for money, no matter how vile.
4) The glorious, shining liberators who free the enslaved and oppressed everywhere, because you can't be a beacon of goodness unless somewhere else there is a darkness to shine against.
etc.