If most other empires can close their borders to certain classes of pops, or prioritize the flow of consumer goods (and *maybe* food) away from them, or disallow their settlement on core systems, or enact population controls... why should they be entitled to purging pops at all? Even if all forms of purging were not locked to xenophobes, there must be some xenophobe-only options available to distinguish them.
Also remember that factions will petition their empire according to their own ethics-based agendas, and ethics will be more flexible under the new faction system. It may be possible for empires who have conquered a repugnant species to take advantage of rising xenophobic sentiments among their pops.
Lastly, I should say that neutering / sterilization is definitely a worse fate than displacement. And to be frank, this overstated concern over features like enslavement and purging on this forum is clearly indicative of a fantastic ignorance about the history of slavery and genocide in real life. Just like in real life, it will be impossible to ignore marginalized peoples entirely, mostly because to remove each and every "undesirable" esp. on an individual basis is a terrible bureaucratic and administrative feat, requiring sustained supremacist zeal, resources, and labor. We can definitely attribute that to the xenophobes.
PS: Just like in real life, people can petition to withhold the extension of their own rights from others, regardless of legality or broader ethical concerns. I mean, hitler was elected into office! A xenophobic-egalitarian empire is definitely possible, and already achievable in Stellaris.