It is a real dilemma. A society too "Egalitarian" to recognize the phobias of its own population. Unrest seems a reasonable result if the real world is anything to go by.
That's not a very good comparison given they can legally holocaust the "other" no problem. It's just the slavery part they have a problem with.
Your example would apply to non-xenophobic egalitarian governments.