Unless you used slaves.. which seems to be the major benefit of xenophobia.
Essentially, it seems like they kind of do the same thing in terms of making your empire able to tolerate alien pops, it's just a question of whether you want the increased productivity but increased risk associated with xenoslavery, or the general all round multispecies chillness associated with xenophilia.
Bear in mind that in both cases the benefit is that different pops will have different traits, so being able to import a species which gets a big bonus to food production and have them work in your farms, for example, would be a pretty significant benefit, above and beyond how generally cool it is to have a multi-species empire.
You are probably right about that. I think that's the ONLY way to have your xenophobic species accept being able to live next to another one. They have to be slaves.