Alright, well "if going multi" then I agree, there is almost no point to ever even unlock Terraforming.
You are of course, not forced to go multi.
The only phenomenon that ever forces you to have xenos in your empire is if you decide to conquest planets that already have Filth growing on them.
There are a number of ways to play "tall as hell" that do not require you to really ever conquest. These include Megacorp playstyles, Pacifist playstyles, Federations, Liberators, Vassalizers. Diplomacy isnt great, but there are still tons of ways to use diplomacy to exert influence over the Galaxy, and benefit from it, without directly controlling filth planets.
In terms of mono-species conquest options, Xenophobe has the best purge options, and its still not clear to me whether you find slavery acceptable, but everyone at least has access to Displacement.
This leaves you to find a good way to deal with Habitability while you conquest without resorting to employing all that Filth.
I'll point out that its much easier to conquest in a monospecies way if you first restrain yourself from overconquesting before you have the techs you need to do so.
1. Terraforming
This is probably the most vanilla solution. However, its also actually quite easy to get. Since you are returning, you may not know that its quite possible/easy to strategize your acquisition of the Terrestrial Sculpting tech.
a. Researching using a New Worlds scientist gives Terrestrial Sculpting a 25% increased chances to appear
b. Adopting the Expansion tradition gives Terrestrial Sculpting an additional 25% chance to appear
c. Also note, that you will not receive all those annoying techs to clear 5 billion types of tile blockers if those tile blockers do not appear on any of your planets. Terraforming changes the tile blockers on the planet to ones appropriate for your habitability type, so if you Terraform first before ever colonizing Red or even Yellow habitability planets you will actually have a decent chunk of savings in terms of society techs that you get to skip.
d. Terraforming is only a tier 2 tech, which should be quite easy to obtain, but it can actually be obscured by overcolonizing early, since you will get bombarded with the aforementioned Blocker techs if you do colonize less habitable planet types
2. Gene Tailoring
This is the starter to the Genetics ascension path. While its great for Xenophiles, its also great for Xenophobes and nearly everyone else as well. Relevantly to the current discussion, it also comes with the passive Self-Modification feature.
What Self-Modification does is that when you have at least 3 pops of your Main species on a planet where they have less than 40% habitability, they will after an average of 200 months self-modify to have the same habitability as the planet they are on.
This is a tier 3 tech however, so its slightly harder to start than Terraforming
3. Adaptive
There are at least 3 excellent traits you can get early to increase your pops adaptiveness.
4. Robots
Robo playstyles dont really have a need for slavery or any kind of menial worker, since they already have all sorts of robots for that. Robots come with 200% habitability everywhere, meaning you can specialize your Red and Yellow planets as Rural Robo worlds. Then, if the robot player chooses to fully pursue the robot ascension, Synths come with their own Synth assimilation style.
Your ultimate question however is whether or not any of these options are strictly better than just playing Multi-xenophile. The answer is tied up in what ethics and playstyle you use in addition to any of the above strategies.
None of the tall playstyles are at all hurt by not choosing to go multi-xeno, and these playstyles are frequently at least partly either Pacifist of Militarist. Its possible to use either Pacifist or Militarist as an adjunct to Xenophile, but its also possible to either focus on them or combine them with many other playstyles.
If you're playing Wide and decide to conquest a planet full of xeno-filth... Only Xenophobes get any real benefit from purging that filth. Authoritarians could technically try to sell all the filth on the slave market.
If you're playing with one of the wide options, but you're not Authoritarian or Xenophobe, purging out the filth basically just comes down to a cost/benefit of how you mange these alien pops, and whether in the long run you save resources by doing so. Most players would tell you that "growth is king" so any pop is a valuable pop whether its xeno-filth or not. If you subscribe to that, but have no way to enslave the filth, then yes your whole empire will eventually swing Xenophile.
You do, in the early and mid-game, at least have the option with some Ethics of implementing population controls on the filth, so that it will eventually be replaced by your main species. This gives you all the upsides of whatever neato main species you play, with few downsides.