Yeah, I think that no matter how vague GW is about the dark elven population growth, anything that is not a positive number just sounds like they would have gone extinct a long time ago.
The Asur, in comparison, live a much more peaceful life but their population is still going down, meaning that it is somethign about Ulthuan or the Asur which is the cause of this. My guess is the Vortex, but it doesn't really matter... the important thing is that the Asur are losing numbers even during times of peace, while the Druchii and Asrai apparently are not.
Just mentioning, I remember reading something about an official Statement from GW that they have an internal policy on the elven population.
Which basically says that there are always exactly as many elves as the storyteller actually needs to have for his story to make sense.
So the Asur might be close to extinction one century, then field hundreds of thousands of soldiers in another, and so on. Same thing with the Dark Elves.
Mostly because, as you said, it wouldn't make any sense if they actually gave definite numbers on this for any of them.
From a gameplay standing, your idea with the Vortex seems legit (for me) and as long as there is a legitimate way of reversing or slowing it at least (say, not doing any unnecessary "colonisation" projects) then I'd back that completely.
As to dwarves, I also think that they mostly die because the Slann messed up their whole tunnel system, everyone is shut off from the others, and they have to fight 24/7 on about 340 days of the year (some few days are free

) for sheer survival against all the explosive breeders out there. They are a little less fertile than humans, as is average, but they are much much sturdier and they never change their ways, which is what will kill them off in the end.
They don't really have a way to get things properly going, and in those rare cases where they meet, they either can't decide (because every dwarf thinks differently and all are too stubborn to change their opinion) or they decide for the known way of staying on the defense until things outside look better.
I dearly love those grumbling little mead barrels with beards, but they are the race for which I personally see no hope of survival on their own in the End Times.