I think the biggest problem is that there actually were some cases of cultures being massively displaced over the EUIV time period. The Tatar lands, for example, were massively settled and colonized by the Russians as they expanded southward and eastward.
On the other hand, it is ridiculous to see the Balkans entirely Sunni and Turkish in a few decades, or Austria turning the entire Netherlands and chunks of Italy into Austrians. For this reason, I think culture conversion should be inordinately expensive in 'civilized' areas of the world, so you don't see random enclaves of cultures popping up in bizarre places.
Another alternative could be making colonists the agents needed for cultural conversion, that way only nations which take exploration/expansion (or have colonists in national ideas) would be able to conduct cultural conversions. This would also fit in nicely with my Tatar example above, since Russia's insanely fast colonization of Siberia could be slowed by making them use their colonists for culture-flipping the Tatar provinces from the former Hordes.