I think there is migration happening between most empires with open borders to each other, but it's not enough to show on the planet view. A pop is probably somewhere around a hundred million individuals, and a few thousand people migrating to and from an empire just isn't enough to register at the administrative level you're playing on.
Migration pacts, on the other hand, would be two empires promoting migration between them and and actively assisting with it, causing much greater numbers of people to seek the other empire as their home.
Same with the refugees, likely most empires get some of them and the event just means that enough of them reached your space to form a pop.
But I mostly agree with what you're saying, there should be some extremely slow migration between suitable empires, and you'd have to use a specific policy to stop that or possibly increase that.