A major problem of the current system is the population of the colony. You send a 100 colonists each time, they either fail or succeed. And after a 1000 people it is of no use to send more colonists.
I think a few things should change.
Firstly colonists come from either your own pool or can be "bought". The latest group are simply from outside your nation. The costs are for shipping and seeking them. The further the colony, the higher the costs for shipment. You can get several discounts for shipping colonists (from both pools), freedom in the colony is one discount. If you allow people from other religions to live in the colony it becomes cheaper to inhabit the colony. As does the richness of the land. Rumors of gold or rich farmland should make the colony a lot more popular, and thus more populated.
Because of this system, which causes people from different cultures to live in your colony, the culture of the colony could be different than the culture in the motherland. It could also be called mixed, multi-cultural or multi-ethnic or be "colonial".
If the motherland succeeds in keeping this colony and the relation between the motherland and colony stays strong the dominant culture will become the same as that of the motherland. If the motherland does not succeed in keeping the colony, or keeping the relation tight, then the colonys stays a different culture and can develop it's own.
This way smaller nations should have more trouble in getting enough colonists, especially when the motherland is rich, peacefull and stable and people have no reason to leave.
Other things this system does, when for example you allow more religious freedom in the colony, the religion of the colony might be different than the religion in the motherland. This can cause trouble later, while it also gets more people to your colony.
Big nations with a big population are more likely to dominate the colonial trade, such as England, France and Spain, as they historically did. Richer nations can still affort a large colonial empire, but need people from outside the country to populate the colonies as well, as the Netherlands historically did.
If a region in Europe is plaqued by war, famine or other problems more people will leave this region and become available for shipment as colonists, the price should be less, downside is that their religion and culture might spread in your colonies.
Later in the game other question might arrive, such as those mentioned by the OP. If the colonists have been given the same freedom and rights as the people in the motherland, if they pay the same taxes and have been given the same love (roads and other government structures) they are more likely to stay loyal to you. But this is more expensive for you, you have to spend a lot of money on your colonies, money that just as well good be spend on expanding and improving your empire in Europe.
But if you decide to keep spending in your colonies as low as possible, if you don't give them any rights. They start to feel disconnected from the motherland. If they are already a different culture and religion they are very likely to rebel against you. So you should keep a strong military in your colonies.
If you treat your colonists the same as the people in the motherland, then colonists are more likely to rebel, as they are further from the reach of your hands.