Yeah I can field that question because I had previously modded Open/Turkish to be available for Christians and tried it for a few games.
Your son with the highest title tier seems to win automatically, and if several are equal, then it looks at the amounts of titles.
So if you have two sons, A and B, and A is a king while B has twelve dukedoms, its A.
If they're both kings, but A has 1 kingdom and 12 duchies while B has 2 kingdoms and a single county, its B.
In the case of an absolute tie, I believe it defaults to age. 1 kingdom, 2 duchies 4 counties each, oldest will inherit, but a single extra barony for B means he wins.
- Sons outside your realm count too -. If you manage to make your son the ruler of a foreign nation, say, by pressing a carefully engineered claim, and ensures none of his brothers are more landed than he is, then inheritance will "merge" your realms into one. I accidently did that once and ended up having roughly Angevin lands plus... Russia when my ruler died. Odd game.
Either way, the sons who lose all get succession CBs and are extra likely to use them. I've routinely had brothers with a mere single dukedom DOW my "two empires 10 kingdoms" main guy. Whats nice is that you only need to beat them once and then they lose the claim and fall in line, becoming good vassals overnight. Although this is much less true for Muslims, who pretty much always hate their brothers at least a little. I'd advise landing properly only one guy and leaving barebone scraps to the others.