I wish dynasties would die out of their own accord more, families just tend to balloon
But historically they did. If we ever get cadet branches then this may end up happening. Of course, it should be possible to continue playing if your heir is of a cadet branch.
Historically, they died out due to lack of heirs. Matrilineal marriages are definitely not common, and many kings did die without heirs, or their heirs were killed either through intrigue or war or just plain old bad luck.
Another thing is the lack of royalty and nobility going into the church, often third and lower sons would be go into the priesthood to become part of the higher clergy.