Another weakness is that the Ottomans are EXTREMELY vulnerable to European players, even more so than other countries. They are nearby enough to get your forces there quickly, and usually have some tempting provinces (Greece, parts of Syria, Thrace). They are the perfect religion to attack, and can fairly easily be converted to the Christianity of your choice. They are moderately powerful, but technically backward and will not have the money, manpower, land or naval force limits of any consolidated European country. Once you get Burgundy or Castille up and running you can totally crush the Ottomans with no effort and the kicker is that they will basically NEVER be able to fight back against you. Even if they had a navy large enough to get to you, which is doubtful (and their navy is usually 75% galleys anyway) they will never be able to land enough men to really hurt you, and all the other blobs in Europe hate them and would never let them keep anything even if they managed to take it.
The Ottomans are very easy and tempting to attack and are basically incapable of counter-attack. Even if you misjudged and the Ottomans sank your fleet and killed your army you're going to have a new army before they can ever get to you, not to mention that half of Europe would probably also be at war with them.
The Ottomans are basically Byzantium-lite: all of the headaches, none of the advantages.
It is quite realistic for the Ottomans to never get past the Balkan region - it just wasn't practical to hold onto, and probably not worth trying - but it should be equally difficult for the Europeans to go trapsing around Anatolia and Syria. It isn't.
The same is true of the Mamluks. I'ved DoWed the Mamluks within 3 years of start as Castille (for Jerusalem) and it was a piece of cake. I crushed their stupid galley navy and casually took all their provinces with alternating infantry mobs (military access in Achea + 15 transports = insta-healing armies). The Muslims only last past the 15th century because no one but Castille feels like conquering them. They are really, really crippled in EU.
The crusades tore through Anatolia to GET to Jerusalem, so I don't see why the OE themselves can't be the target.
Aside from the total lack of enthusiasm for Crusades, you mean? The Crusade feature is very anachronistic. The Habsburgs had trouble getting any enthusiasm out of the German princes for evicting the Ottos from Hungary and Bosnia, much less Anatolia itself. No one raised a finger to break the first siege of Vienna by the Osymuns, and nobody gave a farthing or a whistle to help the Habsburgs get out from under the tribute they were required to pay to the Ottomans. This is DESPITE the Habsburgs being the HRE and Spain through some of this period. And at no point did Castille occupy every Muslim power from Morocco to Iran, as they frequently seem to do in the game. Castille didn't like France. Castille wanted to control the straits, Italy and the Low-Countries. Castille did not care about Syria or Turkey or the Balkans or anywhere else that was not a convenient and plausible addition to their dominions.
For all the Holy War rhetoric employed it was mostly just rhetoric. The Ottomans were more or less politically accepted as another European state, one which could perhaps be attacked without upsetting the rest of Europe (except for France, who allied with and funded the Ottomans at several points) but which was not viewed as either an existential threat to Christianity or as being any worse than Spain (most of Europe was much more worried about the Habsburg blob than some distant Turkish Sultan).
In any case, Muslim powers are very weak in EU and perform very poorly; and their territory is very easy to hold on to and convert. This is a persistent problem in Paradox games, it is very easy to take the Holy Land and most of N. Africa within 40 years of the game start in Crusader Kings, and any European power can kick the tar out of the Ottomans in Victoria.
The Ottomans and the Mamluks can do incredibly well in player hands, but that just has to do with them not being in a totally crap tech group and having several provinces. Any country of Muslim tech or higher with 5+ provinces could be turned into a super-power by a player without much difficulty.