So I finally did it. I had to uninstall the game, install it again, but without the patch. The plan was to make Japan go bankrupt, force convert them to Islam, vassalize them and annex them. It worked.
I sold Japan a 1 Base Tax Muslim province in their culture group (my Ryukyu being Muslim). I attacked Japan and took their Capital province, leaving them with the poor province I sold to them earlier. I attacked again and embargoed them (me owning all CoT:s in Asia making it impossible for them to trade at all). My goal was to make them go bankrupt, which they refused to do. So I funded several nationalist, which was possible since my Ryukyu had made a culture shift to Chihan and the province being an old colony with my culture. Since Chihan share the same culture group as Japan they defected to Japan the same moment the province fell. Since I was afraid they would disband their army, I attacked again (stab hit of -5, ouch) to make them keep their army and raise upkeep. Since the rebels were funded by me, they did not fight my troops when I occupied their province. Japan being broke already and getting no income, instantly went bankrupt. So I made peace and funded some more rebels which made their government fall... and bang! Forced conversion. Japan became Muslim. Since I was playing 5.0 and not 5.1 I was then able to vassalize them (the cost for vassalization being 350 % warscore in 5.1 and 8 % warscore in 5.0). 10 years later I annexed them. Japan is gone from the map!
Compared to the Jihad achievement where you have to convert everything to Sunni Islam, while Najd being Shia Islam, this was almost easy. Anyways, this really annoys me. Paradox made two challenging achievements and screwed them both up.
I did not have any problems annexing Japan in earlier games. My guess is it is possible to annex them in a normal way if some of the Daimyos are gone from the map. So If you are lucky the AI will do the job for you.