If I should hesitate a guess based on my (limited) knowledge of those to states history, I would say it was them being in a state of almost perpetual civil war, either openly or within the halls of the emperor(s) involving both the aristocracy and the bureaucracy. I know this is probably an extremely limited and poor understanding, but my impression of both Chinese and Japanese history is that they were a lot more fragmented than contemporary states.
"Perpetual" is an exaggeration. Line up wars and social disturbances in East Asia against the same era in Europe or the Middle East and you wouldn't get the impression that China or Japan were particularly unstable. But, we learn about China through the Three Kingdoms and Japan through the Warring States, so the idea that they were lands of constant civil war catches on.
"Fragmented" also depends on what exactly you mean by that. For one, China and Japan were very distinct, socially and culturally. China's political system was very top-down and while its sheer size meant a fair bit of leeway for provincial authorities, I wouldn't call China "fragmented" except in a few specific time periods. Other posters have mostly criticized the Chinese system for being
too centralized,
too uniform, and
too hierarchical, and while I think this is also an oversimplification it's definitely more accurate than the belief that China was plagued by constant disorder, plague and war. Keep in mind that Chinese provinces are on the scale of nations in other parts of the world so even in disunity China was made up of fairly large, internally-coherent chunks. Japan, meanwhile, was fragmented in the sense that the Imperial house's efforts to create a Chinese-style unitary, monarchic state were not successful and the landed aristocracy militarized, becoming something like European nobility. But again, the idea of Japan in constant turmoil is false. This perception comes almost entirely from the Warring States period which ran from the 1450's until around 1600- before and after that period Japan was a reasonably peaceful place. For scale, Japan's population was about as large as France's throughout medieval and early-modern times.
My thinking is mostly based on how China through its history keeps disintegrating into huge cataclysmic civil wars until emerging with a new ruling family and then the cycle continues. Regarding Japan, my thinking is based on how all the clans keep fighting each other, while the emperor sits as a powerless figurehead at the top. Basically I view these states as two other "Europes" - meaning lots of nations/entities competing with each other for more power over their neighbors.
This isn't really accurate. For all the struggles in keeping political unity over the subcontinent that is China or the mountainous archipelago that is Japan, both Chinese and Japanese had a strong sense of nationhood born out of religion, culture, language, and founding myths. In periods of disunity, it was taken for granted that
somebody was going to reunify it. It was unthinkable that there would be more than one China (see: PRC-ROC relations). Japanese daimyo jockeyed for power and influence but never thought of seceding from Japan. There was nothing comparable in Europe. Frankish kingdoms were seen as sovereign and legitimate, not some kind of transitory stage between Rome and the next European empire.
Also, I believe the Chinese were very aware of what they considered China and therefore of interest and not China and therefore meaningless. I'm less sure about Japanese territorial ambitions, but if they all fight each other on the home islands then it doesn't leave much wealth for exploration - Maybe they did explore different parts of the pacific but those in power couldn't be bothered to "waste" precious resources that could be used to beat up their neighbor instead.
If my "theory"(maybe idea is a better word) above holds up to any scrutiny it would make sense why they didn't attempt to colonize anything. Everything was focused on preserving internal stability or attain more power.
Another possibility is that they didn't have any wants they couldn't produce or easily trade for themselves. That seems to make exploration and colonization quite pointless and an actual waste of money.
There was a great deal of emigration out of China in medieval and early modern times, especially from Southern provinces, but they were merchants who moved to foreign ports along the existing commercial network. So there were large Chinese populations in Thailand, the Philippines, and Indonesia long before there was a serious effort to colonize Taiwan. There were fewer Japanese emigres but they also had settlements in the Philippines and Thailand prior to the Tokugawa Shogunate. So, the people who chose to leave China or Japan were not in search of virgin land or natural resources, they were pursuing business opportunities, and they went where the harbors were. Chinese went to Taiwan to fish but very rarely to settle. Also, Taiwanese aborigines were not affected by Eurasian diseases like Amerindians or Australian aborigines were and posed a serious danger to humble farmers looking for lands to settle.
To compare Europeans colonizing the Americas to Asians not colonizing Taiwan, we need to look at how and why Europeans chose to immigrate. European explorers moved in small expeditions, funded by investors (private or royal) who were interested in discovering new routes to the ports of the Indies. In Central and South America, they found gold and silver, which drew further settlers looking to strike it rich. In the Caribbean and Southeastern US, they found climates suitable for sugar, cotton, tobacco, rice, and other crops that couldn't be grown in much of Europe- investors shipped indentured or enslaved laborers to these plantations as human capital. It wasn't until the mix of mines, plantations, fur-trapping, slave-trading, and the towns, ports and shipping to support them had developed for a few centuries that large-scale, voluntary immigration to the Americas took place in the 18th and especially 19th centuries.
So, in contrast- European investors would fund exploration of, and emigration to, the Americas because there were scarce resources there and they could get a return on those labor and shipping costs (the age of a common person just buying a ticket and hopping on a transatlantic voyage was still far away). In contrast, if you're a merchant house in China, Korea or Japan you're going to send people to the commercial hubs of Southeast Asia, exactly the ports which Europeans were seeking access to, rather than send them to Taiwan to fight aborigines and dig irrigation ditches. European colonization was not about individual people sailing halfway across the world to grow wheat, it was about governments, joint-stock ventures and merchants pursuing serious commercial opportunity.