Because Imperial Japan had no other useful use for its transports and tankers than to transport troops on the other side of the Pacific Ocean, and have them return empty-handed. Raw material and oil, be damned, let's invade San Francisco! In fact, one of the main arguments against attacking Midway, which is half the distance in the middle of the Pacific, was that such an island was hard to supply, would take precisious cargo fleets away from key supply lines, would be left nigh-on defenceless to US bombers from Hawaii even after they've taken Midway and left, and would consume fuel twice because the cargo would return with their keels empty.
Also, never, in a million years, the Imperial Army would have agreed to an invasion of such a scale. They barely agreed to liberate the equivalent of five divisions to the Navy for their whole six months campaign after Pearl Harbor, and they bitterly fought agaisnt the Navy to refuse any military operation that would have taking divisions away from the Kwantung Army in China.
