The disastrous failure to assess the true military strength of the Soviet Union is hard to fathom, but then again, how would you collect all of that scattered information. The Soviet Union was heavily decentralized, which makes it difficult for an insider to control, but makes it even harder for an outsider to spy on and get reliable information on anything outside of the immediate vicinity. There are a LOT of vicinities, and a lot less spies. Ultimately, the Germans destroyed more of the Soviet army than they thought existed. It wasn't even close to being enough. The ability of Stalin to prevent panic and the dissolution of the state under the stresses of losing those early battles was enough to turn the "improbable" into the "impossible" for Germany.
On the contrary, the USSR was heavily centralized. Due to the obsessive and paranoid culture of the Communist Party, pretty much all information, especially about the army and industry, ended up with the leadership. STAVKA was supposed to know everything about its' respective military units, though in theory the widespread corruption in the Red Army created great discrepancies between the numbers STAVKA got and what was actually true in the field.
What's more pertinent to Germany's problem are three things:
1) The USSR was a police state. Reading private letters, eavesdropping on conversations and wiretapping phones of medium and high ranking officials was the rule, not the exception. The Soviet leadership, Stalin in particular,
was paranoid. Information was severely restricted and compartmentalized. This reason would hold true for most of the Cold War too, which partially explains why Western intelligence agencies had such a hard time getting good intel out of the USSR.
2) The Great Purge removed the chance of infiltrating the Red Army. Whatever or not an actual conspiracy against Stalin and the Communist Party actually existed, the purges made sure that any foreign infiltrators were unable to operate, if they weren't outright killed to begin with. The level of fear and paranoia in the Red Army prior to WW2 was insane, far above the already high levels found in civilian Soviet society. To be the agent of a foreign state in an organization where even unfounded accusations often led to imprisonment or death was to play Russian Roulette every time you acted even slightly suspicious.
3) German intelligence gathering was very never good. The Abwehr's biggest claim to fame is the number of double agents that
double crossed it. German military intelligence had a propensity towards underestimating its' enemies and everywhere except on the operational and tactical level (where the Wehrmacht was very proficient in reconnaissance) German intelligence gathering efforts were sub par compared to the efforts of its' enemies.
The initial stages of the invasion were a huge tactical success and a significant strategic one for Germany, but were simply insufficient on a Grand Strategy level. The German army pulled off a miracle, but found that several more of such miracles were required. The Soviets replaced their losses and just kept fighting. Other nations' armies reinforced understrength divisions in the field; Germany waited until its divisions were reduced to combat ineffectiveness, then pulled them back as a unit to receive replacements and rest. Until they reached that pathetic state, the Germans typically pulled their skilled technical and support personnel off their normal assignments in those divisions and threw them onto the front lines in desperation, where they were chewed up almost as quickly as raw recruits would have been. You can train and replace riflemen in a matter of a couple of weeks or months, but training and replacing the technical support elements takes years. Germany never recovered. In essence, Germany's armed forces "ate" their own logistical tails. While it can work to provide a final push in a short war, it's a recipe for disaster in a long campaign.
Germany also had really small "front line" contingents, even in their front line infantry divisions. Out of 16,000 men in an infantry division, only 3,000 or so were riflemen and another 1,000 or so front line support like machine gunners and pioneers. This meant that German divisions quickly lost their fighting capability because even moderate losses would take large tolls on their front line strength. Compared to a Soviet divisions where more than 2/3rds of the infantry division personnel were riflemen or front line support. This allowed the Soviet divisions to absorb far more casualties before being rendered combat ineffective.