All the current scholarship stresses above and beyond all else that it's simply impossible to authoritatively state whether populations and economies rose or shrank significantly during the later empire. The sources arguing for massive deaths and economic catastrophe are fairly untrustworthy, while the evidence arguing against the deaths and depressions is too sketchy to be convincing either.
For instance, Whittaker and Garnsey (the authors of one of the essays on the 4th century rural economy) state that for all the sources claiming that land was going fallow and there were not enough people, there is a great deal of archaeological evidence for an expansion in the number of farms at both the medium and large levels (the former apparently reflecting the reemergence of villages). Britain emerged during this period as a major corn exporter and North Africa shows a dramatic increase in the number of villas.
Obviously no direct comparison can be made between the late 4th century and the late 1st century. The societies, even if there had been no disease or wars, was massively different. But the evidence we do have argues for ample prosperity in the 4th century and beyond. In the mid-6th century and early-7th century, even after the disastrous bubonic plague, the emperors had enough armies to finish the war in Italy, pacify reconquered Africa, fight off Slavic and Avar invaders across the Danube, and engage in a 30 year war of total destruction against a Persian army which initially was surely a match for the army the Arabs would soon provide.
The evidence for dislocation caused by the plague is there to see (see Averil Cameron's Procopius and the 6th Century, for example, on how much of his invective against tyranny now appears to be his reaction against the measures necessitated by plague (of course, he drew no connection between the plague and its policy consequences).
As Goffart, I believe, said, the world would long since have been wiped clean if people died as often as the sources said. This doesn't mean that the plagues didn't happen or weren't catastrophic events (particularly the 6th century one). However, these societies adapted and survived. You can argue that plague added another stress to a difficult situation for the empire (in any century - viewed from one point of view, the empire was always on the brink of death), but you can't prove it as the major and determinative source of later downfall.
driftwood
, I'd like to also point out that Gothic revolts linked to Adrianople were due more to the general incompetence of Valens' administration during the entire affair than to a fundamental weakness caused by reliance on Goths.
