North Africa peaked during the Carthaginian-Roman era. Cartage became very rich through the trade with classical Egypt, the Achemenid Persia, Greece and the later Hellenistic kingdoms.
Roman North Africa became the main source of grain from the Roman West, and thus became of great interest for the Republic and later the Empire. Since it had to feed italy, senators and emperors heavily invested on their provinces' infrastructure, on top of the benefit of trade that it still enjoyed, albeit a that time the trade lines had moved more to the east (Antioch, Alexandria) and north (Rome).
The collapse of the Western Roman Empire started the region's decline, as the market for the north African grain (Italy) seemingly vanished. The Vandals weren't able to keep the irrigation infrastructure in the same shape that the Romans-Numidians did, and agriculture slowly started to decline. The arabic invasion only worsened this, as the irrigation canals were heavily damaged during the wars between the arabs and the Berbers. However, the trade with the East, and now, with sub-Saharian African somehow keep the region afloat, in spite of the gradual collapse of the agriculture system.
Sadly, while that meant that the region became somewhat more prosperous during the High-Late Middle Ages, it never recovered the glory that it had during the Classic Ages. And eventually, when bigger routes of trade opened during the Early Modern era, now without their grain marked, the region gradually became impoverished.