Another thing that amazed me was how the ERE lost these areas relatively easily and quickly even when the ERE outnumbered these enemies but when under the control of their foes, their foes were able to easily defeat attempts to reclaim them, even though the control of their foes over these lands were relatively short.Perfect example would be Bulgaria. The ERE outnumbered the Bulghars during the latter's invasion of Moesia, but lost control of Moesia despite the terrain's favour of defense. Yet, when the ERE tried to reclaim these areas, once again outnumbering the Bulghars/Bulgarians, they got massacred over and over again because of the terrain.
Well in a sense that goes back to the difference between centralized and horde polities, doesn't it. A centralized state usually loses land following the defeat of a single army (even more so for such a comparatively 'oppressive' state as Byzantium), whereas to wrest control of lands from a horde you basically do have to occupy everything, which generally required resources Byzantium didn't have. I think freedom from Byzantine taxes probably meant that peasants were more than happy to welcome their new masters, in general.
If you're talking about the Bulgar invasion of Moesia in the 7th Century, bear in mind that the Bulgars invaded while the Umayyads were on the verge of destroying the Empire, so I'm not terribly surprised that there wasn't much resistance to their attacks. So when Constantine IV was trying to 'defend' Moesia against the Bulgar invasion, he was in actuality
going on the offense against Bulgarian strongpoints in Moesia in order to force them out of the region. The same applies to the Byzantine losses during the reign of Theophilos.
Your point about the constant Byzantine defeats against the Bulgarians is a good point... you'd have expected the guys to have gotten wise to the Bulgarian tactic of 'ambush in mountain passes' sooner. Bear in mind that most of these ambushes occured after the end of a relatively-decent campaign (like Nikephoros I or Basil II), so fatigue and just general discipline breakdown might have contributed to those as well.
We also do have to note that 1) there was a period of around 100 years after Asparukh where Bulgaria didn't do much and was in fact constantly defeated and raided by monarchs such as Constantine V & Nikephoros I (the latter of course resulting in disaster, but it doesn't diminish the fact), 2) when Bulgaria was faced with two-front wars (against the Holy Roman Empire) Byzantium similarly managed to take land from them, and 3) even under its greatest monarchs Bulgaria didn't achieve its goals vs. the Byzantine Empire (which was to take Constantinople).
At any rate, I was given the impression that the vast majority of the emperors that campaigned in person were in fact militarily competent or had extensive military background. The ones that didn't, like Michael VII Doukas generally sent out a general or two in command of their armies instead.
The short answer is probably that the bad emperors commanding armies were the ones who lost the land. The amount of desertions/defections that Romanos IV sustained during the Manzikert campaign doesn't seem to be an indicator of good generalship to me, and the prevalence of Byzantine armies being 'ambushed' seems to indicate that there was a certain degree of tactical incompetence around at the highest levels.
Curiously enough, it seems that the competent generals were rarely in a hurry to re-annex all lost lands. Constantine V, for example, didn't bother formally annexing Bulgaria; John I didn't bother keeping the Levant when he pushed all the way to Nazareth. Perhaps they were only interested in the plunder; but you could also make an argument that the competent generals were not Alexander-types who wanted to conquer everything, only to keep what was strategically important for the Empire (like the border fortresses/gazi emirates of Tarsus, Melitene and Germanikeia).
So basically, you are saying that the ERE's army's performance was pretty average (or even pretty bad considering the amount of resources,numbers and training they are getting) against it's enemies overall because it's the jack of all trades and master of none?
I wouldn't put it that way; it's more like the Byzantine army can't be the master of
all trades - it has a doctrine tailored to a specific enemy and a specific way of fighting; and anything that doesn't line up with that is likely to go terribly wrong, especially in the hands of incompetents. Training and tactics need time to adapt, and in the course of transition Byzantium generally performed poorly. But then again, so do most nations in that situation.
As an example, the
Strategikon, with its emphasis on Roman-esque fortified camps and sources of food and water placed safely in the army's rear, seems to be written more for Sassanid columns (or large Arab armies), rather than the light skirmishing and maneuver of the early Arab invasions. The
Tactica of Leo VI was written with the expectation of massive Caliphate armies invading on multiple fronts with the intent of capturing major cities, not for fractured emirates that were content with small-scale raiding with occasional spurts of activity.
Praecepta Militaria was written in light of Nikephoros II Phocas' experiences against the gazi emirates on the Anatolian border who raided for booty, not against Turkish hordes who invaded and then stayed.
So in conclusion could Byzantium have done better? Sure. But to say it
constantly performed poorly, even with good generals and good armies, is a bit of an overstatement.