I really like this feature. I played about 100 hours so far, and it's frustrating just the right amount. Plus, I like that fact that if you chase them too far, they can very well turn back on you and make YOU retreat. Feels perfectly logical. You're in enemy territory, after all.
If you don't want it happening, take the Quality idea. That bonus to morale replenishment is better than forced march, imho.
Edit : what I'd love though is the ability to follow a fleeing army automatically. Checking directions and pausing each time does get old.