When I'm small and fresh, every polity around me - independent and subservient - makes me feel like I'm swimming with sharks. The AI will declare war if they're powerful enough, and will take advantage of me. I gave up a Semien game because the AI would either: a) declare war on me relentlessly, or b) wait for me to declare war someplace that would really help me out, then all declare war at once. It was brutal.
Once you get big and stable enough, your neighbours will have second thoughts about attacking you. They're computers, after all. Their decision is based on calculation. But staying stable is another matter: even if you know how to butter your vassals up perfectly, perfect micromanagement is impossible and you will get ambitious vassals, the occasional civil war, improperly raised children, a few decades of low opinion and inefficient demesne/vassal cap management and you might start looking tasty to your neighbours again.
And that's after nearly 3000 hours of the game.
So... yeah. I think that even though the game's difficulty depends a LOT on individual player skill and how stable your realm is, fighting your way to that point and keeping it that way both offer suitable difficulty and different sorts of challenges that cannot be discounted.