Difficulty levels

[Castellon]

I don't think 5 was ever restricted to Very Hard.
Also difficulty level effects your "luck"

 

[Castellon]

And how about the aggression levels? What, specifically, changes?

Aggression level determines how likely the AI is to declare war.
So it may not produce a harder game by putting this to the highest level since then the AI will burn itself out. Making it easier for the human player to walk over it.
That is one of the reasons it is seperated from the Difficulty setting.