Game runs well, but lags when units start moving. What's wrong?

I'm running the game on max graphics on a huge map with 6 players. We are 250 turns in and the game lags when units react to commands. For instance when I order a cog to unload on shore, the ship goes ashore, the game locks for about a second and then the unit appears in place of the ship. Also when I cast a spell that has an area of effect it will lock up similarly halfway through the animation for a second before continuing. Otherwise the game is still running completely fluidly.

I have
AMD Athlon II X4 645 3,1GHz
AMD HD7770 1Gb DDR5
4GB DDR3 1333Mhz