The AI isn't programmed to retreat once it engages. It will fight both land and naval battles to the end. Programming retreat is probably not very easy; AI programming in general is difficult, because seemingly-straightforward logic can sometimes lead to rather odd outcomes.
Anyways. I'm starting to wonder if somehow the capture mechanic for ships was broken in the latest patch. I had a rather odd experience by accident a few days ago, sadly on an Ironman game so I cannot replicate it easily:
My fleet of 12 early carracks engaged a 24-ish fleet of assorted ship types. Because I forgot to set the naval funding slider up before declaring war, my fleet's morale was very low. After my fleet's morale went to 0, my ships started disappearing from the battle information screen when they hit about 50% hull. My guess is that these ships were supposed to have been captured. But after the battle was over, all of my ships but 3 were reported sunk; the 3 survivors retreated to port.
The problem is not the lethality of fleet combat itself, but the lethality combined with the current naval AI, shipbuilding time/cost, and importance of naval power for controlling the profitable seaborne trade routes. It is currently much too easy to trap and destroy AI fleets, provided that the player does not do something silly in the process. And once the AI fleet is destroyed, it very rarely is able to build its forces back up to challenge the player's naval supremacy before the next conflict, and the cost of ships means that the loser of a major fleet engagement will fall behind in infrastructure development, at least in the first hundred years of the game.
If the AI could better control its fleets, and not patrol with four heavies when the player has twenty heavies two regions over, then the current system would be awesome. As it stands, naval battles mean the player either hunts down the AI fleets and annihilates them, or does so later.
