All the forumlas are provided on the wiki by bitmode:
https://hoi4.paradoxwikis.com/Naval_battle#Naval_strike
First there is target selection for Naval Strikes, which is much much higher for Capitals and Carriers than Screens. This is why you put AA on your Capitals.
Then for the targeted Ship it gets calculated if it gets a chance to shoot back and if yes how much planes they shoot down. So this shows a higher AA value on your capitals will shoot down lots of planes, especially against Kamikaze, since then you always hit the enemy wing, but with enough airplanes and to little own ships they just overwhelm you (1 Super Heavy AA Platform vs 1000 Ships).
If then some planes are still there for the attack, they attack you ship but the damage gets reduced by your fleet AA, which you can calculate yourself from scratch, or use the provided fleet AA value in a fleet battle. A value of around 50 Fleet AA gives around -33% damage reduction, of the surviving planes of the naval strike.
This is why you put AA on everything, an AA DDs are also a good idea. Rise your fleet AA cheaply by lots of cheap ships with AA, so you get a passive damage reduction. But also put good AA on your capitals, so they shoot down enemy planes and survive naval strike over naval strike. Also have lots of Capitals, so the enemy planes get spreaded from ship to ship, so no single ship is completly overwhelmed.
Lot's beeing relative though, since already 10 Capitals instead of 1 make a huge difference, but having much more makes it difficulty to screen against torpedos.