Depending on the specific mission, either every plane counts for the stacking penalty, or else every plane after the first counts. That means, on some missions, sending 3 planes generates a 30% penalty, and other missions will generate a 20% penalty.
There is always a tradeoff between per-plane efficiency and total firepower. With a 10% per plane penalty, the total firepower can be maximized at around 5 aircraft, but the 4th doesn't add all that much, and the 5th adds even less. The numbers are as follows:
1 plane @ 10% penalty = 1 x 0.9 = 0.9 total firepower
2 planes @ 20% penalty = 2 x 0.8 = 1.6 total firepower
3 planes @ 30% penalty = 3 x 0.7 = 2.1 total firepower
4 planes @ 40% penalty = 4 x 0.6 = 2.4 total firepower
5 planes @ 50% penalty = 5 x 0.5 = 2.5 total firepower
6 planes @ 60% penalty = 6 x 0.4 = 2.4 total firepower
Missions where the first plane is not counted for stacking will provide the same maximum of 3.0 firepower at both 5 and 6 (5 x 0.6 versus 6 x 0.5), and decline from there.
Beyond that, the total firepower continues to decline until you hit the 90% penalty cap, after which you can keep adding aircraft at 10% effectiveness to create a "doomstack":
20 planes @ 90% penalty = 20 x 0.1 = 2.0
30 planes @ 90% penalty = 30 x 0.1 = 3.0
40 planes @ 90% penalty = 40 x 0.1 = 4.0
...and so on.
Personally, I'd rather run two groups of 5 planes, one attacking by day and one by night. Note that in the beginning of the war, when air units are at full strength and organization, running 6 to a group may have marginally lower firepower, but will distribute damage a lot more widely, easily "outlasting" a group of 3-4 planes. You can accomplish this by running groups of 3, and sending two groups to the same location. Once you have air superiority, splitting them up into groups of 2 or 3 provides better per-plane efficiency, although 3 will be more survivable than 2 if they run into opposition.
