Another bug in 1.2? Analysis of "convoy_def_eff" and "convoy_raiding"

[MartinBG]

Another bug in 1.2? Analysis of "convoy_def_eff" and "convoy_raiding" - First results

The test results of convoy_def_eff against submarines are in post #28

Is convoy_def_eff broken? Look at post #48 and decide for yourself

How IMO convoy_def_eff and convoy_raiding (are supposed to) work in-game - post #57

What is "convoy_def_eff" supposed to do? Is this a modifier to escorts defending convoys? At least this is what I thinking of it before I made my tests.

Test configuration
1.Custom scenario. Pause and save at the start. All changes to modifiers are made in save-game. Tests are run from this save-game file.

British have 1000 transports and 100 escorts and must supply 15 divisions in Constanta from Izmit through South Black See.

Germany - 6 lvl 3 Subs, no leader.

Test - Convoy Riding for 5 months (150 days), 24h

(I'll add UK asw efficiency into my tests too, just in case )

Test 1
UK convoy_def_eff = 0.5500 (55%)
UK ASW = 0.7000 (70%)
GER convoy_raiding = 0.9000 (90%)
Result:
22/9 convoy/escorts sunk
No subs sunk, little damage

Test 2
UK convoy_def_eff = 0.5500 (55%)
UK ASW = 0.0000 (00%)
GER convoy_raiding = 0.9000 (90%)
Result:
21/5 convoy/escorts sunk
No subs sunk, little damage

Test 3
UK convoy_def_eff = 0.0000 (00%)
UK ASW = 0.0000 (00%)
GER convoy_raiding = 0.9000 (90%)
Result:
22/4 convoy/escorts sunk
No subs sunk, little damage

Test 4
UK convoy_def_eff = -50.0000 (-5000%)
UK ASW = -50.0000 (-5000%)
GER convoy_raiding = 0.9000 (90%)
Result:
21/5 convoy/escorts sunk
No subs sunk, little damage

Test 5
UK convoy_def_eff = 50.0000 (5000%)
UK ASW = 50.0000 (5000%)
GER convoy_raiding = 0.9000 (90%)
Result:
24/11 convoy/escorts sunk
No subs sunk, little damage

Any comments on this results?
For me at the best convoy_def_eff do nothing for better protection of your convoys.

Let's see if there is any way to make some damage to submarines using only convoy escorts and "playing" with convoy_def_eff and convoy_raiding values

Test 6
UK convoy_def_eff = 0.5500 (55%)
UK ASW = 00.7000 (70%)
GER convoy_raiding = 5.0000 (500%)
Result:
73/35 convoy/escorts sunk
3 subs sunk

Test 7
UK convoy_def_eff = 0.5500 (55%)
UK ASW = 5.0000 (500%)
GER convoy_raiding = 5.0000 (500%)
Result:
80/34 convoy/escorts sunk
3 subs sunk

Test 8
UK convoy_def_eff = 0.5500 (55%)
UK ASW = 0.7000 (500%)
GER convoy_raiding = -5.0000 (-500%)
Result:
0/0 convoy/escorts sunk
0 subs sunk, No convoys spoted!
(convoy_raiding isn't bugged )

Test 9
UK convoy_def_eff = 0.0000 (0%)
UK ASW = 0.0000 (0%)
GER convoy_raiding = 5.0000 (500%)
Result:
120/51 convoy/escorts sunk
4 subs sunk

Test 10
UK convoy_def_eff = 0.5500 (55%)
UK ASW = 0.0000 (0%)
GER convoy_raiding = 5.0000 (500%)
Result:
151/54 convoy/escorts sunk
3 subs sunk

Test 11
UK convoy_def_eff = 0.5500 (55%)
UK ASW = 25.0000 (2500%)
GER convoy_raiding = 5.0000 (500%)
Result:
140/56 convoy/escorts sunk
5 subs sunk

Test 12
UK convoy_def_eff = 5.0000 (500%)
UK ASW = 25.0000 (2500%)
GER convoy_raiding = 5.0000 (500%)
Result:
120/52 convoy/escorts sunk
2 subs sunk

Test 13
UK convoy_def_eff = -5.0000 (-500%)
UK ASW = 25.0000 (2500%)
GER convoy_raiding = 5.0000 (500%)
Result:
121/53 convoy/escorts sunk
4 subs sunk

Test 14
UK convoy_def_eff = 0.5500 (55%)
UK ASW = 0.0000 (0%)
GER convoy_raiding = 50.0000 (5000%)
Result:
177/61 convoy/escorts sunk
6 subs sunk in 120 days!!!


If anyone wants to run these or other tests by himself, I can upload my test scenario somewere to download.

Later I'll add results from Air convoy raiding to see is there any use of convoy_def_eff in this case.

 

[Gormadoc]

convoy_def_eff is used for convoy escorts only. ASW mission eff. is for regular naval units DD's CL's and soforth, it does nothing towards how convoy escorts perform.

To get a clean result you should only change one parameter at the time and keep all other parameters static. I tested this in arma 1.1 and got the result that higher convoy defence efficiency resulted in less transport sunk & more damage done to the subs.

I assume that UK has no other unit. Did you note how many convoy to well protected messages you get with the different convoy defence eff ?

 

[MartinBG]

To get a clean result you should only change one parameter at the time and keep all other parameters static.

That is what I done In Test 1-5 - change only to convoy_def_eff (as you mentioned ASW did not affect convoy defence) and the results are very close.

Did you note how many convoy to well protected messages you get with the different convoy defence eff ?

No, I didn't count the exact numbers, but there are less such messages when convoy_raiding is high (in my last test I get only 2 of them before my last sub get sunk)

Later I'll run these tests on 1.1 too and will post my observations.


Here are another test results:

Test 15
UK convoy_def_eff = 500.0000 (50000%) !!!
UK ASW = 0.0000 (0%)
GER convoy_raiding = 5.0000 (500%)
Result:
120/50 convoy/escorts sunk
2 subs sunk, No convoy to well protected messages

Test 16
UK convoy_def_eff = 1.0000 (100%) !!!
UK ASW = 0.0000 (0%)
GER convoy_raiding = 5.0000 (500%)
Result:
119/52 convoy/escorts sunk
3 subs sunk (and 1 with >5 Str.), No convoy to well protected messages

 

[MartinBG]

Here are the results of convoy bombing:

Test configuration

1.Custom scenario. Pause and save at the start. All changes to modifiers are made in save-game. Tests are run from this save-game file.



British have 1000 transports and 100 escorts and must supply 15 divisions in Constanta from Izmit through South Black See.

Germany - 4 lvl 2 NAV, no leader.

Test - Convoy Bombing for 3 months (90 days), 24h, static weather

GER convoy_air_raiding = 1.0000 (100%) in all tests

Test 1
UK convoy_def_eff = -1.0000 (-100%) 28/62 convoy/escorts sunk

Test 2
UK convoy_def_eff = 0.0000 (0%) 21/56 convoy/escorts sunk

Test 3
UK convoy_def_eff = 1.0000 (100%)23/68 convoy/escorts sunk

Test 4
UK convoy_def_eff = 5.0000 (500%) 19/52 convoy/escorts sunk

 

[henryjai]

i think you need to collect 10 data for each set to get a more convincing result.

 

[unmerged(57311)]

i think you need to collect 10 data for each set to get a more convincing result.

Yes, as my physics teacher used to say: 'One experiment is not experiment!'

 

[semaphore]

i think you need to collect 10 data for each set to get a more convincing result.

Actually, he needs 10 tries for each set before the results are even statistically significant. I also think that ridiculously large numbers might be breaking the game engine.

 

[Gormadoc]

No, I didn't count the exact numbers, but there are less such messages when convoy_raiding is high (in my last test I get only 2 of them before my last sub get sunk)

That is what is expected convoy_raiding & convoy_defence works relatively to each other. If high convoy defence and low convoy raiding you should get more convoy to well protected messages.

As mentioned above you should start with values within the normal range of mission efficiency's only if these do not deliver any results should you go outside the range. When i tested it in 1.1 there was signifikant difference between 50% ME, 100% & 150% ME.

I still have my test scenario so i will run some test on this then we can compare numbers.

 

[ForzaA]

Just a hypothesis (I've never quite understood the assignments for convoys ) ..but maybe the AI just sends fewer escorts to escort a particular supply line if it has a higher convoy_defense ? Obviously wouldn't help one iota in this situation, but might help an empire stretched thin that needs escorts everywhere..

 

[MartinBG]

You're all right to criticize my tests and results so far. What I post here till now is very far away from the right to call itself analysis so I'll have to fix that... or just to change that Thread name :rofl:

(Posted above results are just quick runs to see is there any significant influence of convoy_def_eff and convoy_raiding over convoy raiding results and to help me to decide which value ranges and test configuration to use in the real tests. The second propose of my posts (so far) and this thread name was to drag into this discusion more people to help me to precise these tests configuration and to analize results from them together. Help in testing will be also nice, because I already spent a night doing this :wacko: )


My current plans for the real test are:

I'll use described above test configuration:
Custom scenario, constant weather, no leader(s), 150 (?) days / 24h mission, no retreats to the port.

Run 16 series of 5-12 tests each:

Test series 1:
convoy_def_eff = 0%
convoy_raiding = 0%

Test series 2:
convoy_def_eff = 50%
convoy_raiding = 0%

Test series 3:
convoy_def_eff = 100%
convoy_raiding = 0%

Test series 4:
convoy_def_eff = 150%
convoy_raiding = 0%

Test series 5:
convoy_def_eff = 0%
convoy_raiding = 50%

Test series 6:
convoy_def_eff = 50%
convoy_raiding = 50%

Test series 7:
convoy_def_eff = 100%
convoy_raiding = 50%

Test series 8:
convoy_def_eff = 150%
convoy_raiding = 50%

Test series 9:
convoy_def_eff = 0%
convoy_raiding = 100%

Test series 10:
convoy_def_eff = 50%
convoy_raiding = 100%

Test series 11:
convoy_def_eff = 100%
convoy_raiding = 100%

Test series 12:
convoy_def_eff = 150%
convoy_raiding = 100%

Test series 13:
convoy_def_eff = 0%
convoy_raiding = 150%

Test series 14:
convoy_def_eff = 50%
convoy_raiding = 150%

Test series 15:
convoy_def_eff = 100%
convoy_raiding = 150%

Test series 16:
convoy_def_eff = 150%
convoy_raiding = 150%


After each run I'll note how many convoys/escorts are sunk, how many " to well protected convoy" messages I get, how many of my subs was sunk and what was damage taken by them.

If no one joins me in this task I'll most probaly make only 5-7 tests and will eliminate from calculation 1-2 of them if their results are too off the others. The rest will be sumarized and divided by their count.

I'm not sure how many subs to use in these tests. May be 6 are too powerful for this test configuration?

Any comments?

 

[MartinBG]

Here are my test scenario files - download (12.8KB), for anyone who decides to take part in the testing or just to check them for errors.

 

[semaphore]

After each run I'll note how many convoys/escorts are sunk, how many " to well protected convoy" messages I get, how many of my subs was sunk and what was damage taken by them.

If no one joins me in this task I'll most probaly make only 5-7 tests and will eliminate from calculation 1-2 of them if their results are too off the others. The rest will be sumarized and divided by their count.

I'm not sure how many subs to use in these tests. May be 6 are too powerful for this test configuration?

Any comments?

I will advise that you do just two tests:

Test One:
convoy_def_eff = 25%
convoy_raiding = 50%

Test Two:
convoy_def_eff = 75%
convoy_raiding = 50%

And instead focus on getting as many data sets as possible for each. 20 tries for each seem like a good number to aim for (anything less than 10 is useless from a statistical point of view). Don't eliminate anything, even if they look completely off the charts.

You can then perform a t-test on the results to see if there are any evidence that a 50% difference in convoy_def_eff makes a difference in convoy defence.

 

[Gormadoc]

From just one test i can definitely say AI's uses of subs is markedly changed.

The AI sent a 19 sub stack on convoy raiding never saw any thing like that in arma 1.1. In 1.1 it always parked 1 sub in one sea province while it had around 5-10 on convoy raiding depending on settings.

This priliminary observation looks very promising indeed.

 

[Caesar_Augustus]

From just one test i can definitely say AI's uses of subs is markedly changed.

The AI sent a 19 sub stack on convoy raiding never saw any thing like that in arma 1.1. In 1.1 it always parked 1 sub in one sea province while it had around 5-10 on convoy raiding depending on settings.

This priliminary observation looks very promising indeed.

Great news from Gormadoc himself! I wonder... the AI behaviour depends not only on the AI files themselves, but also on the HoI2.exe, correct? As far as I've seen, nobody is advocating returning to 1.1. Glad to hear things are indeed looking up!

 

[DrinkingHeavily]

I TA'd design of experiments during my masters.

Could someone post the factors and ranges that you're trying to experiment over? I read this thread 2x and still can't figure that out.

experimenting with one factor changing at a time, known as OFAT, doesn't allow you to capture second-order effects with the different variables... which may in fact be more significant in the model than the factor you're changing. (This may also mean nothing, but you're only capturing about 60% of the variance with a one-factor model.)

Or PM me if you'd like.

 

[unmerged(52476)]

I TA'd design of experiments during my masters.

Could someone post the factors and ranges that you're trying to experiment over? I read this thread 2x and still can't figure that out.

experimenting with one factor changing at a time, known as OFAT, doesn't allow you to capture second-order effects with the different variables... which may in fact be more significant in the model than the factor you're changing. (This may also mean nothing, but you're only capturing about 60% of the variance with a one-factor model.)

Or PM me if you'd like.

The original question was if convoy_def_eff or convoy_raiding are broken. So it's a good idea to figure out first (by changing only one of these parameters while everything else is constant) what exactly these parameters do. Once you know that, you can worry about second-order effects (which indeed might be interesting).

 

[MartinBG]

If I'm not wrong convoy raiding success depends on following factors (and combination of them):

- Convoy attack ability of used unit - in my test submarines or NAV. This value is constant
- convoy_raiding modifier - this is one of the values I'll exeriment with. In-game it is about 40~100% (have to check that when in home)
- Leader skill level (Experience) - I'll run my tests without leaders in command (they gain Exp. during convoy raiding and will mess my result data)
- Leader's traits may have impact on convoy raiding too (Sea wolf as example) - no leaders in my test
- Minister modifiers - I'm not sure if there are any
- weather - every unit has its own weather modifier. I'll use constant weather to disable its effect.
- time of the day (day/night) - I'll try to remove any day/night modifiers to tested units (SUBs and NAVs) during my tests.
- convoy_def_eff - this modifier is (probably) only for convoy escors and is supposed improve their ability to scare of or to sunk enemy subs. This is the second value I'll change in my tests. In-game it is about 40-100% (have to check that too)
- war luck - there is some random value which may affect test results and because of it I'll run every test more then one time.

Did I miss something? :wacko:

EDIT:
I forgot (at least) one thing o

- Difficulty level - there are bonus on V.E. and E. and penalty on H. and V.H. . I'll run my tests on N/N

EDIT 2:

- attacking unit expirience - expirience adds bonus in all battles, so it may hame impact on convoy raiding too. In my tests all units start with 0 Exp. and didn't get much more during convoy raiding because of relative short test time for every test run.

- may be some of the following modifiers affects convoy raiding too. I equalize them for both Germany and England in my tests:

    intelligence = 75.0000 
    enemy_intelligence = 20.0000 
    army_detection = 50.0000 
    enemy_army_detection = 50.0000 
    land_surprise = 1.0000 
    naval_surprise = 1.0000 
    air_surprise = 1.0000

 

[Caesar_Augustus]

I think you've pretty much covered the variables, MartinBG. Thanks ever so much for the time and effort your're putting into this... we are anxiously awaiting good results!

 

[Gormadoc]

If I'm not wrong convoy raiding success depends on following factors (and combination of them):

- Convoy attack ability of used unit - in my test submarines or NAV. This value is constant
- convoy_raiding modifier - this is one of the values I'll exeriment with. In-game it is about 40~100% (have to check that when in home)
- Leader skill level (Experience) - I'll run my tests without leaders in command (they gain Exp. during convoy raiding and will mess my result data)
- Leader's traits may have impact on convoy raiding too (Sea wolf as example) - no leaders in my test
- Minister modifiers - I'm not sure if there are any
- weather - every unit has its own weather modifier. I'll use constant weather to disable its effect.
- time of the day (day/night) - I'll try to remove any day/night modifiers to tested units (SUBs and NAVs) during my tests.
- convoy_def_eff - this modifier is (probably) only for convoy escors and is supposed improve their ability to scare of or to sunk enemy subs. This is the second value I'll change in my tests. In-game it is about 40-100% (have to check that too)
- war luck - there is some random value which may affect test results and because of it I'll run every test more then one time.

Did I miss something? :wacko:

You dont need to run so many test as long as the ones you run are long. In essence every hour of the game is a seperate test. Since all the calculations about detecting convoys and shoting at them are done every hour. I suggest you give more than 100 escorts, to make sure the AI has as many escorts as it would like to use give it 1000 escorts.

 

[MartinBG]

Last chance to comment before I begin my tests

I just managed to disable all weather/time modifiers to my test units.

Now i'm checking which ministers to use so they do as little effect as possible in my tests. For England and Germany I think to use default ones from 1941 Grand campain. Are they OK?

If I'm right about convoy raiding success factors, then I'll have only 1 undefined value in my tests - "War luck" - with all others disabled or constants, so I'm optimistic about "purity" of my test environment.

Other changes:
I gave 100000 convoys and 10000 escorts to England

How many subs to use? 3, 6 or more.

Gormadoc, very nice idea to run few but longer tests! It's OK for me as long as no one complains about it. So is anyone against this change?
Let's say 3 tests x 1(or 2) game year(s) long?


I equalized the following values for both Ger. and UK just in case they have any impact on convoy raiding:

    intelligence = 75.0000 
    enemy_intelligence = 20.0000 
    army_detection = 50.0000 
    enemy_army_detection = 50.0000 
    land_surprise = 1.0000 
    naval_surprise = 1.0000 
    air_surprise = 1.0000