HOI4 Dev Diary - Patch 1.3.3 Update #2

[that-guy-is-it]

More optimization related diaries please. They are interesting and I think we all would really like to know how progress in that front is going...

 

[Riekopo]

Great improvements all around.

 

[208]

1. yes, its not a global penalty its an increasing penalty for each factory lacking resources effectively (but depends on how many resources each factory uses too, so goes faster for inf weapons 3 than 1)
2. Lets use an example and maybe it becomes clearer. My nation has a total of 9 steel. each factory uses 3 so to makeit extra clear I used up all steel on the first line. Second line gets no steel:

Thats 5% penalty by each missing resource (factory wants 3, we lack 3)

if we switch to Weapons 2 which needs only 2 steel per factory there is still no steel, but penalty is smaller:


if we add more factories (more need) the penalty goes up, but doesnt affect output of stuff coming before it. so:
Clearer now?

So now the solution, once you start running short on resources, is to split all your "excess" production into single-factory lines that will only have a relatively small penalty?

 

[Denkt]

As I said in my example, which you should read, is that my heavy fighters were highly upgraded AND the Ai was sending in unescorted unupgraded strat bombers. The situation would be even more unfavorable if I was against a decent player who would've used fighters as escorts, and didnt spread out their strat bombers, AND used their experience to upgrade the fighters, AND then also chewed through my poor heavy fighters even worse than the strat bombers.

600 strategic bombers for 400 heavy fighters is a very good deal in terms of production losses. If you want better result you should research air doctrine, use more fighters, build radars and such.

 

[Ghost73]

So now the solution, once you start running short on resources, is to split all your "excess" production into single-factory lines that will only have a relatively small penalty?

I came to the same conclusion too.
HoI4 team really needs to take another good look at the math behind this.

 

[208]

There is already a key who helps to put a project on top. Currently I am not on a game, so I could tell for sure, but If I use SHIFT or CTRL (not sure, which one) with a left mouse click to project jumps on top of the list. Maybe you did something, that it stays on top, no matter, if I use the named key combination again. Otherwise, it's not a feature of 1.3.3 and you've just forget to tell about it

That key you mention is used in the production interface for rearranging stuff already in the queue. I think the new shortcut Podcat mentioned is for when you actually add new stuff to the queue, while holding CTRL it will be added to the front instead of the end - which saves a step going in afterwards to rearrange the queue.

 

[Colonel Chris]

Especially in SP-games the US should favor to lend-lease weapons to the USSR, like they did during Barbarossa.

 

[Denkt]

So now the solution, once you start running short on resources, is to split all your "excess" production into single-factory lines that will only have a relatively small penalty?

The penalty probably work on a factory basis not on a line basis.

So if the first factory suffer 15% penalty the second will suffer 15% +15% penalty = 30% and the third will suffer 45% penalty so making 3 single lines or one line will not matter.

 

[fabius]

If I understand it you get a penalty to all lines that lack the particular resource no matter their priority and that penalty can completely stop your production if you go to negative.

Popcat clarified this up. This change will be interesting, looking forward to my next Germany game.

 

[hapster]

Beta patch this friday pretty please? I'll be good!

 

[MSIOCGenie]

A really neat QoL improvement would be to add shift-click in the army tab. Say that you start as Russia and want to select all infantry divisions. Right now you need to manually ctrl-click over a hundred units. Instead you should just click on unit 1, hold shift, click on the last unit, and all units in between will also be selected.

I really hope this isn't already in there, because that would mean I've clicked thousands of units in vain..

 

[Sleight of Hand]

A really neat QoL improvement would be to add shift-click in the army tab. Say that you start as Russia and want to select all infantry divisions. Right now you need to manually ctrl-click over a hundred units. Instead you should just click on unit 1, hold shift, click on the last unit, and all units in between will also be selected.

I really hope this isn't already in there, because that would mean I've clicked thousands of units in vain..

Yes!!!!

I keep doing this anyway, assuming it will do just as you say, but it never does.

 

[Jorlem]

In this screenshot, is the penalty from lack of resources being neutralized by the bonuses from Limited Exports and Dispersed Industry 1?

There is one last that is my favorite. In our Clausewitz engine, there was an old code loop that nobody ever dared to touch. It was processing all the user interface elements in a "flat manner" instead of the "tree hierarchy". This means, that the more windows and buttons we add to the game, this loop was heavier and heavier. And the windows didn't even had to be shown for it to slow down the game. We always knew about this infamous spot, however reworking it without breaking all the interfaces was nearly impossible. Until now. I found the way! Previously that code loop had ~120 000 passes in each frame, now it's under 700, processing only the necessary interface elements. By that I mean, when you are looking at the technology trees, we are not processing through the hidden production windows and buttons, etc.

Here's hoping this can make it into the other games as well.

Thats 5% penalty by each missing resource (factory wants 3, we lack 3)

if we switch to Weapons 2 which needs only 2 steel per factory there is still no steel, but penalty is smaller:

if we add more factories (more need) the penalty goes up, but doesnt affect output of stuff coming before it. so:

I'm somewhat confused how that second line is able to produce anything at all in any of the screenshots. The lack of resources appears to be pushing the production efficiency below zero, so it seems like nothing should be made.

 

[Ghost73]

The penalty probably work on a factory basis not on a line basis.

So if the first factory suffer 15% penalty the second will suffer 15% +15% penalty = 30% and the third will suffer 45% penalty so making 3 single lines or one line will not matter.

Looking at the tooltip it appears the penalty is applied to the whole line.

Thus the best way to reduce the penalty is to make a whole bunch of single factory lines so each receives the minimum penalty instead of stacking the factories. I don't think this is what the devs were intending.

 

[Sanny]

Hope you are feeling better @SteelVolt , really appreciate all the work you're putting into the AI.

 

[ltccone]

With the factory changes I'm concerned about AI Germany's fighter production. The only rubber Germany has after WW2 starts is from the synthetic oil refineries. The UK and USAF always have a massive fighter advantage and I'm afraid this change will make it worse.

 

[csgdzjtpxj]

Hi everyone and welcome to the last update on the 1.3.3 patch as we are now getting into final testing of it. This is going to be a big diary with guest writing both from @SteelVolt (AI) and @TomaszKowalczyk (tech lead) as well. Grab a cup of the blackest coffee and enjoy!

Industry - resource lack changes
To continue last weeks theme on industry we have a final change: currently production lines can get a max penalty of -80% due to lack of resources. That effectively mean that a resource lack can be combated by just having more available factories. We felt this was both unrealistic and was making resources less important than we wanted for big nations. Penalties due to lack of resource are now instead calculated by each missing resource over multiple lines and steadily get higher until they can actually get -100% penalty.
View attachment 241049


Quality of life!
As normal with patches its not just bug fixes and balance, we've done a bunch of quality of life improvements, what while small do feel big when playing. Lets go:

• Do you need to start a building construction with top priority? No problem. Now you can CTRL+Click when constructing, so it goes directly to the top of your constructions.
• Now something that should really be there long ago. When selecting a division, it now scrolls the list of the divisions to show them.View attachment 241054
• Did you have problems to find all your units in a crowdy war? Now there is a button (right bottom side of the screen) with hotkey 'M' that hides ALL counters except your units.
  View attachment 241052
• Because its sometimes hard to plan ahead over areas where you have little intel we have also added a toggle button to turn off the fog of war effect should you need to to better see terrain etc.
• We invented a revolutionary new thing which we have decided to call a "loading progress bar"View attachment 241050
• Do you also have that one friend that is always lagging behind in multiplayer? Got annoyed by the "player lagging behind" message popping up in your face? Now it's an alert. The tooltip explains who is lagging behind and how much. Of course it also automatically disappears when that player catch up.
• To be consistent with air wings, any unassigned divisions now also get an alert. It's super useful, when for example, a friend is sending you expeditionary forces, and you have no idea where the frell they are. Now just click on the alert to toggle through the unassigned divisions!View attachment 241051

AI Update!
Several people on the forums have asked for more in-depth details on AI improvements like the ones we did for the Together For Victory release, and you have been heard. Here is an update from @SteelVolt:

Greetings everyone! Taking a short break from otherwise working super hard on improving the Hearts of Iron 4 AI (not counting last week when a flu virus knocked me out), I am once again here to bring you some news about what is going on in my field.

As last time I did these, I thought I would start out with a story from the trenches. This one happened during the current development cycle. After having done some work in several areas of the AI, we noticed in our hands off runs that Germany was fairly consistently defeating England. Getting past the Royal Navy is no small feat, but we initially thought that a Germany that had beat Soviet (which they had started doing at that time) would have industry enough to counter this with airplanes. Turns out that was a bit optimistic. What was really happening was that the naval AI had an issue that made it extremely difficult for it to change regions for assigned fleets, and England had already sent most of their fleet away.

These are the related entries in the changelog:

• Improved naval AI region evaluation and made it better at reacting more efficiently to changing situations.
• Naval AI will now attempt to use strike fleets to aid in coastal land combats if they are not on a mission to aid in a naval combat.

The second one being an improvement to how the AI uses its strike fleet (which is the fleets it used not to guard or hunt, but to destroy the enemies main fleet)

While the coming patch does not cover all AI improvements in the works, there are several reasons to feel excited about the AI for the next update, and I thought that I would highlight some of the more interesting ones here.

First of all there is the infamous unit spam:

• Made AI better at calculating supply use when deciding how many divisions it can field.
• Added equipment_stockpile ai strategy that lets you affect how much stockpile ai feels it needs
• All nations with many divisions will now attempt to accumulate bigger stockpiles for safety

There were a couple of minor bugs, but also some room for improvements, in the code where the AI estimates how many divisions it thinks it can afford to support. This seems way better now. We also made the AI more careful about its stockpile of equipment as well as making it aim for bigger ones for nations who already had a lot of divisions.

• Improved deployment AI to manage lines better and improved calculation of desired number of divisions.

Partially connected to the previous point; the AI was way too eager to put a major number of lines working on units but then under certain circumstances just deleting them and starting over. All of this has been given an overhaul, and the AI now acts more moderately both when planning and when reacting to changes.

• AI will now remove deployment lines for types it has reached its wanted number for.

Really what it says. Before, the AI would keep producing the units it had started working on, which was very inflexible for changing situations.

Moving on, I am surprised this one was not caught earlier:

• Fixed a serious issue where AI would never counter resistance with area defense orders, completely ignoring conquered territory.

While there had been mentions that the AI could probably do better when it came to countering resistance, it really turned out that it never did this at all. This was due to a code mistake in a single if-statement that made the AI exclusively considered OWNED territory, as opposed to CONTROLLED.

To weather the tightening up and changes we have done to industry has also meant changes for the AI in those areas:

• Improved logic for when and whom AI should be lend leasing
• AI is now able to send old equipment as lend lease not just its newest
• AI can now cancel lend leases it can no longer afford or with nations that no longer make sense
• AI should no longer be so happy on naval construction over military when it had also alot of civilian industry

The first couple are pretty straight forward: you should be seeing more effective lend leasing now. The last one was an issue affecting many nations (but USA most of all) where having a lot of civilian industry made it think it needed much less military industry in favor of naval construction.

And lastly, the AI is now a lot better at handling safe fronts. It used to sometimes assign significant numbers of units to fronts that posed no threat. Some of these were to countries fighting a common enemy, even when they were winning against said enemy, and others were just...not a threat in any way. Bottom line: more of the AIs units should be making an actual difference now

Lets close with a screenshot that makes me feel pretty happy: Japan not declaring war on the Soviet Union until it has secured the chinese front properly.
View attachment 241055


Performance and Optimizations
This part is for those interested in code details and technical stuff. Just a warning! We figured fans might be interested in more technical details as well as getting some insight in what the job is like for coders. So let us know if this was interesting or made your brains melt! Now I'm gonna hand the word over to our project tech lead @TomaszKowalczyk:

Hi everyone, I'm going to be talking about all the optimization work I did for the patch. We are of course planning to improve it further more in the next patches, however in 1.3.3 the overall speed up is about 14-18%. This may look like not too much, however the main focus was to solve performance "spikes". Those moments when the game literally freezes at a certain moments (usually every 24th hour, when the next day begins). At the very late game date, they were even more annoying than ever. Those I've improved a lot. I was comparing a certain very late save-game, with 1.3.2 and 1.3.3. The first daily tick after hit unpause, in 1.3.2 it took about 4 seconds to advance. In 1.3.3 it was less than ~0.4 sec.

Usually we don't share the nitty-gritty technical details, but this time I'm going to write what and why was slowing down the game:

• I reworked a bit of the peace conference code that was computing which country can get what. In a gigantic war, where a major country with lots of territory, have lost, pressing SKIP and SKIP and SKIP all over again, was making the game running slower and slower, until almost freeze. This exists in all peace conferences. Just in small wars, it wasn't that badly noticeable. The problem was in a badly nested code loop. Rewriting them worked like a charm.
• Every time the player click on a navy, and assign the mission in some region(s), the game is doing a "naval path-find" and sends the fleet to go to the necessary area. There was AI logic that wanted to be sure that their fleets were doing the right missions, and it turns out it was spamming "go there", "go there", "go there", while obviously it was enough to tell once. Any kind of path-finding is a pretty complicated and performance heavy thing, but luckily adding a single line of code to stop that solved the problem with the spam!
• The improvements to the supply system was what had the biggest impact.
  
  One problem was that plenty of game elements are relying on information about "supplies supported" vs "supplies required". Which is, how many units we can fit in the area before we get the penalty. Every piece of the code that was asking the region "what is the supported and required values?", was recomputing those numbers from scratch. It was asking all units in the area, how high is your supply usage, and how much the region supports (another complicated formula). And it was done hundreds of time by various game elements, in the same computing frame. Instead, I made it so it computes that information only once, and store it in "cached memory". All the game elements now fetch the data directly from cache, which costs almost no performance at all.
  
  Another problem in the supply systems was the large amount of units. In the very late game, having lots of divisions is a true problem, as it requires lots of computing power. This time it almost killed the supply system. I did plenty of tricks here to improve it. I had to rewrite the way we compute the supplies consumption in each area. Instead of looking at each province and iterating through all the units, asking them "what is your supply consumption value?", I do it from the other side. Every hour, each unit "reports" the supply consumption at it's location, bumping up the overall value in the region. Sounds like same thing, but not really from the technical side. Thanks to that, I could completely remove the super heavy code loop from the supply system. To be honest, that was the nicest optimization ever. How to optimize the function? Remove it
  
  There was so many improvements in the supply system that it doesn't really make sense to list them all, so instead as a summary I will say, that previously the supply system was taking about 65% of the computing time in the daily "spikes". Now it's under 6%.

• Another big optimization to daily "spikes" involved the part of the AI playing around with the division designer to achieve its desired templates. The fix was quite simple here. I've discovered that whenever we try add/remove/replace the division in any slot, the overall template stats are recomputed. That's heavy. Well, maybe not that much heavy, but they became heavy when all countries in the world try almost all possible combinations, in a single frame pass. Turns out, it was enough to refresh the template stats only once after the AI is done filling all the slots.
• There was also bunch of optimizations in: AI planning naval invasions, AI code responsible for spreading units along the front, the fleets calculating the bonuses from the radars, convoy routes checking which way to go if the certain channels are blocked, and plenty of other hard to explain improvements.
• There is one last that is my favorite. In our Clausewitz engine, there was an old code loop that nobody ever dared to touch. It was processing all the user interface elements in a "flat manner" instead of the "tree hierarchy". This means, that the more windows and buttons we add to the game, this loop was heavier and heavier. And the windows didn't even had to be shown for it to slow down the game. We always knew about this infamous spot, however reworking it without breaking all the interfaces was nearly impossible. Until now. I found the way! Previously that code loop had ~120 000 passes in each frame, now it's under 700, processing only the necessary interface elements. By that I mean, when you are looking at the technology trees, we are not processing through the hidden production windows and buttons, etc.

As I previously said, we are still working on late-game performance. I've spent only 1,5 week on it so far, and we already get quite much of it so you can expect us to keep doing improvements like this in future patches.

Don't forget to tune in at 16:00CET on www.twitch.tv/paradoxinteractive today where me and Daniel will be continuing our campaign in Kaiserreich for World War Wednesday!

wow- quite a substantial number of optimization work- great job. Can't speak for everyone, but I find these types of updates interesting.

And side note, the other Clauswitz engine users owe you a beer for the interface update!

 

[Sleight of Hand]

With the factory changes I'm concerned about AI Germany's fighter production. The only rubber Germany has after WW2 starts is from the synthetic oil refineries. The UK and USAF always have a massive fighter advantage and I'm afraid this change will make it worse.

A related point...

It is true that the Axis powers are often short of rubber and oil, and unless Japan does well in East Asia (and is able to trade with Germany and Italy) they are unlikely to get any... the best bet is for Germany to take the Caucasus.

I think a good sideways solution to this would be to increase the amount of rubber in the Belgian Congo -- this has the twin effect of increasing the amount of rubber the Axis can get (independent of what Japan is up to) and give Italy in particular a more defined goal in its African campaign.

Just a thought... @podcat ?

 

[Aldahen]

1. yes, its not a global penalty its an increasing penalty for each factory lacking resources effectively (but depends on how many resources each factory uses too, so goes faster for inf weapons 3 than 1)
2. Lets use an example and maybe it becomes clearer. My nation has a total of 9 steel. each factory uses 3 so to makeit extra clear I used up all steel on the first line. Second line gets no steel:
View attachment 241065
Thats 5% penalty by each missing resource (factory wants 3, we lack 3)

if we switch to Weapons 2 which needs only 2 steel per factory there is still no steel, but penalty is smaller:
View attachment 241066

if we add more factories (more need) the penalty goes up, but doesnt affect output of stuff coming before it. so:
View attachment 241067
Clearer now?

3. only if its lines before it (higher priority) that "took" the steel. So priority matters just like the old system worked
4. it used the biggest bottleneck as limit, so if you have 2 of 3 resources you still get the penalty. It was too easy to ignore oil etc since most of those lines used so much steel before.

Thanks for the explanation. So, if I understand correctly it is simply:
Penalty = number_of_missing_resources X 5%

What still is not evident is does resource types factor in it? I.e. is
number_of_missing_resources = the number of missing resource for the resource with the biggest deficit for this line
OR
number_of_missing_resources = the total number of all missing resources for this line

Also I don't completely get why the penalty in the 3rd picture is only -30% and not -45% as those 3 factories are missing now 9 steel?

Further more....unless I'm missing some info, the system is exploitable with added micro management.
In the last example there you added more factories to the line and the penalty went from -15% to -30%. If you would have instead created 2 new lines with one factory each the penalty would have remained at -15%?

 

[Gamer_1745]

Great improvements!!!