• We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.

HOI4 Dev Diary - Tech bugaloo II - Dragonslaying

Hi guys! Today's Diary is going to be a bit of a short one as I am away at a conference (it has free breakfast! Two most magic words!)

Last week we celebrated HOI4’s 3 year anniversary and released 1.7 ‘Hydra’ along with Radio Pack and Axis Armor. I hope you enjoyed them :)

After the weekend we looked at our telemetry data after 1.7 released and noticed that the amount of multiplayer out of syncs were more common than before. This indicates that we introduced a new OOS problem. While HOI4 has resync and hotjoin its still pretty annoying when you out of sync so we are currently investigating this for a small hotfix patch (1.7.1). Out of syncs can be really tricky to find and nail down so no definite ETA yet on when the patch is ready as we are still hunting. But we acquired a solid lead on the problem just yesterday, and we're currently working out a good solution.

Technical section (warning!): What is an Out of Sync?
For those interested what an out of sync is I figured I’d dig into it a bit. Feel free to skip this if you are a... normal human being, I guess ;D

An Out of Sync (OOS) happens when the host and clients in a multiplayer game start acting differently. This could for example be something like a battle ending in the favor of Germany on one of the computers and in the favor of Soviets on the other. Usually though its nothing that big as the state difference is usually spotted earlier at say one of those units having a 1% higher organization than the other or the like. Once it happens everyone’s experience will very quickly start diverging, so we stop and alert the players. At this point the host can click a ‘Resync’ button to bring the game back into sync. Resyncing will reset the state of the game, send the current host state over as a savegame and have everyone load that, and then things can resume.

So what can cause an OOS? This is where it gets tricky, and its pretty much always a new reason when a problem appears. Some good candidates are multiplayer between different platforms because underlying code libraries can behave differently in some cases. Other common reasons are multithreading. We thread a lot of our code, yet to stay in sync we must assure that events still happen in the same order on all machines in the end if they affect the world. There can also be issues like touching illegal memory or the like that can alter the game state in unplanned ways (or crash… but those are easy to spot and fix!).

Finding and fixing an OOS can be a long process simply because they are often quite rare occurances and it usually takes many steps and iteration to home in on exactly what it is. To find them we run multiplayer tests with QA with special settings that spit out giant log files (which makes everything horribly slow generally) and once and OOS happens we compare log files and savegames to see what differs. This will usually give us an area to start looking at. Lets say for example that you have a unit’s org being different. This could be due to many reasons - battle damage, weather, bad supply etc. So we add more logging to the relevant code areas and do another test. Hopefully this will tell us which of our guesses was right, and we repeat again with more logs and more details for this area. Of course the most fun-to-find OOS errors disappears when you add logging and framerate slows ;P

Some of this can be done automatically over night as well if the problem is unrelated to players, but this is often not the case.

Once found and fixed this is usually the stage we make an open beta patch to verify that it is indeed fixed, or if we deem it sure that we found the problem we will go straight to regular patch. Speaking of beta patches, thanks for the help testing 1.7! we got something like 30k game sessions of testing on the weekend before the final build which was a great help!

Hopefully this little look into some of the technicalities behind working on HOI4 was interesting. If you got questions feel free to ask away!

The part of the team that isn’t trying to solve this OOS has now moved fully on to 1.8 ‘Husky’ and next expansion work, but its early days and its going to be a while until we have things to show off. So this will be the last dev diary in a while as we go into radio silence (and soon glorious socialist swedish summer vacation!) until we have things that are ready to show off.

See you on the other side! And keep an eye on the forum for announcement about the 1.7.1 hotfix when that is ready.
 
Last edited:
We had an repoducible OOS on 1.7.0 beta (d12e) when a friend in multiplayer clicked one of his airports as Czech.
I hosted this LAN game as Yogoslavia and another friend was with as playing Romania.
 
I think this vastly overestimates the amount of money they actually earn compared to much of the game industry. There's a vast gulf between what you are thinking of them being and "small indie developers". Their revenue for 2017 didn't even break the $100 million mark. The actual development side of things for Paradox is far smaller than the publishing side, so equating the two is absurd.
HOI4 is literally in the list of best selling games and by now has likely sold a few 100K more, with DLC on top. I didn't equate them to anything but a corporation (which they obviously are). Yes there are some even bigger fish in the sea but selling over a million copies is a lot in the PC gaming industry.
If they halted development it's not just that they won't make money with new DLC, it will also negatively impact future sales of existing DLC and other Paradox games because they rely more than other studios on the players' good will.
 
Last edited:
What did you say in the post? What was the tone? Was it off-topic of the OPs thread?
Well, we will never know, what exactly I wrote but I will tell you how it came about. I went from the HOI4 forum to the one of Stellaris and I was reading the thread about the latest patch, which was very disappointing. I won't play both games, in the foreseeable future.
 
Is Podcat in Los Angeles, California at E3?


upload_2019-6-12_12-41-3.png
 
I think you guys are demanding too much. Husky won't be out for another year. We are lucky to even get this 1.7.1 patch.

If you've paid $50 for the game (on average, including DLC) very little goes to actual development.

30% goes to billionaire Gabe Newell (although he gave AAA devs like Paradox a better deal, 25% after $10 million total revenue, but only recently)

20% goes to Paradox corporate overhead

20% goes to the European/Swedish Welfare State

only about 30%, or $15, is left for the developers.

Maybe if Paradox start developing in Eastern Europe and selling exclusively on the Epic Games Store you can expect more for your money.
 
Well Operation Husky was the Allied invasion of Sicily, so possibly something related to Italy, and so fingers crossed the rest of the Mediterranean - although if it is this option than I doubt it's deliberate as I don't think Paradox would make it this obvious.
Italy/France/Spain/Ethiopia or Greece DLC confirmed? :p

Or rather, that's my best guess for a Mediterranean DLC - Italy and France as the two majors, Spain is the obvious minor in a Mediterranean pack, and Ethiopia and Greece are the minors with the most obvious involvement with those majors, excluding those that were in previous DLC and fit the 2 majors + 2-3 minors model of WtT and MtG (yes, WtT technically covered many more minors, but it's basically PRC, CHI, and other warlords and there's just a lot using that last tree)

Though it really would be a shame to leave the USSR as the last major to get a rework because their focus tree was among the worst of all majors before they started reworking other majors' focus trees.
 
About the resync problem. Me and my friends had that problem, and the we have the soulution. If you get the resync error message there is always the player name who had the error. So the soultion is the fowllong. Save the game. Load in single player and continue with that player country who had the error. Let run the game 4-5 game hour (i mean in game hours not real). Save it, and load back in the MP. It seem so that the game can solve the error itself in SP.
 
I think you guys are demanding too much. Husky won't be out for another year. We are lucky to even get this 1.7.1 patch.

If you've paid $50 for the game (on average, including DLC) very little goes to actual development.

30% goes to billionaire Gabe Newell (although he gave AAA devs like Paradox a better deal, 25% after $10 million total revenue, but only recently)

20% goes to Paradox corporate overhead

20% goes to the European/Swedish Welfare State

only about 30%, or $15, is left for the developers.
By your own calculation that results in upwards of $15,000,000 of revenue. And the 20% "corporate overhead" provided the developers with an existing game engine, staff familiar with said engine, and assets from previous iterations of the game. So no, I don't think we are demanding too much.
 
only about 30%, or $15, is left for the developers.
Actually it is less than that because much of the Money would be spent on Advertisement, support development of new games and so on.

Maybe less than 10% of the Money goes back into the game, the rest is consumed by other things that don't directly have to do with the game.
 
Last edited:
By your own calculation that results in upwards of $15,000,000 of revenue. And the 20% "corporate overhead" provided the developers with an existing game engine, staff familiar with said engine, and assets from previous iterations of the game. So no, I don't think we are demanding too much.
That's... not what corporate overhead means. Corporate overhead includes such things as fixed expenses (rent/mortgage payments, asset depreciation, legal fees, insurance policies, utilities) and variable expenses (advertising for products, telecommunications costs, office supplies; these tend to go up as more product is sold). It is the expenses needed for the business to keep running regardless of whether they are selling product or not. And $15,000,000 is a minuscule amount for a company with around 300 employees.

I'm finding more and more that people have a poor sense of scale when it comes to business.
 
Cheers for the DD Podcat (and enjoy that breakfast :D) and for the extra info Andy :cool:. Best of luck hunting down the OOS and other bugs.

please add dragons to the game

I fully agree - here's the kind of Dragon I'd like to see (ORP Dragon, a British Danae class light cruiser given to the Polish Navy):

ORP_dragon.jpg


Two dev posts to state that you're looking into some OoS problems? What about... bugs?

Isn't an OOS issue, strictly speaking, a bug?
 
@podcat - I do hope you guys do give us a bugfix sooner rather than later, I'd hate to have bugs just... kinda stay in the game until next DLC is released. Because that feels slightly unreasonable. even by Paradox standards.
 
A hotfix specifically for MP? Very nice.

Resyncing will reset the state of the game

Doesn't this also have the effect of resetting event MTTH? I think every really late Spanish Civil War I've seen has been precipitated by one or more resyncs or hotjoins (which reset the gamestate like resyncs do).

If so, I'd edit the SCW to ensure it always triggers in 1936, because very late SCWs can screw up the balance as Allies can get blocked until January 1938 (JAP DoW on CHI in MP) from doing certain actions that require at least 5.1% world tension.