Good morning all. We're back with another dev diary. As mentioned last week we have been working on something behind the scenes. In a nutshell, we have put a tonne of work into multiplayer, to tackle some long-standing thorns such as Hotjoin and Cross-platform issues. Many users have raised issue with these functions not working well, or indeed at all, and with good reason! Like last week, I'll hand the mic over to the man responsible for the bulk of this, @Alexivan
################
Hi. I’d like to start by saying that multiplayer support for our games is not an easy task, and sometimes we dev’s joke that it would be easier to simply cut that feature. But, seeing people having a great time playing together inspires us to continue supporting it. Not to mention that we love playing ourselves in our Dev Clash MP.
EU4 has a lot of complexity when it comes to multiplayer as it has the largest amount of configurations MP can be played on among our games, the factors being:
Platform
DLC configurations
Hotjoin
Start dates/save game
Custom nations
World randomization
Random new world
Our efforts so far have mostly concentrated on hosting a game with 1444 and continuing a save game. During the development of this update however we’ve put some more effort on making MP work across all the different configurations, in the priority order of the list. I’d like to share some wisdom about some of the issues we’ve encountered and methods we use.
When we started working on the MP for the upcoming 1.23 Persia update the goal was to get hotjoin working again. Hotjoin is basically copying over the current state of the game to the newly connected player which in theory is a pretty simple thing to do. There are two parts of the gamestate that need to be synchronized: all the data that is saved in the gamestate, and all the data that is not. We’ll call these categories persisted and cached.
Starting with the persisted data we noticed that saves were not being saved as they are read, meaning we modify the savegame while reading and writing it. While this works for the standard setup as everyone applies the same changes when they all load the game it causes hotjoin to desynchronize as only the hotjoiners are doing the changes.
We added a console command called check_save to help us find this issue, it’s a console command that saves the game, loads it up, and saves it again. This helped us find most of the issues with savegame inconsistency. Most of the issues here came from calling gameplay logic when reading the save, so reading a country government could for example apply some modifiers or reset some variables. Otherwise it was mostly variables not being written/read properly.
For the cached data we had to do a couple of things. First we added a recalculation step which happens when you load a savegame or a hotjoiner enters the game, this step recalculates all the cached data. A lot of the data in this step was derived from the persisted data, which worked fine. Some however was dependent on other cached data, causing things to be recalculated differently, we solved this by moving all cached values into a separate container so we could easily clear all cached data before calculating it, guaranteeing the same results.
Another issue with persisted data would be load order dependencies. There are two common issues here, the first one being that objects that were loaded sometimes put themselves in a list. This causes the list to be sorted by the order of objects being loaded, rather than whatever order they were in when saving the game. Once these lists were identified the easy fix was to simply sort the lists in a deterministic order. The second issue is that objects being read from the savegame are sometimes referencing other objects in the savegame, which haven’t been read yet. The thing we do here is save the object ID mentioned in the save and dereference in a step after the save has been loaded.
After check_save resulted in the savegames with no differences, we started running MP tests to find more issues. One thing that helps us a lot with this is the randomlog, which is a tool we use to help us track desynchronized computation sequences. The way it works is that we add lines in the code which prints to the randomlog, when a desync happens we can then compare the randomlog data of the host to that of the desynchronized client and see what the client did differently. This shows us what happened incorrectly, but usually isn’t enough to pinpoint why. If we can’t figure out what happened we add more entries in related code to try getting closer to the source issue. This can be a slow process where we sometimes have to add more logs lots of times to help us find the issue, especially if the desync doesn’t happen consistently due to multithreading or the desync being a rare case.
As QA time for finding desyncs is limited, testing was slow, and we still knew we had a lot of desyncs to find so we had to start automating tests. For this we updated the standalone server build and improved on the script we use to help us find desyncs faster, this also allowed us to run the automated tests over the weekends and nights and fix during the days, which greatly improved our efficiency, the script was also updated to restart when an issue has been found, generating a lot of different desyncs.
As we now could get multiple desyncs with logs and data we needed to improve our method for collecting desync data, so we copied over the OOS dumps system from stellaris. This system copies over data relevant to the desync into a unique folder inside Paradox Interactive/Europa Universalis IV/oos folder. The name of the folder is a unique hash describing the desync and can be used to match the desync between clients and host.
All in all, I’ve been working on all this for the last 7 months with a lot of help from the team, a lot of systems had to be fixed and specialized solutions had to be invented, it’s been quite a bigger task than I initially expected. I was hoping to get it to a perfect state for this release but it’s not quite there yet.
So if you get desyncs I recommend letting us know, try giving us as much information as you could and more specifically how the game was set up, the OOS dump, and any savegame you might have loaded. If you manage to get desynced consistently I recommend starting the game with the “-randomlog” and “-hardoos” launch options on the host to generate better data for the OOS dumps. But otherwise it might be better to keep it turned off as it uses memory and slows down the game. I’m going to look into if we could potentially use something like the crash dump uploader to make it easier for you guys to send this data to us.
tl;dr: A lot of work on multiplayer has been done, and it will continue. Cross Platform, Cross DLC, and Hotjoin should now be working. Reporting your desyncs might get them fixed faster. If you guys have any questions or suggestions I’d be glad to discuss in this thread.
##################
Thanks Alexivan. Next week we'll have another mixed bag of Cradle of Civilization goodies. With the release only a couple of weeks away, we need to round off on all the new things coming in 1.23 and Cradle.
Although, this dev diary doesn't have any images in it...hrmm, let's change that.
################
Hi. I’d like to start by saying that multiplayer support for our games is not an easy task, and sometimes we dev’s joke that it would be easier to simply cut that feature. But, seeing people having a great time playing together inspires us to continue supporting it. Not to mention that we love playing ourselves in our Dev Clash MP.
EU4 has a lot of complexity when it comes to multiplayer as it has the largest amount of configurations MP can be played on among our games, the factors being:
Platform
DLC configurations
Hotjoin
Start dates/save game
Custom nations
World randomization
Random new world
Our efforts so far have mostly concentrated on hosting a game with 1444 and continuing a save game. During the development of this update however we’ve put some more effort on making MP work across all the different configurations, in the priority order of the list. I’d like to share some wisdom about some of the issues we’ve encountered and methods we use.
When we started working on the MP for the upcoming 1.23 Persia update the goal was to get hotjoin working again. Hotjoin is basically copying over the current state of the game to the newly connected player which in theory is a pretty simple thing to do. There are two parts of the gamestate that need to be synchronized: all the data that is saved in the gamestate, and all the data that is not. We’ll call these categories persisted and cached.
Starting with the persisted data we noticed that saves were not being saved as they are read, meaning we modify the savegame while reading and writing it. While this works for the standard setup as everyone applies the same changes when they all load the game it causes hotjoin to desynchronize as only the hotjoiners are doing the changes.
We added a console command called check_save to help us find this issue, it’s a console command that saves the game, loads it up, and saves it again. This helped us find most of the issues with savegame inconsistency. Most of the issues here came from calling gameplay logic when reading the save, so reading a country government could for example apply some modifiers or reset some variables. Otherwise it was mostly variables not being written/read properly.
For the cached data we had to do a couple of things. First we added a recalculation step which happens when you load a savegame or a hotjoiner enters the game, this step recalculates all the cached data. A lot of the data in this step was derived from the persisted data, which worked fine. Some however was dependent on other cached data, causing things to be recalculated differently, we solved this by moving all cached values into a separate container so we could easily clear all cached data before calculating it, guaranteeing the same results.
Another issue with persisted data would be load order dependencies. There are two common issues here, the first one being that objects that were loaded sometimes put themselves in a list. This causes the list to be sorted by the order of objects being loaded, rather than whatever order they were in when saving the game. Once these lists were identified the easy fix was to simply sort the lists in a deterministic order. The second issue is that objects being read from the savegame are sometimes referencing other objects in the savegame, which haven’t been read yet. The thing we do here is save the object ID mentioned in the save and dereference in a step after the save has been loaded.
After check_save resulted in the savegames with no differences, we started running MP tests to find more issues. One thing that helps us a lot with this is the randomlog, which is a tool we use to help us track desynchronized computation sequences. The way it works is that we add lines in the code which prints to the randomlog, when a desync happens we can then compare the randomlog data of the host to that of the desynchronized client and see what the client did differently. This shows us what happened incorrectly, but usually isn’t enough to pinpoint why. If we can’t figure out what happened we add more entries in related code to try getting closer to the source issue. This can be a slow process where we sometimes have to add more logs lots of times to help us find the issue, especially if the desync doesn’t happen consistently due to multithreading or the desync being a rare case.
As QA time for finding desyncs is limited, testing was slow, and we still knew we had a lot of desyncs to find so we had to start automating tests. For this we updated the standalone server build and improved on the script we use to help us find desyncs faster, this also allowed us to run the automated tests over the weekends and nights and fix during the days, which greatly improved our efficiency, the script was also updated to restart when an issue has been found, generating a lot of different desyncs.
As we now could get multiple desyncs with logs and data we needed to improve our method for collecting desync data, so we copied over the OOS dumps system from stellaris. This system copies over data relevant to the desync into a unique folder inside Paradox Interactive/Europa Universalis IV/oos folder. The name of the folder is a unique hash describing the desync and can be used to match the desync between clients and host.
All in all, I’ve been working on all this for the last 7 months with a lot of help from the team, a lot of systems had to be fixed and specialized solutions had to be invented, it’s been quite a bigger task than I initially expected. I was hoping to get it to a perfect state for this release but it’s not quite there yet.
So if you get desyncs I recommend letting us know, try giving us as much information as you could and more specifically how the game was set up, the OOS dump, and any savegame you might have loaded. If you manage to get desynced consistently I recommend starting the game with the “-randomlog” and “-hardoos” launch options on the host to generate better data for the OOS dumps. But otherwise it might be better to keep it turned off as it uses memory and slows down the game. I’m going to look into if we could potentially use something like the crash dump uploader to make it easier for you guys to send this data to us.
tl;dr: A lot of work on multiplayer has been done, and it will continue. Cross Platform, Cross DLC, and Hotjoin should now be working. Reporting your desyncs might get them fixed faster. If you guys have any questions or suggestions I’d be glad to discuss in this thread.
##################
Thanks Alexivan. Next week we'll have another mixed bag of Cradle of Civilization goodies. With the release only a couple of weeks away, we need to round off on all the new things coming in 1.23 and Cradle.
Although, this dev diary doesn't have any images in it...hrmm, let's change that.
Will make big multiplayer games so much more stable and viable again.
