• We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.

Stellaris Dev Diary #181 : Threading and Loading Times

Hello everyone, this is The French Paradox speaking!

On behalf of the whole Stellaris team, we hope you've had a good summer vacation, with current circumstances and all!

We're all back to work, although not at the office yet. It is going to be a very exciting autumn and winter with a lot of interesting news! We are incredibly excited to be able to share the news with you over the coming weeks and months!

Today I open the first look at the upcoming 2.8 release with some of the technical stuff that we programmers have been working on over summer. The rest of the team will reveal more about the upcoming content and features in the following diaries.

Without further ado, let's talk about threads!

Threads? What threads?

There is a running joke that says fans are always wondering which one will come first: Victoria III or a PDS game using more than one thread.

image (26).png

Don't lie, I know that's how some of you think our big decision meetings go

I’m afraid I’ll have to dispel the myth (again): all PDS games in production today use threads, from EU4 to CK3. Even Stellaris! To better explain the meme and where it comes from, we have to go through a little history. I’m told you guys like history.

For a long time, the software industry relied on “Moore’s Law”, which states that a CPU built in two years will be roughly twice as efficient as one today.
This was especially true in the 90s, when CPUs went from 50 MHz to 1GHz in the span of a decade. The trend continued until 2005 when we reached up to 3.8GHz. And then the clock speed stopped growing. In the 15 years since, the frequency of CPUs has stayed roughly the same.
As it turns out, the laws of physics make it quite inefficient to increase speeds beyond 3-4 GHz. So instead manufacturers went in another direction and started “splitting” their CPUs into several cores and hardware threads. This is why today you’ll look at how many cores your CPU has and won’t spend much time checking the frequency. Moore’s Law is still valid, but, to put it in strategy terms, the CPU industry reached a soft cap while trying to play tall so they changed the meta and started playing wide.

This shift profoundly changed the software industry, as writing code that will run faster on a CPU with a higher speed is trivial: most code will naturally do just that. But making usage of threads and cores is another story. Programs do not magically “split” their work in 2, 4 or 8 to be able to run on several cores simultaneously, it’s up to us programmers to design around that.

Threading nowhere faster

Which brings us back to our games and a concern we keep reading on the forums: “is the game using threads?”. The answer is yes, of course! In fact, we use them so much that we had a critical issue a few releases back where the game would not start on machines with 2 cores or less.

But I suspect the real question is : “are you making efficient usage of threads?”. Then the answer is “it depends”. As I mentioned previously, making efficient use of more cores is a much more complex issue than making use of more clock cycles. In our case, there are two main challenges to overcome when distributing work among threads: sequencing and ordering.

Sequencing issues occur when 2 computations running simultaneously need to access the same data. For example let’s say we are computing the production of 2 pops: a Prikki-Ti and a Blorg. They both access the current energy stockpile, add their energy production to it and write the value back. Depending on the sequence, they could both read the initial value (say 100), add their production (say 12 and 3, the Blorg was having a bad day) and write back. Ideally we want to end up with 115 (100 + 12 + 3). But potentially both would read 100, then compute and overwrite each other ending up with 112 or 103.
The simple way around it is to introduce locks: the Prikki-Ti would “lock” the energy value until it’s done with its computation and has written the new value back, then the Blog would take its turn and add his own. While this solves the problem, it introduces a greater one: the actions are now sequential again, and the benefit of doing them on concurrent threads has been lost. Worse, due to the cost of locking, unlocking and synchronizing, the whole thing will likely take longer than if we simply computed both on the same thread in the first place.

The second issue is ordering, or “order dependency”. Meaning in some cases changing the order of operations changes the outcome. For example let’s say our previous Prikki-Ti and Blorg decide to resolve a dispute in a friendly manner. We know the combat system will process both combatants, but since there are potentially hundreds of combat actions happening, we don’t know which one will happen first. And potentially on 2 different machines the order will differ. For example on the server the Prikki-Ti action will happen first, while on the client the Blorg will act first.

OOS.png

#BlorgShotFirst

On the server the Prikki-Ti action is resolved first, killing the Blorg. The Blorg action that comes after (possibly on another thread) is discarded as dead Blorgs can’t shoot (it’s a scientific fact). The client however distributed the computation in another way (maybe it has more cores than the server) and in his world the Blorg dispatched the Prikki-Ti first, which in turn couldn’t fight back. Then both players get the dreaded “Player is Out of Sync” popup as their realities have diverged.

There are, of course, ways to solve the problem, but they usually require redoing the design in a way that satisfies both constraints. For example in our first case each thread could store the production output of each pop to add to each empire, and then those could be consolidated at the end. In the same fashion our 2 duelists problem could be solved by recording damage immediately, but applying the effects in another phase to eliminate the need for a deterministic order.

As you can imagine, it is much easier to design something with threading in mind rather than retrofitting an existing system for it. If you don’t believe me just look at how much time is spent retrofitting your fleets, I’ll wait.

The good news

This is all nice and good, but what’s in it for you in the next patch, concretely? Well you will be happy to hear that I used some time to apply this to one of the oldest bits of our engine: the files and assets loading system.

For the longest time we have used a 3rd party software to handle this. While it saved us a lot of trouble, it has also turned out to be quite bad at threading. Up to the point that it was sometimes slower with more cores than less, most notably to the locking issues I mentioned before.
In conjunction with a few other optimizations, it has enabled us to drastically reduce the startup time of the game.
I could spend another thousand word explaining why, but I think this video will speak better:


This comparison was done on my home PC, which uses a venerable i7 2600K and an SSD drive. Both were “hot” startups (the game had been launched recently), but in my experiments I found that even on a “cold” start it makes a serious difference.

To achieve the best speedup, you will need to use the new beta DirectX11 rendering engine. Yes, you read correctly: the next patch will also offer an open beta which replaces the old DX9 renderer by a more recent DX11 version that was initially made by our friends at Tantalus for the console edition of Stellaris. While visually identical, using DX11 to render graphics enables a whole range of multi-threading optimizations that are hard or impossible to achieve with DX9. Playing with the old renderer will still net you some nice speedup on startup, the splash screen step should still be much faster, but you’re unlikely to see the progress bar “jump” as it does with DX11 when the game loads the models and textures.

Some of those optimizations have also been applied to newer versions of Clausewitz, and will be part of CK3 on release. Imperator should also benefit from it. It might be possible to also apply it to EU4 and HoI4, but so far my experiments with EU4 haven’t shown a huge speedup like it did for Stellaris and CK3.

If you want to read more technical details about the optimizations that were applied to speedup Stellaris, you can check out the article I recently published on my blog.

And with that I will leave you for now. This will likely be my last dev diary on Stellaris, as next month I will be moving teams to lead the HoI4 programmers. You can consider those optimizations my farewell gift.
This may have been a short time for me on Stellaris but don’t worry: even if I go, Jeff will still be there for you!
 
Last edited:
  • 145Like
  • 38Love
  • 24
  • 6
  • 5Haha
  • 4
Reactions:
Will there be more attention for the technical limitations of the game when designing features for future expansions? The game is still a lot slower then it was before the Megacorp dlc. New features can be nice and all but if the late game stays as slow as it currently is, I wouldn't consider it a lot of fun.
 
  • 12
  • 2Like
Reactions:
Shame I was wondering too how the Linux version would benefit from this change. Have you done any comparison between the native linux build and the new DX11 renderer running under proton with DXVK?

Would we ever be likely to see a Vulkan rendered? Seeing as this could benefit multiple platforms.
I was also wondering how DX11 will affect Stellaris on Linux. And thanks for the update :)
DX11 will obviously not change anything outside of windows.
Our engine team has been experimenting with Vulkan and DX12 for the new games but there are no plans for Stellaris as far as I know.
Will there be more attention for the technical limitations of the game when designing features for future expansions? The game is still a lot slower then it was before the Megacorp dlc. New features can be nice and all but if the late game stays as slow as it currently is, I wouldn't consider it a lot of fun.
We are aware about concerns of game speed. I can take a moment to clarify a bit.
Features are usually not the reason why the game performance varies. The big impact from Megacorp is due to the population system rework in the free patch. Galaxies nowadays have at least 4-5 times more population to simulate now, which comes with a cost.
As for why we looked at startup speed in the first place, first we developers restart the game a _lot_ during development (think something like 20+ times a day) so that's a huge productivity improvement for us. Second as I mentioned this could be shared with other titles, making it valuable for most of PDS and not just the Stellaris team.
 
  • 24
  • 7Like
  • 4
Reactions:
It's good to see a Dev Diary. That performance boost looks really cool. Will we get the same benefit in game or just during launch.

Also will the next Dev Diary have information about content for 2.8, maybe a DLC? If so please say it's an espionage/intelligence update
 
  • 1
Reactions:
Thanks for the update, looking forward to the next dev diaries. I hope there will be more updates on the late game performances (and maybe the AI).
 
  • 1
Reactions:
Features are usually not the reason why the game performance varies. The big impact from Megacorp is due to the population system rework in the free patch. Galaxies nowadays have at least 4-5 times more population to simulate now, which comes with a cost.
As for why we looked at startup speed in the first place, first we developers restart the game a _lot_ during development (think something like 20+ times a day) so that's a huge productivity improvement for us. Second as I mentioned this could be shared with other titles, making it valuable for most of PDS and not just the Stellaris team.

Thanks that is good to know but I should have been more clear with the question. With features I really meant all changes that are introduced including all changes the free patches introduce. My hope is that since Megacorp the impact on performance will have more attention when designing any new change to the game.

You don't have to answer any of this but because of Megacorp I have strong reservations about any new change that might be added. Adding something on top of a system that doesn't perform as well as it should or could might end up making the problem worse. We have a lot more pops now but what does it really add? You barely interact with them, or interact with factions, or leaders. I would completely agree with cutting down the total number of pops if that means that we can have far more interesting interactions with them.
 
Last edited:
  • 5
  • 2Love
Reactions:
It's good to see a Dev Diary. That performance boost looks really cool. Will we get the same benefit in game or just during launch.
Those changes are purely about loading times. As I mentioned the in-game speed is limited by completely different factors.
Also will the next Dev Diary have information about content for 2.8, maybe a DLC?
You will hear more about the content of the 2.8 update in the upcoming dev diaries.
Although it'll mostly be about Jeff...
 
  • 11
  • 5
  • 1Like
Reactions:
I dare you to try that loading time check again on a non-SSD hard drive and a hundred mods.
This is my life playing Stellaris at the moment. You can guess I'm over-hyped at this announcement. :)
 
  • 1
Reactions:
Our engine team has been experimenting with Vulkan and DX12 for the new games but there are no plans for Stellaris as far as I know.


Thanks, good to here you have been looking at the newer gfx pipelines, obviously I'm biased and I would hope that you go Vulkan over dx12 if only because I would love you to continue supporting linux natively.
 
  • 1Like
  • 1
Reactions:
Nice to hear that some under the hood performance is still being worked on even after the lag issues were alleviated (though not solved entirely). Hopefully there's a similar level of effort going into balance, gameplay, and bugs for the next version too.
 
  • 1Like
  • 1
Reactions:
Features are usually not the reason why the game performance varies. The big impact from Megacorp is due to the population system rework in the free patch. Galaxies nowadays have at least 4-5 times more population to simulate now, which comes with a cost.
To at least partly address this issue, if there isn't anything else viable to address it, would the team consider having pops that instead produce a good chunk more, but grow slower realtive to that increased producion?
 
  • 8
Reactions:
So, let me get this straight, see if I understood this correctly:

The programmers on Stellaris have worked all summer just to increase the start-up speed of the game, instead of the speed of the gameplay itself? Or will we see another optimization dev diary explaining in detail the optimizations made to the gameplay itself?

Excuse me for being so blunt, but starting up the game faster, doesn't make the game itself faster.
 
  • 13
  • 7
  • 2Like
Reactions:
My potato battery laptop thanks you for these optimisations.

Also, I knew something was up during 2.5 when my old laptop just refused to boot up the game!
 
  • 1
Reactions:
To at least partly address this issue, if there isn't anything else viable to address it, would the team consider having pops that instead produce a good chunk more, but grow slower realtive to that increased producion?
Exactly. I would gladly cut the number of starting pops in half and decrease their growth rates if it means better performance and using that computing power to make the game more interesting. The logic pops use to switch job, their migration logic, how they support factions or how crime works could all be improved upon and could be tied to interesting decisions for the player.
 
  • 2
  • 1Like
Reactions:
So, let me get this straight, see if I understood this correctly:

The programmers on Stellaris have worked all summer just to increase the start-up speed of the game, instead of the speed of the gameplay itself? Or will we see another optimization dev diary explaining in detail the optimizations made to the gameplay itself?

Excuse me for being so blunt, but starting up the game faster, doesn't make the game itself faster.
This isn't the only we worked on, no. This is merely what I toyed with during the slow summer month with a bit of outside help.
Why can't you calculate everything on host's machine and stream data to clients?
This is an idea that comes back from time to time. Thing is, it requires an efficient way of discerning what data changed during an update and then another efficient way to transmit that over the wire. So far we haven't seen a lot of empirical evidence that this would prove more efficient.
 
  • 8Like
  • 7
  • 4
Reactions:
So all this applies just to loading game and not to actually running it? Cuz Stellaris actually runs the worst for me while Imperator Runs the best and the smoothest. Personally don't mind loading times as much as how much time it takes for one tick to happen.

Anyways, idk these things well, but am quite curious.
 
  • 1
Reactions: