• We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.

Stellaris Dev Diary #181 : Threading and Loading Times

Hello everyone, this is The French Paradox speaking!

On behalf of the whole Stellaris team, we hope you've had a good summer vacation, with current circumstances and all!

We're all back to work, although not at the office yet. It is going to be a very exciting autumn and winter with a lot of interesting news! We are incredibly excited to be able to share the news with you over the coming weeks and months!

Today I open the first look at the upcoming 2.8 release with some of the technical stuff that we programmers have been working on over summer. The rest of the team will reveal more about the upcoming content and features in the following diaries.

Without further ado, let's talk about threads!

Threads? What threads?

There is a running joke that says fans are always wondering which one will come first: Victoria III or a PDS game using more than one thread.

image (26).png

Don't lie, I know that's how some of you think our big decision meetings go

I’m afraid I’ll have to dispel the myth (again): all PDS games in production today use threads, from EU4 to CK3. Even Stellaris! To better explain the meme and where it comes from, we have to go through a little history. I’m told you guys like history.

For a long time, the software industry relied on “Moore’s Law”, which states that a CPU built in two years will be roughly twice as efficient as one today.
This was especially true in the 90s, when CPUs went from 50 MHz to 1GHz in the span of a decade. The trend continued until 2005 when we reached up to 3.8GHz. And then the clock speed stopped growing. In the 15 years since, the frequency of CPUs has stayed roughly the same.
As it turns out, the laws of physics make it quite inefficient to increase speeds beyond 3-4 GHz. So instead manufacturers went in another direction and started “splitting” their CPUs into several cores and hardware threads. This is why today you’ll look at how many cores your CPU has and won’t spend much time checking the frequency. Moore’s Law is still valid, but, to put it in strategy terms, the CPU industry reached a soft cap while trying to play tall so they changed the meta and started playing wide.

This shift profoundly changed the software industry, as writing code that will run faster on a CPU with a higher speed is trivial: most code will naturally do just that. But making usage of threads and cores is another story. Programs do not magically “split” their work in 2, 4 or 8 to be able to run on several cores simultaneously, it’s up to us programmers to design around that.

Threading nowhere faster

Which brings us back to our games and a concern we keep reading on the forums: “is the game using threads?”. The answer is yes, of course! In fact, we use them so much that we had a critical issue a few releases back where the game would not start on machines with 2 cores or less.

But I suspect the real question is : “are you making efficient usage of threads?”. Then the answer is “it depends”. As I mentioned previously, making efficient use of more cores is a much more complex issue than making use of more clock cycles. In our case, there are two main challenges to overcome when distributing work among threads: sequencing and ordering.

Sequencing issues occur when 2 computations running simultaneously need to access the same data. For example let’s say we are computing the production of 2 pops: a Prikki-Ti and a Blorg. They both access the current energy stockpile, add their energy production to it and write the value back. Depending on the sequence, they could both read the initial value (say 100), add their production (say 12 and 3, the Blorg was having a bad day) and write back. Ideally we want to end up with 115 (100 + 12 + 3). But potentially both would read 100, then compute and overwrite each other ending up with 112 or 103.
The simple way around it is to introduce locks: the Prikki-Ti would “lock” the energy value until it’s done with its computation and has written the new value back, then the Blog would take its turn and add his own. While this solves the problem, it introduces a greater one: the actions are now sequential again, and the benefit of doing them on concurrent threads has been lost. Worse, due to the cost of locking, unlocking and synchronizing, the whole thing will likely take longer than if we simply computed both on the same thread in the first place.

The second issue is ordering, or “order dependency”. Meaning in some cases changing the order of operations changes the outcome. For example let’s say our previous Prikki-Ti and Blorg decide to resolve a dispute in a friendly manner. We know the combat system will process both combatants, but since there are potentially hundreds of combat actions happening, we don’t know which one will happen first. And potentially on 2 different machines the order will differ. For example on the server the Prikki-Ti action will happen first, while on the client the Blorg will act first.

OOS.png

#BlorgShotFirst

On the server the Prikki-Ti action is resolved first, killing the Blorg. The Blorg action that comes after (possibly on another thread) is discarded as dead Blorgs can’t shoot (it’s a scientific fact). The client however distributed the computation in another way (maybe it has more cores than the server) and in his world the Blorg dispatched the Prikki-Ti first, which in turn couldn’t fight back. Then both players get the dreaded “Player is Out of Sync” popup as their realities have diverged.

There are, of course, ways to solve the problem, but they usually require redoing the design in a way that satisfies both constraints. For example in our first case each thread could store the production output of each pop to add to each empire, and then those could be consolidated at the end. In the same fashion our 2 duelists problem could be solved by recording damage immediately, but applying the effects in another phase to eliminate the need for a deterministic order.

As you can imagine, it is much easier to design something with threading in mind rather than retrofitting an existing system for it. If you don’t believe me just look at how much time is spent retrofitting your fleets, I’ll wait.

The good news

This is all nice and good, but what’s in it for you in the next patch, concretely? Well you will be happy to hear that I used some time to apply this to one of the oldest bits of our engine: the files and assets loading system.

For the longest time we have used a 3rd party software to handle this. While it saved us a lot of trouble, it has also turned out to be quite bad at threading. Up to the point that it was sometimes slower with more cores than less, most notably to the locking issues I mentioned before.
In conjunction with a few other optimizations, it has enabled us to drastically reduce the startup time of the game.
I could spend another thousand word explaining why, but I think this video will speak better:


This comparison was done on my home PC, which uses a venerable i7 2600K and an SSD drive. Both were “hot” startups (the game had been launched recently), but in my experiments I found that even on a “cold” start it makes a serious difference.

To achieve the best speedup, you will need to use the new beta DirectX11 rendering engine. Yes, you read correctly: the next patch will also offer an open beta which replaces the old DX9 renderer by a more recent DX11 version that was initially made by our friends at Tantalus for the console edition of Stellaris. While visually identical, using DX11 to render graphics enables a whole range of multi-threading optimizations that are hard or impossible to achieve with DX9. Playing with the old renderer will still net you some nice speedup on startup, the splash screen step should still be much faster, but you’re unlikely to see the progress bar “jump” as it does with DX11 when the game loads the models and textures.

Some of those optimizations have also been applied to newer versions of Clausewitz, and will be part of CK3 on release. Imperator should also benefit from it. It might be possible to also apply it to EU4 and HoI4, but so far my experiments with EU4 haven’t shown a huge speedup like it did for Stellaris and CK3.

If you want to read more technical details about the optimizations that were applied to speedup Stellaris, you can check out the article I recently published on my blog.

And with that I will leave you for now. This will likely be my last dev diary on Stellaris, as next month I will be moving teams to lead the HoI4 programmers. You can consider those optimizations my farewell gift.
This may have been a short time for me on Stellaris but don’t worry: even if I go, Jeff will still be there for you!
 
Last edited:
  • 145Like
  • 38Love
  • 24
  • 6
  • 5Haha
  • 4
Reactions:
Edit: quoting broke, this is intended as a reply to Pyzayt.

Trying to play devil's advocate is kind of disingenuous in this context. I've played Stellaris from the day it was released, ever since that day the game was riddled with problems. But I thought to myself -- I do like this game very much, and its a PDX game, they will make it better over time. Stellaris was released 4 years ago, megacorp that broke AI, sector AI, crisis AI (should I even continue this list?) was released two *years* ago. And over those years all I heard from PDX was "we're working on it". Fact of the matter is, 2/3 of the systems that were present prior to 2.2 release are still in the worse state today. Call it complaining, but I just dont believe empty promises at this point, I need some tangible results. And for Pete's sake stop talking about the virus like only devs at PDX are dealing with it. I don't remember taking a 2-months summer vacation, so forgive me for saying that they probably won't be overworked at this point. There are countless problems that people were complaining about, legitimate problems I might add, and then they got a DD that talked about a problem I've never heard *anyone* complaining in *four* years. Cool that they did it, but like someone said in this thread, prioritize please. I just won't be fed empty promises that they are "working on it", I've heard that one too many times without seeing any results. So don't try to spin it like people are just hating on paradox for no reason, that they just need to wait, be content with what they have and so on, absolutely ridiculous statement.
 
  • 13
  • 2Like
  • 1
Reactions:
you forget that stellaris fans are the kind of people who can spend a hundred hours on a game and then call it unplayable.
This is far too true.
So in other words: If I've reached an arbitrary limit of played hours ( so that I can be called an experienced player on top ) then it's no longer justified that I express any form of critique ?

How about the thing that critiques refer to the latest version of this game ( V2.7.2 ) whereas the counter for played hours is an accumulated one that includes not only the current, but all previous versions as well ? Wanna tell me that a guy or girl who had played ( for example ) V1.9.1 extensively is as satisfied with V2.7.2, just because he / she had accumulated a zillion + 1 hour of play-time altough said zillion hours are from V1.9.1 whereas just said 1 hour is actually from V2.7.2 ? In theory, this counter for played hours has to be resetted every time this game gets a new update.

It's also a thing that this counter for played hours counts also when you're playing with mods: You two adorn Paradox with borrowed plumes if you think it's Paradox merit that players have accumulated a zillion hours of play-time altough they had actually played a completed, fixed, optimised, balanced, changed and extended ( modded ) version of this game that makes it not only playable, but enjoyable, too.

How about the "wasted" hours of play-time ( that're also included in said counter ) due to bugs that have led to CtDs or due to other bugs that have led to abandoned games ?

Stellaris players always seem to need to find a reason to complain about what the devs are doing, and while I think that Stellaris could certainly be improved (and frankly does need more polish for its existing systems), the way people here go about it is... not okay.
Bad customer, BAD !

We got radio silence, and we complained.
So, you think that 3 months of radio silence is something that should be celebrated instead ?

They said they're looking at performance,
So, you think that a non-committal intention ( I've "never" "ever" heard before on top ) is something that should be celebrated, too ?

We demanded they fix systems, and they did, and we complained (specifically about it not being enough in this case, despite "enough" being subjective).
I'm sorry, but it's the ingame-performance-problems that most people complain about, not such outgame-performance-"problems" how Stellaris wouldn't boot, save and load that fast ( and we're just talking about seconds here on top ). Stellaris is not a Total-War-game in which something like this would make an actual difference ( several "breaks" in a session due to switches between the campaign-map and battles ) since it's still a Paradox-game that runs continuously once you're in your session.

Frankly, every time this community asks for something, when we get it, people complain.
It's rather the other way around: People have to complain in order to get what ( patches ) they want, especially if it's not that profitable as ... for example ... DLCs since otherwise Paradox gets the excuse that everything seems to be fine and dandy.

PDX has been absurdly tolerant of our crap, and seriously deserves a break from it all.
You've never heard about these rules against this so called "toxicity" ? And the 3 months of radio silence ( that qualifies as such a break ) you've already forgotten ?

I don't want to see the Stellaris dev team go the same way a lot of my favourite old things went; abandoned and despised by its creators for the people that came to call themselves fans.
My concern is more the other way around that Paradox continues to throw out DLCs ( as long as said DLCs get selled ) without having the prioritized intention or even the ability to overcome the state of this game.
 
Last edited:
  • 16
  • 2Like
  • 1
Reactions:
Edit: quoting broke, this is intended as a reply to Pyzayt.

Trying to play devil's advocate is kind of disingenuous in this context. I've played Stellaris from the day it was released, ever since that day the game was riddled with problems. But I thought to myself -- I do like this game very much, and its a PDX game, they will make it better over time. Stellaris was released 4 years ago, megacorp that broke AI, sector AI, crisis AI (should I even continue this list?) was released two *years* ago. And over those years all I heard from PDX was "we're working on it". Fact of the matter is, 2/3 of the systems that were present prior to 2.2 release are still in the worse state today. Call it complaining, but I just dont believe empty promises at this point, I need some tangible results. And for Pete's sake stop talking about the virus like only devs at PDX are dealing with it. I don't remember taking a 2-months summer vacation, so forgive me for saying that they probably won't be overworked at this point. There are countless problems that people were complaining about, legitimate problems I might add, and then they got a DD that talked about a problem I've never heard *anyone* complaining in *four* years. Cool that they did it, but like someone said in this thread, prioritize please. I just won't be fed empty promises that they are "working on it", I've heard that one too many times without seeing any results. So don't try to spin it like people are just hating on paradox for no reason, that they just need to wait, be content with what they have and so on, absolutely ridiculous statement.


As a day one Stellaris player (first paradox game bought on release date) i can only agree with you. Firstly i will never more buy a paradox game on release and secondly i stopped buyed DLC since Apocalypse patch. I prefer to wait and see what they are going with this game first. We paid for an enjoyable game and, 4 years latter, Stellaris still full of problems and still not enjoyable with AI and pop management problems (without counting the economy, internal politic and diplomacy near non existance and the fact that only warfare is rewarded).

If they are doing like CK2 (6 years of support) we still only have 2 years of support to make it an enjoyable game, i admit i'm kind of sceptic here. I have already throw too much money into this game and will follow patches and DLC to see what they are doing and if the game is trully improving before doing anything else and buy another DLC. And, if it don't i will never more buy a paradox game of my life, simply as it. First Stellaris state, secondly vampire the masquerade lead narrative designer fired without reasons, i start to have high doubts about this company and their methods.
 
Last edited:
  • 5
  • 2Like
  • 1
Reactions:
As a day one Stellaris player (first paradox game bought on release date) i can only agree with you. Firstly i will never more buy a paradox game on release and secondly i stopped buyed DLC since Apocalypse patch. I prefer to wait and see what they are going with this game first. We paid for an enjoyable game and, 4 years latter, Stellaris still full of problems and still not enjoyable with AI and pop management problems (without counting the economy, internal politic and diplomacy near non existance and the fact that only warfare is rewarded).

If they are doing like CK2 (6 years of support) we still only have 2 years of support to make it an enjoyable game, i admit i'm kind of sceptic here. I have already throw too much money into this game and will follow patches and DLC to see what they are doing and if the game is trully improving before doing anything else and buy another DLC. And, if it don't i will never more buy a paradox game of my life, simply as it. First Stellaris state, secondly vampire the masquerade lead narrative designer fired without reasons, i start to have high doubts about this company and their methods.

I have to agree with your stance. I highly doubt I'll buy any other game from PDX, perhaps only Victoria 3 and then, only if it will be good (doubt it). Stellaris will be my last. They've lost all their credibility as quality developers.

I'll tell you what I'll buy though: Factorio DLCs and expansions. Wube is King on quality.
 
  • 4Like
  • 2
  • 1
Reactions:
I saw the comparison video and I must say: the game NEVER took this long to load on my SSD. It loaded in, maybe, 10 secs?

So after those improvements, starting the game should cause my PC to go back in time I guess.

That is not to say I don't appreciate these improvements.

I got it on SSD too and it never loaded so fast, never.
 
It depends what SSD, and what mods you have. With NVME and no mods, it's really fast.
All DLCs + NVMe Samsung 970 500GB. It's still not "really fast" mate :) My PC is definitely not a potato.

I haven't found this to be the case on an NVME drive, even with no mods, still takes a good minute to load. It's possible having less DLCs might speed loading up.
This.

----
I'm still more curious how much faster saves load. As usually when i turn on Stellaris I just go make a coffee or something.
 
There are many post to criticize game performance, and when one dev comes and explains that they are working on it, it's still not enough !

From what I read in his post they are working on threading, not only on loading times.

Now bashers can do what they like most : bash the devs
 
  • 10
  • 7
Reactions:
There are many post to criticize game performance, and when one dev comes and explains that they are working on it, it's still not enough !
It seems like that you weren't really around the last 4+ years since otherwise you would be aware of the mantra that they've always worked on performance-issues so that they're still doing it today. But since the performance-issues got / get out of hand, ... well ... I don't want to take away the joy from you to put the dot(s) together what this means whether Paradox efforts ( with actual results ( no non-committal ( and illusive ) intentions like "they are ( and always ) working on it" ) ) were / are "enough" or not.

Now bashers can do what they like most : bash the devs
Express any form of critique equals "bashing", "hating", "trolling" or whatever, I get it, but on a second thought, NO.
 
Last edited:
  • 8
  • 4Like
  • 4
Reactions:
the simple fact of the matter is that most of the issues that people are always "bashing" the devs for have been around for years(bad ai, performance) and people have mostly lost patience with the constant "we are always working on performance" line, and the fact that every update seems to bring more issues than it solves, even the much vaunted 2.6 bugfix patch barely made a dent in the massive array of issues, and then they promptly broke the ai immediately after they said they fixed it

can you really blame people for being skeptical?
 
  • 8
  • 7Like
  • 2Love
  • 1
Reactions:
the simple fact of the matter is that most of the issues that people are always "bashing" the devs for have been around for years(bad ai, performance) and people have mostly lost patience with the constant "we are always working on performance" line, and the fact that every update seems to bring more issues than it solves, even the much vaunted 2.6 bugfix patch barely made a dent in the massive array of issues, and then they promptly broke the ai immediately after they said they fixed it

can you really blame people for being skeptical?

Exactly this, been hearing the same line for years to no effect.

The crisis has essentially had some kind of issue in every patch since release.

War in heaven has almost never properly worked.

Performance was bad before they redesigned the economic system.

I remember in the build up to megacorp how one of the dev posts said that this rework would have some benefits for performance. We all know how that turned out....

I just don't understand how whilst the Devs were sat around a table discussing the economy system someone didn't say "we need to make it so the AI can use it and so its not a performance bottleneck"

I've said this a few times now, but at this point I believe stellaris to be unsalvageable due to bad design choices and poor implementation of said design.

There's two scenarios here:

They are unable to make stellaris into a decent state.

Or they are unwilling to expend the required resources.
 
  • 7
  • 2Like
  • 1
  • 1
Reactions:
After playing with both EU4 and Stellaris loading benchmarks, I can give you one right now: try avoiding have lots of small files (like EU4 does with one text file per province). Windows is just terrible at loading those efficiently. My first idea was to "compile" the game text files into one big archive after the first load to speed up subsequent startups, but it would have taken some time to implement safely and wouldn't have benefited Stellaris as much as the other changes I made.
I still keep it in mind for later, but as you may guess there's always more things we could do than time to actually do them.
Loading things from an archive instead of straight from the file system you say... Isn't that one of the big reasons for using PhysFS in the first place and also what you are already doing for mods downloaded from the workshop?
 
  • 1
Reactions: