• We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.

Stellaris Dev Diary #170 - Performance and other technical issues

Hello, my friends! This is Moah, Tech Lead of Stellaris typing. I can finally talk about what you’ve all been waiting for: How many new platypi will there be in Federations? After weeks of…

Well, apparently, I should be "more technical." But before we jump into the mysteries of the Stellaris code, I want to take the time to talk a little about the balance between adding new features, improving performance and stability – especially in terms of multiplayer and the dreaded out-of-syncs (dreaded at least by me).

The Delicate Balance
Stellaris, like most decently sized code bases, is like a complex game of Mikado or Jenga: every part is connected in some way to every other part. When you add a feature, you add more connections. If you’re careful, you add only a few, if you’re in a rush you add a bit too many. This generally leads to Unplanned Features (aka bugs). In addition, once we see them perform in the actual game, we tend to expand features in new, unexpected ways, leading to more Unplanned Features(tm).

Once we realize what is happening, we start being more careful. Maybe too careful. Checking too many things, too often, ensuring that this interaction that is supposed to never actually happen is actually not happening. Not now, not later. Not ever.

So you have removed the unplanned features, but the game is a bit, ah… too careful. Some would say slow.

So you remove some of these checks. You realize that you don’t need to loop around the galaxy, you can just loop around this one tiny planet. Then you go one step further, and think “well I can maybe do that check only every three weeks, and this calculation needed by all these checks, I could store it in here and reuse it until the next time it changes.”

So now the game isn’t so careful anymore, we’re back in unplanned feature territory. But if the caching (storing/reusing calculations) happens at different times on different machines, you get slightly different results (like asking a developer for something before and after they had coffee).

Slightly different results are what OOS thrives on! Clients and servers have 0.0001 cost difference, compounded over time, that corvette is bought on the server but not on the client.

So you remove your “smart” algorithm. You replace it with the correct algorithm. You lose half of what you gained in step 2 and reintroduce some bugs. Probably.
Rinse and repeat.

But enough about my morning routine! Let’s talk about…

Performance
Stellaris fans are like C++ programmers: performance is always on their mind. To be fair, it has also been on ours a lot lately. We know that it’s not all that it could be, especially in late game and with the bigger galaxies. With that in mind, we’ve taken time to improve performance in a bit more depth than we usually can. We looked at what was taking the most time, and as everyone knows that is…

G3Zg2ENmwufWgqUXGFjTEebkxlbQzYRGI0diuSOCrFfUcSl9Xn8EkYCyzAUtWAyCdVXt5biT3vv65T4n-EnA5YmHZXb_Gpp9ydvqh28lj_Oa7py3yU3MHETwURjuo1QD4sFZiZNB


Pops.

There are many reasons why pops consume a lot of time in Stellaris, but the main one is that by endgame we have SO MANY of them. SO So so so so many. And they do so much! Pops have to calculate how good they’d be at every job (they do so every 7 days). Then they have to fight every other pop on the planet to get the job they’re best at. They also have to check if they could have a specific ethic. If they could join a specific faction. How happy they are. How happy they could be. How happy they would be on that planet over there.
All these things trigger modifiers calculations. If you remember my last dev diary, you know that modifiers are the only thing more numerous than Pops in Stellaris. And they all depend on each other. Calculating them is like pulling on a thread and getting the whole sweater.


OK, but what did we actually do about it?
Well first, I’ll admit I may have been a bit pigheaded on the whole “we need to do the jobs distribution every day because we don’t know when new jobs are added.” We reexamined this assumption, and jobs distribution is now only done on demand. It was also rewritten to iterate over a lot fewer things.

We also noticed a few triggers going through every pop of an empire to check if one or more are enslaved, decadent, or other things that can be tested at the species level. So we made new triggers to test these things at the species levels. In the same spirit, we had events going through every ship to find a fleet, so we added triggers at the fleet level.

Second, We’ve also reworked the approach to checking if pops can change ethics (and also made it work again), or if they can join factions.

Finally, we’ve looked for (and found) opportunities to use more multi threading.

But enough talk! What’s the result? Well, if a picture is worth a thousand words, here’s the answer at 30000 words a second:


The video compares the performance of 2.5.1 “Shelley” to 2.6 “Verne'' when running a save game from the community, which can be found attached to this post, with over 20000 pops. It was recorded on my work computer (Intel Core7-7900X @ 3.30Ghz, 10 cores and 20 threads, and AMD R9 Fury). You won’t necessarily get the same results, the exact difference in performance will vary with your computer, and the exact situation in your own save games, of course. On average, we’ve found something between 15% and 30% improvement in late game situations.
This save is just ideal to showcase the impact of the pops improvement.

DYxcPB_pqZfHKxxtAj0sh_Y3nx7zXM4OMcUHTkgNsDK9csuQgEECkgc6jVmUEgWpoa6lD2e9kfYdssD61j2I57mhM0XcyT20wfu8fFIZbP-Usqnw2PShuEAD0_-n-ZTNFcH0NJR6


What is this average anyway? How do you know?
Well, we have synths playing the game all night, every night. In the morning, we check how far they were able to go. We also ask them how many errors they encountered, what their endgame looked like, whether they got any OOS and then put all of that in tables and graphs, with many colors. Then we wipe the synths, so they don’t ask pesky questions about souls and whatnot.

EwNw1Mhvr5FLcwYQYuZClsoMxr8qHs3nF3VPqExEcAJrWCvISTEc2fcl3fNLWzQlWKdxuDLAGHEagL9FXOrtio6XazmKpx_rsR7Ri58Ts2tFbq7OcWPdsIG_ayumIutkMGm2VnD_


In conclusion
Although we keep performance in mind and do our best to keep it reasonable, we’re happy we had a chance to take a deeper dive into the issue. Hopefully the changes will spark as much joy for you as it did for us, and we’re looking forward to your feedback!

Next week will feature another dev diary about the other thing you’ve all been waiting for… MORE PLATYPI!

PS: The save file we're using is from the community, one of the performance threads. We are however unsure where we originally got it from. So if you recognize it, or if it's yours please tell us so we can credit you properly.
 

Attachments

  • perf_massive.sav
    4 MB · Views: 289
Last edited:
  • 1Like
Reactions:
Has the development team considered ways to limit the growth of POPs late game?

From a player stand point, playing with 20000 POPs is no more fun than 2000, actually probably less. I am thinking about mechanics like the Carry Capacity Mod, and possible demographic disasters (the rise and fall of empires).
 
OK, but what did we actually do about it?
Well first, I’ll admit I may have been a bit pigheaded on the whole “we need to do the jobs distribution every day because we don’t know when new jobs are added.” We reexamined this assumption, and jobs distribution is now only done on demand. It was also rewritten to iterate over a lot fewer things.
Forgive me if I'm missing something painfully obvious, but every time I think about this I wonder why anything regarding jobs is checked more often than once per month.

As far as I know, all the consequences of jobs (resources, amenities, pop growth, defense armies and what not) are only checked at end of month, meaning that it is only at that very instant the jobs actually have any influence on the game. It does not matter if half your population is unemployed from the 3rd to the 17th day of the month if everything is brought in order immediately before job consequences are checked - those 15 days of unemployment do not in any way register in the end of month outcome.Similarly, I conclude that there is no point to correctly ordering jobs multiple times per month, as it is only the final ordering immediately before this monthly check that has any impact on the game.

Is it because you think monthly checks would possible confuse players and lead them to wonder why pops aren't moving?
 
Next week will feature another dev diary about the other thing you’ve all been waiting for… MORE PLATYPI!

Maybe I’m missing something but what’s the platypi reference?

This dev diary is great news. I was hoping the next would be AI improvements, both with AI empires and automation features. Lack of good automation is what has stopped me picking up the game for a long time. Managing large empires becomes a chore and sector/planet automation aren’t great.
 
Looks good, and can't wait to try the new version on a new game to see how it will perform. You seem to have adressed the major issue I discovered on pops and jobs.
If we can play games on large with 20k pops until 2600 then this is huge, I will consider supporting the game again with the federations DLC.
It will be a fresh breath of Air if I can finally play without having to be genocidal in order to keep the game rolling.

However, as I pointed out in the performance thread months ago, there are also other issues with performance. A few urgent ones on top of my head are:

1. Using the pop resetlement window is deathly slow and almost unusable if there is a large number of pops. Populating and updating the GUI takes ages, even while the game is paused.

2. xeno compatibility spaming subspecies like crazy.

3. Trade collection calculations and gateway pathing - in combination and seperately.

For all of the above players ended up using mods to remove or short-circuit those features to be able to effectively play:
1. Use mods that auto migrade pops - completely avoid the manual re-settlement gui. OR Stellaris Immortal.
2. Use mods to ban use of the ability.
3. Play gestalts/machine empires to remove all trade calculations (they don't do trade) - OR Stellaris immortal that revamps it.

@Moah Can you please advise if any extra measures were taken in the new patch for the above?
 
15% more than the current state is still a long way from acceptable.

Are there plans to continue focusing on performance to the same extent, or will we return to the status quo of "we are always working on performance"?
But, we ARE always working on performance. Always looking for new things we could improve. Today's improvements don't come out of nowhere, they come out of the work we've put in in the past, etc. Being skeptical about it doesn't make it less true.

can you say something about the improved multithread use? It would be very interesting to know, especially if there is a thread dedicated for checking the jobs or fleet movement. Actually i've looked many times in my performance monitor and one of my threads is at its peak and the other ones are idle the most time. Will you share a glimpse of your wisdom with us? :)
Unfortunately there's not much to say about it. It's not as thorough as most people would like, but there are a couple things we've manage to parallelize, and we've reworked our thread pool to be more reactive. We also improved how we distribute the tasks across the thread pool.
We're still looking into better usage, but we focused on the pops right now, as it was what would give us the biggest win.

Have you ever thought about just reducing the amount of indvidual pops (i.e. lower pop growth rate as well as housing and all the population dependent thresholds, while accordingly increasing the ressources create by jobs?).

An indivudual pop is an abstract unit anyway, so not why make the unit a bit bigger? I wouldnt care if my planets are maxed out late game with 40-60 or 100-120 pops if the ressources provided / supply needed is the same. I mean, before 2.0 the hard cap for pop units per planet was the number of tiles, i.e. 25.
Well there's two different answers to that. Yes, I've thought about it. A lot. It's even be discussed with @grekulf and the design team. But at the same time, as a programmer, it feels bad to have to change the design of the game because of technical limitations. We (The programmers of Stellaris) would much rather have a system where design decides what they want without having to think about technical issues.

Still I'm a bit confused and sceptical about the "jobs distribution is now only done on demand" as the solution to all slow downs. Will this really solve our core problem? I remember the Perfomance Megathread stating the problem of slow downs is not the pops checking for new jobs every day. According to the thread it is rather the pops checking once a new (vacant) job opened up - meaning once a new job is created all pops on the planet check everyday if they should leave their current job and take the new job. (EDIT: Here's GnoSIS post with the analysis: https://forum.paradoxplaza.com/forum/index.php?threads/performance-megathread.1253705/page-22 - Post 427)
Lots of (unemployed) pop were supposed to be even beneficial and their normal regular checks were much less of a problem in comparison.
It's not a "solution to all slowdowns" It's a solution to one of the most common slowdowns, and one that was compounded several times over through various mechanisms. It's not a magical bullet that will allow everyone to run a 10k stars game with 200 AI well into the 3000s.
The video does represent a best case scenario, and I think that games that don't have as many pops but have many fleet won't benefit as much.
We'll still be working to improve performance. My personal goal is to reach a point where people ask for a slower fastests speed, like they have on CK2.

Out of curiosity, are job checks/tests performed on "invalid pops"?
  • E.g. If i have 1000 pops in an undesirables cast [e.g. being purged] on a planet is the game running fewer checks on them vs someone in ruler/specialist/worker cast (on a daily - or weekly/monthly basis, now, i guess?)
Being purged is a job! So yes we also run checks on pops to see if they need to take the "being purged" or "being assimilated" job.

15% and 30% improvement in late game situations
More like 15-30% depending on your specific save and your machine, across the board.
 
Forgive me if I'm missing something painfully obvious, but every time I think about this I wonder why anything regarding jobs is checked more often than once per month.

As far as I know, all the consequences of jobs (resources, amenities, pop growth, defense armies and what not) are only checked at end of month, meaning that it is only at that very instant the jobs actually have any influence on the game. It does not matter if half your population is unemployed from the 3rd to the 17th day of the month if everything is brought in order immediately before job consequences are checked - those 15 days of unemployment do not in any way register in the end of month outcome.Similarly, I conclude that there is no point to correctly ordering jobs multiple times per month, as it is only the final ordering immediately before this monthly check that has any impact on the game.

Is it because you think monthly checks would possible confuse players and lead them to wonder why pops aren't moving?
That's exactly why. Immediate feedback is important to know what's going and plan ahead.
 
That video shows the game running 3x faster in 2.6 than in 2.5 (6 months versus 2). That’s fantastic. It makes the lategsme look playable in even an extreme situation. Brilliant to see, just what I hoped ongoing work on performance would look like. Thank you for this.
 
moah from the graph there are 4 big outliner in performace i cant tell the color and what thing it is. but what could be done to lower that even futher? i prefer having an inefficient game and be able to play. realistically even if there are available jobs people end up unemployed for a long time due to inefficiency, arbetsförmedlingen is a joke so easing up on checks even more i wouldnt mind.
 
could u slow down the checks even more to blurbs point. what if you could retroactively given modifiers and gain "lost" recourses every 2nd month. could you lockin and group pops if they already have jobs on a planet so 20pops show as 1 if they all work on farming. and only do checks if there is a war going on. most of the game you never move pops around. how about a break check if user is doing anything in popscreen
 
i really appreciate the transparancy for once! could you be more open and indepth in a personal blog article once a month. if you could point modders in the right direction of performance hogs and what could theoretically be done to lower latancies. by having an open discussion of where losses occur you could maybe get a fresh opinon on what could be done to futher improve time.
 
Unfortunately there's not much to say about it. It's not as thorough as most people would like, but there are a couple things we've manage to parallelize, and we've reworked our thread pool to be more reactive. We also improved how we distribute the tasks across the thread pool.
We're still looking into better usage, but we focused on the pops right now, as it was what would give us the biggest win.
Thank you for the answer anyways :). Please go on with the good work! But as a general idea, why not dedicate a thread for every specific game task (as long enough threads available of course). That would maybe give much better (balanced) CPU usage and reduce lag. But i'm shure you already had that idea and there are reasons why it didn't happen until now.
 
Great dev diary, glad to see direct proof of improvements being made. It seemed that you were worried about being too technical; you shouldn't be. Most Dev Diary readers would rather get that extra little bit of detail, even if they do have to google a term or two.

Are any of your synths purchased from a certain fruit-themed Megacorp? There have been consistent problems with desyncs and crashes on Macs in MP (Linux too, pretty sure). Purging some more of those unplanned features would be greatly appreciated by both myself and many other mp players.
 
. Populating and updating the GUI takes ages, even while the game is paused.
You just reminded me that the Empire Opinion galactic map filter is a good way to hang a game. Even when paused it grinds my machine down from 60 fps to 2-3 sometimes lol.

As I understand it the game is seemingly constantly (each tick?) checking every empire's opinion against every otther empires opinion/modifiers and then shading them accordingly (it becomes bad when there are many [>20?] empires in play at once).

It makes for a nasty lagfest - kind of reminds me of a chugging excel sheet - and it's a shame, galactic map filters are an excellent way to view the galaxy.

I imagine a lot more people will try and.. experience that laggy opinion filter in federations when they check to see who likes/hates them for diplomacy.

I'd also love to know if we'll see any new map filters with Federations - e.g. one showing which empires owe you favours / which empires you owe favours to (shaded by #favours), or a geographical view of who's voting for, against, abstaining a GC resolution.
 
Last edited: