Stellaris Performance Analysis #2

  • We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.

serpentskirt

Prince with a Thousand enemies
94 Badges
Feb 9, 2014
518
459
  • Europa Universalis IV
  • Europa Universalis IV: Call to arms event
  • Stellaris
  • Europa Universalis IV: Dharma
  • Age of Wonders: Planetfall
  • Hearts of Iron IV: La Resistance
  • Stellaris: Federations
  • Crusader Kings III: Royal Edition
  • Battle for Bosporus
  • Europa Universalis 4: Emperor
  • Stellaris: Necroids
  • Stellaris: Nemesis
  • Victoria 3 Sign Up
  • Hearts of Iron IV: By Blood Alone
  • Hearts of Iron IV: No Step Back
Hello and welcome to Stellaris Performance Analysis #2. This issue is split in two (update: three) parts - one compares performance of different Stellaris versions; the other shows how some different galaxy setups impact performance of version 2.2.3. Update: third part shows impact of population and wars on performance. Same method is used as described in first issue. Unfortunately, save games will not be uploaded this time as uploading 22 GB will take some outrageous amount of time. However, specific saves can be available by request.

It should be specially highlighted once again that what is measured is, in fact, performance in observer mode. This doesn't translate one to one to actual gameplay performance but provides rather convenient way of obtaining data and is allows to measure impact of different game settings.

As well, a special remark about stuttering - data logger doesn't care if it took a minute (exaggerated) for a day to change with smooth 60 fps or with 5 fps.

Performance of different versions

After reading first issue, one might raise a rightful doubt about making conclusions based on single sample. To somewhat amend this, a set of 4 games is recorded and a mean value then is used for further comparison. While that is still quite small sample size, it should at least mitigate deviations due to activation of some hypothetical background services (e.g Windows Update, malware scanning, cat walking on keyboard).

Once again the performance is recorded in observer mode on the same settings: reset to default, set spiral 4 arms, set commodore difficulty, UNE empire. Versions covered: 2.1.2, 2.1.4, 2.2.2.1 and 2.2.3 (all plaza versions). Raw data and smoothened one are combined in single plot. Some plots have remarks.

210-1.png

A single game session from previous recordings.

212-1.png

212-2.png

212-3.png
A good example of crisis doing its job properly - due to total annihilation game was able to run smoothly all the way to year 2900.


212-4.png
Another example of total wipeout, running like a charm to year 3000. Crisis arrival caused some slowdown this time.

212-comparison.png

2.1.2 games showed more or less same performance all the way until crisis arrival in 2420s. Post crisis the plot behavior is somewhat similar, which is not a huge surprise given that all the crises were of the same type.

214-1.png

214-2.png
Second scourge appearance and this time with dramatic entrance - that's quite a spike even on smoothened plot.

214-3.png
For a change, spike occurred way past crisis arrival.

214-4.png

214-comparison.png

Again 2.1.4 shows nice performance similarity prior to crisis arrival. Post crisis behavior of scourges differs quite a lot between each other.

2221-1.png
Here one can immediately see increase in spikes quantity and magnitude on non-smoothed plot.

2221-2.png
This one actually doesn't look bad.

2221-3.png
Typical scourge "mountain peaks" are present here and some dramatic jumps around years 2490 and 2620. These kind of jumps were observed when no sleep mode disabler was used during first data logging attempts. This might be caused either by game or by some background process suddenly feeling productive.

2221-4.png
Smooth crisis arrival but a lot of spikes and once again a weird jump around year 2600.

2221-comparison.png

In version 2.2.2.1 the deviation in performance between different runs occurs much earlier and is more severe - there might be more than 100ms difference between different games prior to crisis arrival. It looks like scourge crisis is more performance-heavy than unbidden.

223-1.png

223-2.png

223-3.png

223-4.png

223-comparison.png


Doesn't look like a lot of improvement compared to 2.2.2.1, but one should keep in mind remark about stuttering not being shown in these plots.

version_comparison.png

Clear winner is version 2.1.2 with 2.1.0 and 2.1.4 being a bit behind. Performance in 2.2.2.1 and 2.2.3 took quite a hit, the observer mode runs almost twice as slow in 2500 compared to previous versions.

While 2.2.3 seems to fixed early game performance, not much of improvement is seen past year 2400.

@Moah you were interested in version performance comparison, please find it above.

2.2.3 custom setups

Some custom galaxy setups were recorded to check what is the impact of certain game features. The setups are going to be briefly described first and their impact will be given next to comparison plot for better accessibility.

223-noholes.png
This setup has wormhole and gateway sliders set to 0.

223-nomarauders.png
This setup has marauder empires set to 0.

223-nomigration.png
This setup has marauder and advanced empires set to 0. All the empires have Inwards Perfection civic by using forced spawn of custom-made empires.

223-custom-zoomed.png

First 200 years.

223-custom.png

400 years in.

First custom setup was aimed at seeing the effect of wormholes and gateways as they were reported numerous times in performance issues threads. This brings performance to 2.1.4 levels for the first 100-150 years, but by year 2450 it almost catches up with regular games. The catch-up is most probably caused by FEs starting to build gateways, so it does indeed look like wormholes/gateways screw up performance.

Having overlooked the fact that FEs can build gateways (playing game is actually useful, should not only run test cases) second setup was made with assumption that migration might be an issue, and marauders were cut out of the galaxy to reduce slaves/refugees stream from raids/horde. This resulted in smoothest run in 2.2.3 so far.

Looking more into it, the first 150 years could be explained quite easily - in early game marauders are the sole users of wormholes. Most probably there were no gateways built (to be checked) by FEs either and the crisis was Unbidden - looking back at all the 2.2.2.1 and 2.2.3 data there is tendency for games with Unbidden to run faster. No surprises with that, as Scourge fails at properly wiping out planets.

But at that time the fact about FEs ability to build gateways/wormholes was still overlooked, so final setup was meant to be final nail in the migration coffin - all empires in the galaxy are Inwards Perfectionists, meaning no migration and refugees. Marauders were turned off as well, the future looked bright.

The result was the worst performance of all custom setups. Despite not having early wars, marauding raids (and thus as well early wormhole users) this galaxy matched regular setups in terms of performance. The most obvious explanation is the sheer amount of pops - being Inwards Perfectionist on top of Xenophobe cranks up pop growth ratio so it's a fair assumption.

Update: Link to part 3 (population and wars taken into account)

Results

Bottom line, the performance offenders so far identified in latest beta:
  • Wormholes and gateways (update: highly unlikely)
  • Scourge (update: and contingency) failing at properly purging
  • POPs related routines once enough have been grown
 

Attachments

  • 223-custom-zoomed.png
    223-custom-zoomed.png
    185,6 KB · Views: 103
  • csv.7z
    9,4 MB · Views: 45
Last edited:
i suppose they have gradually decreased fleet sizes so they have more processing power for something like the new and pointless pop system, sadly it doesn't work out as it seems
 
Thanks for the great testing and analysis. It's starting to become clear that there isn't one single cause of the performance issues in 2.2, but rather a culmination of changes made over the last few versions has slowed down a game that already ran somewhat poorly to start with.

Hopefully Paradox takes some real time to work on not only bugs, but clearing out some of the technical debt on the performance side, beyond just making the game play at 2.1 performance.
 
Nice to see what everyone's talking about visualized and quantified. This latest update is a an absolute mess and piles useless calcs on a system that was already at heavy risk of CPU throttling. All while being sold as the solution to "performance"...so instead of a few hundred pops per empire tops you now can have many empires that easily run into the thousands. With each individual pop checking for jobs every tick...trade calcs/pathfinding every tick....tick tick tick until top stops.

On top of the end game crisis being unable to purge and save the galaxy from lag....how does this go live?
 
Is that confirmed to be the case? It seems unnecessary.

It seems crazy! I don’t understand how anyone could’ve thought this new system would save on calcs. The old tile system was route simple. A tile was either occupied or unoccupied. Pop movement was confined to immigration. They never switched. The complaint was it eventually became a pain to min max your pops. But that was fixable.

Here? This new system is more expensive computationally while offering the player less control!
 
Thank you for another excellent analyses.

I take it that the time in ms, is the time it takes to process each day/tick, it is interesting that 2.2.3 is actually slightly worse than 2.2.2, though I personally experience less stutter, and I will gladly sacrafice a bit of play speed for a smoother play experience, though both departments could use some attention.

I'm curious why you didn't run a test of gestals-only empires to elleminate trade calculations, would be curious to see how big an impact that would have on performance.

After seeing this I think my next game I will set habitable planets to 0.5 or 0.25 to reduce number of pops in the galaxy, hopefully that will help somewhat.
 
Thanks for the great testing and analysis. It's starting to become clear that there isn't one single cause of the performance issues in 2.2, but rather a culmination of changes made over the last few versions has slowed down a game that already ran somewhat poorly to start with.

Hopefully Paradox takes some real time to work on not only bugs, but clearing out some of the technical debt on the performance side, beyond just making the game play at 2.1 performance.

Performance doesn't sell DLCs as well as sparkly features.

It may come to a point where the monetary cost of buggy and badly performing software is high enough (ie: people stop buying the DLCs and the reputational damage may effect other games sales) that those who allocate development resources will have to address the issue. Until that pain becomes real enough few resources will be concentrated on performance, and even then they just need to do the bare minimum to keep the sales up. Unfortunately there's another effect that if people stop buying DLCs because the performance is so bad, then the executives might pull development entirely and go onto the new game that will get higher revenues.
 
All while being sold as the solution to "performance"...
I'm actually curious to find if it was stated in any of the DDs that performance would be better.
I take it that the time in ms, is the time it takes to process each day
Time to go from one day to the next.
it is interesting that 2.2.3 is actually slightly worse than 2.2.2, though I personally experience less stutter,
That's why there is remark on top about stuttering not being measured by such approach.
I'm curious why you didn't run a test of gestals-only empires to elleminate trade calculations, would be curious to see how big an impact that would have on performance.
I just wanted to publish what has been accumulated so far, as amount of media to slap on the post was quite horrifying already. I can run this overnight and next day, probably another session during the weekend. Gestalts-only with and without wormholes/gateways?
 
I'm actually curious to find if it was stated in any of the DDs that performance would be better.

Time to go from one day to the next.

That's why there is remark on top about stuttering not being measured by such approach.

I just wanted to publish what has been accumulated so far, as amount of media to slap on the post was quite horrifying already. I can run this overnight and next day, probably another session during the weekend. Gestalts-only with and without wormholes/gateways?

I can't get over that performance graph....they managed to double the load on a game that was already in grave danger of being CPU locked. And they didn't even bother to make sure the end game "cleanser" that swoops in to save the galaxy from lag-lock worked. #TODOed out. Worse I don't see how the new system will ever be able to approach the old in terms of performance.
 
Have you tried doing some experiments with Glavius AI mods active?
Not yet.

@Volapyk the first gestalt game was finished overnight, thanks to grey tempest killing everyone:
223-custom_plus.png


And pruned version of first 200 years:
223-custom_plus_pruned.png


I created 5 hive minds including devouring swarm and 4 machine intelligences, all were forced to spawn. Rest of the settings were the same as on reference runs - so in this game there were marauders and wormholes/gateways.

It follows no wormholes case quite nicely up to and including khan event in 2300s (compare to smooth curve in no-marauders game).

I think khan was quite efficient at conquering, as there is significant improvement in performance after his conquests - most likely because gestalt pops died after being conquered. To tell that for sure I'll need to write some batch save parser to be able to see correlation between performance and pops number though.

I'm wondering if there will be more improvements w/o wormholes/gateways and/or marauders. Gestalt-only game without holes/gateways but with marauders is being recorded at the moment.
 
I'm actually curious to find if it was stated in any of the DDs that performance would be better.

The only mention of this I remember, is something related to AI's performance would be better, as it would have an easier time figuring out this system than the tiles.

EDIT: Oh wauv, that was fast, and well timed writing a reply and you post results :D

It seems damn close to 2.1.4 actually, I would guess the spikes are wars, which would make sense running slower
 
It seems damn close to 2.1.4 actually, I would guess the spikes are wars, which would make sense running slower
Wars, special events or just some artifacts. Hard to tell without good save game analyzer, looking for these manually takes astronomical amount of time. I'll see if dashboard mod can be used for some quick and dirty information extraction.
 
Initial thoughts then:

Wormhole/gateway issue might be caused by some expensive pathfinding possibly even an infinite loop bug. Population issue might be caused by excessive checking of pops, or indeed more pops. The first issue seems fixable, as it's an algorithm implementation bug, most likely. However I'm not sure about the second system. If it's an implementation bug, them it might be able to be fixed. However, if it's a design issue i.e. we just can't work with so many pops, I'm not sure the solution is palatable to the team.
 
Initial thoughts then:

Wormhole/gateway issue might be caused by some expensive pathfinding possibly even an infinite loop bug. Population issue might be caused by excessive checking of pops, or indeed more pops. The first issue seems fixable, as it's an algorithm implementation bug, most likely. However I'm not sure about the second system. If it's an implementation bug, them it might be able to be fixed. However, if it's a design issue i.e. we just can't work with so many pops, I'm not sure the solution is palatable to the team.

If its releated to vastly increased the number of pops then there’s only one solution to that problem - smash the number of pops back down. In the past that was kept in check by the 25x tiles max per planet. Now it’s unbounded.

Merge pops into “Megapops” at 10:1,100:1 and make em behave as one big pop or something. That or hard cap pop numbers. Why they thought adding more calcs (job checks) for a greatly expaned (and indeed potentially endless) number of pops for a game that’s already cpu bound is beyond me.
 
Wormhole/gateway issue might be caused by some expensive pathfinding possibly even an infinite loop bug.
Well, the pathfinding problem is still listed as test task on PDX career site for developer positions, hopefully it will be resolved soon.