Stellaris Performance Analysis #1

  • We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.

serpentskirt

Prince with a Thousand enemies
94 Badges
Feb 9, 2014
518
459
  • Europa Universalis IV
  • Europa Universalis IV: Call to arms event
  • Stellaris
  • Europa Universalis IV: Dharma
  • Age of Wonders: Planetfall
  • Hearts of Iron IV: La Resistance
  • Stellaris: Federations
  • Crusader Kings III: Royal Edition
  • Battle for Bosporus
  • Europa Universalis 4: Emperor
  • Stellaris: Necroids
  • Stellaris: Nemesis
  • Victoria 3 Sign Up
  • Hearts of Iron IV: By Blood Alone
  • Hearts of Iron IV: No Step Back
Hello and welcome to Stellaris Performance Analysis #1. In the first issue the test philosophy is explained, some reference data is gathered and a simple study is done.

The scope
Performance of Stellaris is quite a popular discussion topic, however, most of these describe performance issues with the "game lags like hell in the endgame" level of precision.

In order to elevate precision level, it is proposed to record the exact time it takes to reach a certain in-game date and then use this number while comparing different cases.

The method
A special data logger is used to capture:
  • in-game date change;
  • game status (paused/unpaused/speed change) change.
Each such change is logged together with a timestamp. Processing this performance log allows to calculate how much time it takes for one day to pass.

The sampling ratio of data logger is set to 1 kHz.

Additionally, a save watcher runs in parallel and backs up save games in case detailed investigation is required. Finally, the idle mode disabler AutoIt script is activated.

The data logger's and save watcher's impact on PC performance is shown on figure below:
load.gif


The limitations
To minimize impact on data logger performance, the data is logged in binary form. PowerShell parsers convert binary log into csv one with recalculated amount of milliseconds it took to pass from one day to another. This method allows to exclude the time game was on pause, however, the pause effect on game performance is at the moment unknown.

As such, it is recommended to record games without pauses. Also, games should run on same speed as there is no method yet to reliably convert values from one game speed to another.

Test setup
All games were run on the following PC:
  • Core i5-3570K
  • 16 GB RAM
  • GeForce GTX 1060 6 GB
  • Win10
The pre-test sequence is following:
  • the PC is restarted before each logging session;
  • the game is launched via Plaza launcher and set up:
    • desired sliders' positions combination is set;
    • desired game speed is set;
    • game is manually saved;
    • game is left to wait on pause.
  • the data logger is launched;
  • the save watcher is launched;
  • the idle mode disabler script is launched;
  • game can be unpaused and played.
Data processing
MATLAB is used for plots building. The typical plot is looking like that:
typical.png


However, given the data density, the plots are quite hard to comprehend. Gaussian smoothing with window of 720 days (guesstimated value) is used to obtain final plot:
smooth.png

This concludes the testing philosophy.
 
Last edited:
All tests in this post were done for game version 2.1.0 plaza (checksum 0c6d) with following DLCs:
Anniversary Portraits
Arachnoid Portrait Pack
Creatures of the Void Portrait Pack
Horizon Signal
Paradox Account Sign-up Bonus
Symbols of Domination
Apocalypse
Utopia
Humanoids Species Pack
Plantoids Species Pack
Distant Stars Story Pack
Leviathans Story Pack
Synthetic Dawn Story Pack

The game speed is set to 4 (fastest).

The approach
A set of games is run, with some settings being changed from game to game in order to find out their impact on the game performance. The following settings are investigated:
  • Galaxy Size;
  • AI Empires;
  • Habitable Planets.
Automation
In order to speed up the process of data gathering, this set of games is recorded with "human_ai" and "observe" console commands activated, while zoomed-out on the galactical map.
This should also reduce GUI impact on the performance, therefore the term RGI-performance (reduced graphics impact) will be used in the following text.

Reference games
First, a set of games is recorded on three different galaxy sizes.
14_observer_3000_years
Code:
Galaxy Size  Medium (600 Stars)
Galaxy Shape  Spiral (4 Arms)
AI Empires  9   Randomize OFF
Advanced AI Starts 2   Randomize OFF
Fallen Empires  2   Randomize OFF
Marauder Empires 2   Randomize OFF
Tech/Tradition Cost 1x
Habitable Worlds 1x
Primitive Civilizations 1x
Crisis Strength  1x
Mid-Game Start Year 2300
End-Game Start Year 2400
Difficulty  Commodore
Scaling Difficulty Off
AI Aggressiveness Normal
Empire Placement Clusters
Advanced Neighbors Off
Hyperlane Density 1x
Abandoned Gateways 1x
Wormhole Pairs  1x
Ironman Mode  Off
15_observer_huge_galaxy
Code:
Galaxy Size  Huge (1000 Stars)
Galaxy Shape  Spiral (4 Arms)
AI Empires  15   Randomize OFF
Advanced AI Starts 4   Randomize OFF
Fallen Empires  4   Randomize OFF
Marauder Empires 3   Randomize OFF
Tech/Tradition Cost 1x
Habitable Worlds 1x
Primitive Civilizations 1x
Crisis Strength  1x
Mid-Game Start Year 2300
End-Game Start Year 2400
Difficulty  Commodore
Scaling Difficulty Off
AI Aggressiveness Normal
Empire Placement Clusters
Advanced Neighbors Off
Hyperlane Density 1x
Abandoned Gateways 1x
Wormhole Pairs  1x
Ironman Mode  Off
16_observer_large_galaxy
Code:
Galaxy Size  Medium (600 Stars)[/COLOR][/SIZE][/LEFT]
[SIZE=14px][COLOR=rgb(255, 255, 255)]
[LEFT]Galaxy Shape  Spiral (4 Arms)
AI Empires  9   Randomize OFF
Advanced AI Starts 2   Randomize OFF
Fallen Empires  2   Randomize OFF
Marauder Empires 2   Randomize OFF
Tech/Tradition Cost 1x
Habitable Worlds 1x
Primitive Civilizations 1x
Crisis Strength  1x
Mid-Game Start Year 2300
End-Game Start Year 2400
Difficulty  Commodore
Scaling Difficulty Off
AI Aggressiveness Normal
Empire Placement Clusters
Advanced Neighbors Off
Hyperlane Density 1x
Abandoned Gateways 1x
Wormhole Pairs  1x
Ironman Mode  Off

These settings are default for their respective map sizes, the only adjustment being that Difficulty setting has been changed to Commodore.


The reference plot:
reference.png


The graphs show more or less same behaviour - it takes more and more time for each day to pass until situation more or less stabilizes when a crisis starts to murder everyone left and right sometime around 24XXs.

The progression seems to be linear, though it is definitely less prominent in case of medium-sized map.

The study
For this post's survey, the impact of various settings on RGI performance will be measured. The main contributors to slowdown are guesstimated to be as following:
  • Galaxy Size: the more stars there are, the more time it will take for pathfinding/expansion planning/etc. to finish;
  • AI Empires: the more empires there are, the more planning/diplomacy/etc. AI needs to do;
  • Habitable Worlds: the more habitable worlds there are, the more time planetary manager will require to sort things out.
Two reference cases are taken - the medium and the huge maps. Measurements will be done on medium map with settings being gradually set to match ones of the huge map:
  • unchanged number of habitable worlds, same amount of empires as on the huge map;
  • same amount of habitable worlds as on the huge map, unchanged number of empires;
  • same amount of habitable worlds as on the huge map, same amount of empires as on the huge map.

Finally, one measurement on huge map with medium map settings will be done for cross-check.

Remarks on settings
For one or another reason a mistake was made while setting number of empires on medium maps. The number used is 18 while it should be 15. The test outcome indicated this as a non-critical mistake.

As the habitable worlds slider affects map generation algorithm in mysterious and arcane ways, several map generation attempts were made until correct setting value was found.

Study results
Following games were recorded:
18_observer_2x_empires
Code:
Galaxy Size  Medium (600 Stars)[/COLOR][/SIZE][/LEFT]
[SIZE=14px][COLOR=rgb(255, 255, 255)]
[LEFT]Galaxy Shape  Spiral (4 Arms)
AI Empires  18   Randomize OFF
Advanced AI Starts 4   Randomize OFF
Fallen Empires  2   Randomize OFF
Marauder Empires 2   Randomize OFF
Tech/Tradition Cost 1x
Habitable Worlds 1x
Primitive Civilizations 1x
Crisis Strength  1x
Mid-Game Start Year 2300
End-Game Start Year 2400
Difficulty  Commodore
Scaling Difficulty Off
AI Aggressiveness Normal
Empire Placement Clusters
Advanced Neighbors Off
Hyperlane Density 1x
Abandoned Gateways 1x
Wormhole Pairs  1x
Ironman Mode  Off




19_observer_4_25x_planets
Code:
Galaxy Size  Medium (600 Stars)
Galaxy Shape  Spiral (4 Arms)
AI Empires  9   Randomize OFF
Advanced AI Starts 4   Randomize OFF
Fallen Empires  2   Randomize OFF
Marauder Empires 2   Randomize OFF
Tech/Tradition Cost 1x
Habitable Worlds 4.25x
Primitive Civilizations 1x
Crisis Strength  1x
Mid-Game Start Year 2300
End-Game Start Year 2400
Difficulty  Commodore
Scaling Difficulty Off
AI Aggressiveness Normal
Empire Placement Clusters
Advanced Neighbors Off
Hyperlane Density 1x
Abandoned Gateways 1x
Wormhole Pairs  1x
Ironman Mode  Off
20_observer_2x_empires_3_25x_planets
Code:
Galaxy Size  Medium (600 Stars)[/COLOR][/SIZE][/LEFT]
[SIZE=14px][COLOR=rgb(255, 255, 255)]
[LEFT]Galaxy Shape  Spiral (4 Arms)
AI Empires  18   Randomize OFF
Advanced AI Starts 4   Randomize OFF
Fallen Empires  2   Randomize OFF
Marauder Empires 2   Randomize OFF
Tech/Tradition Cost 1x
Habitable Worlds 3.25x
Primitive Civilizations 1x
Crisis Strength  1x
Mid-Game Start Year 2300
End-Game Start Year 2400
Difficulty  Commodore
Scaling Difficulty Off
AI Aggressiveness Normal
Empire Placement Clusters
Advanced Neighbors Off
Hyperlane Density 1x
Abandoned Gateways 1x
Wormhole Pairs  1x
Ironman Mode  Off


22_huge_dumbed_down
Code:
Galaxy Size  Huge (1000 Stars)[/COLOR][/SIZE][/LEFT]
[SIZE=14px][COLOR=rgb(255, 255, 255)]
[LEFT]Galaxy Shape  Spiral (4 Arms)
AI Empires  9   Randomize OFF
Advanced AI Starts 2   Randomize OFF
Fallen Empires  2   Randomize OFF
Marauder Empires 2   Randomize OFF
Tech/Tradition Cost 1x
Habitable Worlds 0.25x
Primitive Civilizations 1x
Crisis Strength  1.5x
Mid-Game Start Year 2300
End-Game Start Year 2400
Difficulty  Commodore
Scaling Difficulty Off
AI Aggressiveness Normal
Empire Placement Clusters
Advanced Neighbors Off
Hyperlane Density 1x
Abandoned Gateways 1x
Wormhole Pairs  1x
Ironman Mode  Off

400 years plot:

study.png


200 years plot:
200.png


Observations
Number of AI empires vs. habitable worlds
Number of AI empires has more impact in the beginning of the game (seen on blue and cyan graphs). Habitable worlds' impact catches up and surpasses extra AI empires only after 150-170 years (seen on white and blue graphs).

Galaxy size
Galaxy size contributes around 10% of RGI performance difference between the medium and the huge map (seen on red and magenta; cyan and green graphs). It tends to get way out of control past 2450, probably due to excess pathfinding caused by crisis arrival and/or unnecessary borders update in 2.1.0.

Notable phenomenon
All the graphs except the magenta one demonstrate a steep rise in 2205-2220 range:
40.png


This correlates with the early expansion phase, and may or may not be caused by creation of first sector.

Comparison with non-observer game
The following plot shows graphs for observer and non-observer games with the same settings:
comparison.png


While the graphs behaviour looks more or less the same, the non-observer case shows quite severe performance degradation. Main culprits are suspected to be: flaws in logging due to game pausing, and GUI impact.

Study outcome
The study provided some reference data to compare against in the future, and its results suggest further investigation paths, such as:
  • performing a similar study on non-observer games;
  • looking into the reason behind the steep rise in the beginning of the game;
  • tracking performance changes between different Stellaris versions.
The performed study, however, has provided only rough data on how and in what measure the performance of the game is affected. It is recommended to increase budget for more tests in this field.
 
The save games of these recordings are available on my OneDrive. Ignore the "1_standard_settings_attempt_one" - that one was logged without sleep mode disabler script and data has suffered from additional lag.

Enjoy your DD replacement!
 
Last edited:
Whoa, great work!

Nice to get a systematic measurement how the inefficient GUI (tooltip generation?) is hindering things. Did you keep the outliner open in your non-observer test? Would be interesting to see how much impact that has.
 
Define "actual gameplay".
Like, were you actually playing the game like a normal person, or were you just goldfishing, so your empire remained as an OPM and none of your ships moved 2200-2500?

I ask because the hypothesis has been put forward previously that essentially all lagging complaints are due to increasing numbers of objects to render in the system view.
So if all these games are just you sitting on galaxy view and watching the world go by, including the one labelled "actual gameplay", you are totally missing the cause of the problem every time and none of these simulations are germane to the actual complaint.
 
Define "actual gameplay".
Like, were you actually playing the game like a normal person, or were you just goldfishing, so your empire remained as an OPM and none of your ships moved 2200-2500?

I ask because the hypothesis has been put forward previously that essentially all lagging complaints are due to increasing numbers of objects to render in the system view.
So if all these games are just you sitting on galaxy view and watching the world go by, including the one labelled "actual gameplay", you are totally missing the cause of the problem every time and none of these simulations are germane to the actual complaint.

Yeah playing with 1fps is truly nice with the huge galaxy performance.
 
I don't think it make much sense to talk about pausing and its impact, if he didn't actually play the game...
And he doesn't talk about pausing and its impact, other than to say that "the effect is unknown", so what's your point?
Also, there's "play" and there's "play". If you leave the game running at speed 4 and go take a nap, the game is playing but the player is not, and since nobody cares about performance issues if they're having a nap, I question the relevance of this study wholesale if, indeed, that's the definition of "play" that's being used.
AAAAAALSO in the section on Automation it's implied that he does just goldfish ("all games are zoomed out on the galaxy map"), but since the remarks on the non-observer game come some time after that, I wanted to clarify.
 
Did you keep the outliner open in your non-observer test?
Define "actual gameplay".
Like, were you actually playing the game like a normal person, or were you just goldfishing, so your empire remained as an OPM and none of your ships moved 2200-2500?
The game labeled as "actual gameplay" was played normally, with pausing, diplomacy and warfare; the outliner was indeed present. I tend to play tall, so in the first 100 years I colonized only 7-8 planets and then switched to building ringworlds. I killed several leviathans and stopped playing shortly after destroying first sterilization hub.

I thought AI empires don't use sectors at all?
I'm 80% sure they do use them, but cannot show proof right away. Anyway that's just speculation, it can be related to something else (e.g. pirate spawn calculations) or be a fancy glitch.

I ask because the hypothesis has been put forward previously that essentially all lagging complaints are due to increasing numbers of objects to render in the system view.
I suspect the top bar numbers as another major contributor, so next set of tests are envisioned to be done with "human_ai" but without "observe". I think it will allow to estimate impact of planets/systems on performance with quite good precision.

At the moment I don't have reliable way of measuring fleet rendering impact without actually playing the game. I think forcing camera to be attached to biggest fleet might work, but that will require making another tool.
 
I suspect the top bar numbers as another major contributor
Really? By what mechanism? The difficulty of processing many thousands of sources of income as opposed to just, what, ~10 at the start of the game? Or some sort of under-hood integer overflow when the sum gets too large?

At the moment I don't have reliable way of measuring fleet rendering impact without actually playing the game. I think forcing camera to be attached to biggest fleet might work, but that will require making another tool.
Yeah, something like that would probably work.
Don't get me wrong, I think this is an impressive technical analysis so far. I'm just sorta brainstorming sources of slowdown that the current methodology wouldn't catch.
 
Really? By what mechanism? The difficulty of processing many thousands of sources of income as opposed to just, what, ~10 at the start of the game? Or some sort of under-hood integer overflow when the sum gets too large?
My gut feeling. I've played a lot of one system games and their plots look quite similar to ones from observer runs. Even in non-one system games I tend to have almost non-existent fleet in the first 100 years, so placing blame on expansion is somewhat rationalized.

There is plenty of room for non-optimization in updating top bar. For example, you may update minerals income by looping through all the planets, mining stations, starbases etc. or you may save the value and update it only when POP is moved or station is destroyed - something like that was mentioned in 2.1.1 patch notes, if I read it correctly. Some stuff like unity modifiers per planet type can snowball quite quickly if its calculation is inefficient.

Anyway, all that is pure speculation, but one that I can test and actually prove or disprove.

Yeah, something like that would probably work.
Don't get me wrong, I think this is an impressive technical analysis so far. I'm just sorta brainstorming sources of slowdown that the current methodology wouldn't catch.
No worries, the feedback is always welcome. It will be great, if we'll find a clever way of testing some tricky aspect of the game.
 
This is great! Real data.

An interesting control to minimize variability could be to run a single empire with no competition. Then habitable planets | galaxy size | star lane connections can all be varied in a somewhat isolated controlled experiment to see what effects they have on a single empire. They wouldn't represent real game values, but they should provide some isolated performance scaling info, which is hard to do with all the variables in a normal game.

An interesting number of civs test might be to create a custom map where they are all isolated from each other- I'm thinking zero star lanes ]:) they *should* run faster bc they have less interesting choices but then maybe they run longer bc the AI spins trying to find interesting choices which aren't there.
 
Really? By what mechanism? The difficulty of processing many thousands of sources of income as opposed to just, what, ~10 at the start of the game? Or some sort of under-hood integer overflow when the sum gets too large?

GUI updates probably aren't multithreaded, so if they're doing lookups and recalculating the figures to display every tick while redrawing the UI, it's going to get worse as you expand. The top bar is probably not a huge part of it, but then there's the outliner etc, which all adds up.

Probably not so much the arithmetic itself but rather doing the lookups through the database of ships/planets/pops etc.
 
Great work! I'm very interested in that huge spike in the first graph around 2440.
Are you planning to do the same work on 2.1.x on any chance? would love to see the evolution.

Also, to reduce the impact of graphics even more you could use the "overnight" command instead of "observe" as it reduces the graphical updates.
However, it is fair to say that decent amount of "lag" players perceive is due to the rendering of many fleets (at times), so we might want to measure this as well.
 
An interesting control to minimize variability could be to run a single empire with no competition.
Yep, had the same idea. Mainly to avoid situation when player empire might get eliminated.

Also, to reduce the impact of graphics even more you could use the "overnight" command instead of "observe" as it reduces the graphical updates.
However, it is fair to say that decent amount of "lag" players perceive is due to the rendering of many fleets (at times), so we might want to measure this as well.
Thanks, I somehow missed that command!

Are you planning to do the same work on 2.1.x on any chance? would love to see the evolution.
I'll record another 2.1.0 game, now with "overnight" command for future references and move on to 2.1.1.

Great work! I'm very interested in that huge spike in the first graph around 2440.
Spikes? I have them.

15_observer_huge_galaxy (numerous 20+ seconds spikes, L-cluster open, prethoryn scourge)
Non-smoothed plot:
15_observer_huge_galaxy.png
Screenshot:
15_observer_huge_galaxy_screen.png

22_huge_dumbed_down (numerous 15+ seconds spikes, L-cluster open, prethoryn scourge)
Non-smoothed plot:
22_huge_dumbed_down.png
Screenshot:
22_huge_dumbed_down_screen.png

14_observer_3000_years (no spikes over 5 seconds, L-cluster not open, contingency)
Non-smoothed plot:
14_observer_3000_years.png
Screenshot:
14_observer_3000_years_screen.png

18_observer_2x_empires (some spikes over 5 seconds, L-cluster not open, unbidden)
Non-smoothed plot:
18_observer_2x_empires.png
Screenshot:
18_observer_2x_empires_screen.png

19_observer_4_25x_planets (some spikes over 5 seconds, L-cluster open, contingency)
Non-smoothed plot:
19_observer_4_25x_planets.png
Screenshot:
19_observer_4_25x_planets_screen.png

20_observer_2x_empires_3_25x_planets (some spikes over 5 seconds, L-cluster not open, contingency)
Non-smoothed plot:
20_observer_2x_empires_3_25x_planets.png
Screenshot:
20_observer_2x_empires_3_25x_planets_screen.png

Though, don't know if it's still relevant, I had impression that 2.1.1 fixed majority of these.

As for now, only 14_observer_3000_years has uploaded fully, 15_observer_huge is missing first 100 years. These are located on my OneDrive. As well attached csv files for some excel fun, if needed.
 

Attachments

  • 14_observer_3000_years_performanceMatlab.csv
    4,3 MB · Views: 17
  • 15_observer_huge_galaxy_performanceMatlab.csv
    2,1 MB · Views: 17
  • 18_observer_2x_empires_performanceMatlab.csv
    2,4 MB · Views: 17
  • 19_observer_4_25x_planets_performanceMatlab.csv
    2,7 MB · Views: 18
  • 20_observer_2x_empires_3_25x_planets_performanceMatlab.csv
    2,3 MB · Views: 17
  • 22_huge_dumbed_down_performanceMatlab.csv
    2,5 MB · Views: 18
Can you explain a bit more about how you sampled the data? I would love to run these same tests locally on different hardware.