• Crusader Kings III Available Now!

    The realm rejoices as Paradox Interactive announces the launch of Crusader Kings III, the latest entry in the publisher’s grand strategy role-playing game franchise. Advisors may now jockey for positions of influence and adversaries should save their schemes for another day, because on this day Crusader Kings III can be purchased on Steam, the Paradox Store, and other major online retailers.


    Real Strategy Requires Cunning

AlphaAsh

Miserable Git
41 Badges
Mar 16, 2015
1.199
700
www.alphastrikegames.com
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Stellaris: Megacorp
  • Stellaris: Leviathans Story Pack
  • Stellaris: Ancient Relics
  • Stellaris - Path to Destruction bundle
  • Stellaris: Lithoids
  • Stellaris: Distant Stars
  • Stellaris: Humanoids Species Pack
  • Stellaris: Synthetic Dawn
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Apocalypse
  • Shadowrun Returns
  • Shadowrun: Hong Kong
  • Shadowrun: Dragonfall
  • BATTLETECH
  • BATTLETECH - Beta Backer
  • BATTLETECH - Backer
  • BATTLETECH: Season pass
  • BATTLETECH: Flashpoint
  • Surviving Mars
  • Surviving Mars: First Colony Edition
  • Cities: Skylines Deluxe Edition
  • Cities: Skylines
  • Cities: Skylines - Parklife Pre-Order
  • Cities: Skylines - Parklife
  • Cities: Skylines - Green Cities
  • Cities: Skylines - Campus
  • Cities: Skylines - After Dark
  • Cities: Skylines - Snowfall
  • Cities: Skylines - Natural Disasters
  • Cities: Skylines - Mass Transit
  • Cities: Skylines Industries
  • Pillars of Eternity
  • Tyranny: Archon Edition
Really good gen sir.

Certainly, there's going to be a lot of improvement possible at the architectural level of the game, especially considering how old the engine is. However, this is where business realities really do become the limiter.

If you review updates for PDS' various games, you'll see a lot more investment in trying to improve the performance of games at a higher level. This is 'cheaper', and has proven to be an effective way to get short-term improvements in performance on more easily quantifiable issues (as they are exposed and perceivable by the player, and to a greater degree, the modder).

This is also the level where myself and other modders can work.

Your analysis is informative, but I'm not sure it will really influence PDS' methodology for addressing performance concerns with a game.
 

GnoSIS

First Lieutenant
18 Badges
Jan 27, 2009
289
298
  • Victoria 2: A House Divided
  • Victoria 2: Heart of Darkness
  • Warlock 2: Wrath of the Nagas
  • Victoria: Revolutions
  • Stellaris: Leviathans Story Pack
  • Stellaris: Lithoids
  • Stellaris: Ancient Relics
  • Stellaris: Megacorp
  • Stellaris: Distant Stars
  • Stellaris: Apocalypse
  • Stellaris: Humanoids Species Pack
  • Stellaris - Path to Destruction bundle
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Galaxy Edition
  • Stellaris
  • Warlock 2: The Exiled
  • Victoria 2
  • Stellaris: Synthetic Dawn
As I said:

In conclusion if you are going to PDXcon please ask the director of stellaris this: Is he expecting this game to last 5-8 more years worth of DLC and if yes, are they going to take a cost hit and improve the base engine to have a playable game from start to finish? I understand that it's impossible to expect a smooth experience with 2 million pops, but it should be able to cope with 50-100k. If they can't do that, then they should scale down the complexity and simplify systems to provide a solution.

My engineering insights if you may:

The pop data are not perused all at the same the time, the game has indeed split them up so even with 2M pops if you calculate a portion each day of the month, that would be 2M /30 = 66.666 pops per game day. If we then consider multithreading x10 that would be 6.666 per GAME DAY not FRAME, which is doable on potato machines without any specialized cache, so the f*up is somehwere else, or the game tries to evaluate pops with interpreted scripts and multiple (dozens of) memory redirections, and not optimized C++ code. This paragraph doesn't include multiplayer considerations and the devs can possibly include memory/computation tradeoffs in their algos to make the engine faster at the expense of ram so that a 16 gig rig wouch chop up a galaxy with 500k pops on each frame. New processors come with 16 cores/32 threads and will enter the budget range any time now so they need to wake up and smell the coffe, because imagine the above calculations at 30 threads. Hello 5-10M pops!

I expect that the engine rewrite the post(s) above imply, will happen for clausewitz 2020+ and stellaris 2. If we see a really dramatic improvement before S2 I will be amazed.
 
  • 1Like
Reactions:

Chaines

Corporal
24 Badges
Sep 26, 2019
49
0
  • Stellaris - Path to Destruction bundle
  • Stellaris: Megacorp
  • Shadowrun: Dragonfall
  • Stellaris: Distant Stars
  • Cities: Skylines - Parklife
  • Cities: Skylines - Parklife Pre-Order
  • Stellaris: Apocalypse
  • Stellaris: Humanoids Species Pack
  • Age of Wonders III
  • Prison Architect
  • Stellaris: Ancient Relics
  • Stellaris: Leviathans Story Pack
  • Stellaris: Digital Anniversary Edition
  • Tyranny: Archon Edition
  • Tyranny: Archon Edition
  • Stellaris
  • Pillars of Eternity
  • Cities: Skylines
  • Teleglitch: Die More Edition
  • Stellaris: Lithoids
  • Stellaris: Synthetic Dawn
  • Crusader Kings II
  • BATTLETECH
  • Magicka
In the "Starnet AI mod is excellent" thread, some people seems to be able to play game in the scale of "1000 star galaxy with 24 regular empires, 4 fallen empires and 2 marauder empires. Everything else is standard except for 1.25x habitable planets with guaranteed habitable worlds turned off and 1.25x hyperlanes."

how is that even possible ? I would LOVE to play such games, but on my powerfull enought PC i can't play bigger than small galaxie, and even there i tend to play empire with only one specie. Only difference seems to be the AI mod, but i'm reluctant to play with mod. (i know i may be wrong, but i'm only discovering this game and want to play it plenty before i do)
 

exi123

Captain
26 Badges
Jan 19, 2018
391
617
  • Cities: Skylines
  • Stellaris: Lithoids
  • Stellaris: Ancient Relics
  • Cities: Skylines - Campus
  • Prison Architect
  • Stellaris: Megacorp
  • Cities: Skylines Industries
  • Stellaris: Distant Stars
  • Cities: Skylines - Parklife
  • Cities: Skylines - Parklife Pre-Order
  • Stellaris: Apocalypse
  • Stellaris: Humanoids Species Pack
  • Cities: Skylines - Green Cities
  • Cities: Skylines - Mass Transit
  • Stellaris - Path to Destruction bundle
  • Cities: Skylines - Natural Disasters
  • Stellaris: Leviathans Story Pack
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Galaxy Edition
  • Cities: Skylines - Snowfall
  • Cities: Skylines - After Dark
  • Stellaris: Synthetic Dawn
  • Stellaris: Galaxy Edition
  • Stellaris
  • Cities: Skylines Deluxe Edition
In the "Starnet AI mod is excellent" thread, some people seems to be able to play game in the scale of "1000 star galaxy with 24 regular empires, 4 fallen empires and 2 marauder empires. Everything else is standard except for 1.25x habitable planets with guaranteed habitable worlds turned off and 1.25x hyperlanes."

how is that even possible ? I would LOVE to play such games, but on my powerfull enought PC i can't play bigger than small galaxie, and even there i tend to play empire with only one specie. Only difference seems to be the AI mod, but i'm reluctant to play with mod. (i know i may be wrong, but i'm only discovering this game and want to play it plenty before i do)
Tahts very subjective. I read the same post and thought the same. It is possible that you wouldnt play this savefile even it runs better and your pc. I have the same problem: A good gaming rig, high performance in many games but stellaris crumbles to death (<- in my subjective view) in lategame. If i see lag in menus and stutter when panning through the galaxy i get very tired of playing further very fast. It isnt fun to interact with game for me so i stop playing.

I would love to play a filled, big Galaxy too. With many Races, Ethics and Dangers. With Empires as strong as me, every war is able to end your existence or freedom.
 

Crazy_Monkey

Second Lieutenant
27 Badges
Sep 24, 2017
113
4
  • Hearts of Iron IV: Cadet
  • Stellaris: Lithoids
  • Stellaris: Ancient Relics
  • Imperator: Rome Sign Up
  • Stellaris: Megacorp
  • Stellaris: Distant Stars
  • Cities: Skylines - Parklife
  • Cities: Skylines - Parklife Pre-Order
  • Stellaris: Apocalypse
  • Stellaris: Humanoids Species Pack
  • Age of Wonders III
  • Stellaris: Synthetic Dawn
  • Stellaris: Leviathans Story Pack
  • Stellaris
  • Cities: Skylines
  • Victoria 2
  • Crusader Kings II
  • Stellaris - Path to Destruction bundle
  • Victoria 2: Heart of Darkness
  • Victoria 2: A House Divided
  • Semper Fi
  • Hearts of Iron III Collection
  • Hearts of Iron III: Their Finest Hour
  • Hearts of Iron III
  • For the Motherland
  • Crusader Kings II: The Old Gods
Let's be real, there is no alternative solution. They tried for years and things keep getting worse. You would trade the joy to see AI economy going to shit (it probably was already shit anyway) for good late game performance, if that would depend on me <<Where do I sign?>>.
I would just argue, that their team isn't extremely good, when it comes to performance. A modder fixed their broken AI after updates so well, they incorporated it into the game. I just think they should either expand their expertise or hire somebody, who is specialized in performance issues. They already doing this seeing they hired somebody just for the UI, because they saw, they don't have the necessary experience.
 

AlphaAsh

Miserable Git
41 Badges
Mar 16, 2015
1.199
700
www.alphastrikegames.com
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Stellaris: Megacorp
  • Stellaris: Leviathans Story Pack
  • Stellaris: Ancient Relics
  • Stellaris - Path to Destruction bundle
  • Stellaris: Lithoids
  • Stellaris: Distant Stars
  • Stellaris: Humanoids Species Pack
  • Stellaris: Synthetic Dawn
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Apocalypse
  • Shadowrun Returns
  • Shadowrun: Hong Kong
  • Shadowrun: Dragonfall
  • BATTLETECH
  • BATTLETECH - Beta Backer
  • BATTLETECH - Backer
  • BATTLETECH: Season pass
  • BATTLETECH: Flashpoint
  • Surviving Mars
  • Surviving Mars: First Colony Edition
  • Cities: Skylines Deluxe Edition
  • Cities: Skylines
  • Cities: Skylines - Parklife Pre-Order
  • Cities: Skylines - Parklife
  • Cities: Skylines - Green Cities
  • Cities: Skylines - Campus
  • Cities: Skylines - After Dark
  • Cities: Skylines - Snowfall
  • Cities: Skylines - Natural Disasters
  • Cities: Skylines - Mass Transit
  • Cities: Skylines Industries
  • Pillars of Eternity
  • Tyranny: Archon Edition
I would just argue, that their team isn't extremely good, when it comes to performance. A modder fixed their broken AI after updates so well, they incorporated it into the game. I just think they should either expand their expertise or hire somebody, who is specialized in performance issues. They already doing this seeing they hired somebody just for the UI, because they saw, they don't have the necessary experience.
The skillset available from their staff is, I think, not my business. You are simplifying a complex business process unfairly, I feel. PDS have and have had specialists in UI and AI, varying as you'd expect in a busy games studio, and they have proven many times that they are 'extremely good' at making successful games in a fairly niche market. Regardless of whatever area of expertise is lacking at any given time and appears to need bolstering, because of some 'issue' arising with one game or another.

tldr - Dismissing the team's competency isn't fair.
 
Last edited:

MichaelJanuary

Colonel
42 Badges
Jul 8, 2012
814
277
  • Crusader Kings II
  • For the Motherland
  • Hearts of Iron III
  • Hearts of Iron III: Their Finest Hour
  • Naval War: Arctic Circle
  • Semper Fi
  • Sword of the Stars II
  • Stellaris: Synthetic Dawn
  • Stellaris - Path to Destruction bundle
  • Stellaris: Galaxy Edition
  • Surviving Mars: First Colony Edition
  • Stellaris: Apocalypse
  • Stellaris: Distant Stars
  • Cities: Skylines - Parklife
  • Cities: Skylines - Parklife Pre-Order
  • Surviving Mars: Digital Deluxe Edition
  • Imperator: Rome - Magna Graecia
  • Cities: Skylines Industries
  • Stellaris: Megacorp
  • Imperator: Rome Deluxe Edition
  • Hearts of Iron IV: Expansion Pass
  • Surviving Mars: First Colony Edition
  • Stellaris: Lithoids
  • Hearts of Iron IV: La Resistance
  • Hearts of Iron IV: Cadet
  • 500k Club
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Hearts of Iron IV Sign-up
  • Stellaris: Humanoids Species Pack
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Leviathans Story Pack
  • Hearts of Iron IV: Together for Victory
  • Cities: Skylines - Mass Transit
  • Surviving Mars
  • Hearts of Iron IV: Death or Dishonor
  • Hearts of Iron IV: Expansion Pass
  • Imperator: Rome
  • Stellaris: Ancient Relics
  • Cities: Skylines
It seems clear to me that the 'issue' is not the number of pops, although the problem is obviously scaling with number of pops. It feels to me that the actual issue may be one of the following:

1. Too many calls being made to optimise jobs. There should be a limited number of triggers for job optimisation. (Resettle, unemployed, growth, policy change, construction, tech, etc.) All of these would be limited to a single empire, single species, single stratum, or even single planet.
2. The optimise algorithm is possibly too wide, and not properly localised. Result would be unnecessary pops being processed. There are very few events that would require processing more than a few hundred pops.
3. Threshold values, even hidden thresholds, such as a pop that has been subject to an optimisation decision should have a cooldown before being reconsidered. Even if this will result in temporary inefficiencies. This is a last resort though.
4. Deferred processing. Some common events like pop growth, resettle, unemployment, construction, can be deferred to specific days (every X days, or even once a month).
 
Last edited:

Kent_Lang

Legendary Space Dwarf Cheese Maker
85 Badges
Jan 27, 2011
98
71
  • Pillars of Eternity
  • Lead and Gold
  • Magicka
  • Europa Universalis IV: Res Publica
  • Semper Fi
  • Victoria 2: A House Divided
  • Victoria 2: Heart of Darkness
  • Stellaris - Path to Destruction bundle
  • Europa Universalis IV: Pre-order
  • Arsenal of Democracy
  • Magicka 2
  • Cities: Skylines - After Dark
  • Europa Universalis IV: Cossacks
  • Cities: Skylines - Snowfall
  • Europa Universalis IV: Mare Nostrum
  • Stellaris
  • Europa Universalis IV: Third Rome
  • Stellaris: Leviathans Story Pack
  • Hearts of Iron III: Their Finest Hour
  • Crusader Kings II
  • Crusader Kings II: Charlemagne
  • Crusader Kings II: Legacy of Rome
  • Crusader Kings II: The Old Gods
  • Crusader Kings II: Rajas of India
  • Crusader Kings II: The Republic
  • Crusader Kings II: Sons of Abraham
  • Europa Universalis IV: Art of War
  • Knights of Pen and Paper +1 Edition
  • Hearts of Iron III
  • For the Motherland
  • Europa Universalis IV: Wealth of Nations
  • Europa Universalis IV: Call to arms event
  • Europa Universalis IV: Conquest of Paradise
  • Magicka 2: Ice, Death and Fury
  • Stellaris: Digital Anniversary Edition
  • Tyranny: Archon Edition
  • Europa Universalis IV: Rights of Man
  • Hearts of Iron IV: Cadet
  • Hearts of Iron IV Sign-up
  • Victoria 2
  • Crusader Kings II: Conclave
  • Crusader Kings II: Horse Lords
  • Crusader Kings III
  • Warlock: Master of the Arcane
  • Crusader Kings II: Sunset Invasion
  • Crusader Kings II: Way of Life
  • Mount & Blade: Warband
  • Magicka: Wizard Wars Founder Wizard
  • 500k Club
The skillset available from their staff is, I think, not my business. You are simplifying a complex business process unfairly, I feel. PDS have and have had specialists in UI and AI, varying as you'd expect in a busy games studio, and they have proven many times that they are 'extremely good' at making successful games in a fairly niche market. Regardless of whatever area of expertise is lacking at any given time and appears to need bolstering, because of some 'issue' arising with one game or another.

tldr - Dismissing the team's competency isn't fair.
I don't think it's unfair. The reason why Paradox is successful of making games in niche markets is partly because the markets actually are niche and they have a good overarching formula for developing their games. That however doesn't mean that competence can't be called into question. Every individual and also every company has holes in their competence and it's something to be expected. The problem is that for Paradox, neither performance nor AI should be a point where there is competence holes in their company. Stellaris isn't the only game that suffer from performance problems, Hearts of Iron IV also suffer from performance problems in the late game.

It might just be the Clausewitz engine not being able to support the new features of the games well or maybe it's a difficult engine to do performance enhancing work on, but if modders can implement performance fixes for the game and the PDX developers can't, then that's a lot of evidence pointing toward the PDX staff having holes in their competence surrounding performance.

Some of the choices that devs made surrounding performance in HoI IV was also highlighted by a weird set of priorities. Suggestions from the community saw that less, but larger divisions would improve performance of the game, but Paradox, probably wanting to stick to historical accuracy and not impose penalties on the AI wanted to keep the divisions small and plenty for the AI in the late game.
 
Last edited:

alexti

Field Marshal
24 Badges
Jul 25, 2010
3.146
383
  • Crusader Kings II
  • Victoria 2: Heart of Darkness
  • Victoria 2: A House Divided
  • Europa Universalis IV: Res Publica
  • Europa Universalis III Complete
  • Heir to the Throne
  • Europa Universalis IV: Conquest of Paradise
  • Europa Universalis IV: Art of War
  • Divine Wind
  • Europa Universalis IV: Wealth of Nations
  • Europa Universalis III
  • Stellaris: Ancient Relics
  • Stellaris: Megacorp
  • Stellaris: Distant Stars
  • Europa Universalis IV: Cradle of Civilization
  • Europa Universalis IV: Third Rome
  • Europa Universalis IV: Mandate of Heaven
  • Stellaris: Leviathans Story Pack
  • Hearts of Iron IV: Cadet
  • Stellaris
  • 500k Club
  • Victoria 2
  • Stellaris - Path to Destruction bundle
Some speculations regarding performance and architecture of Stellaris

Preliminaries

Modern days, the most commonly occurred bottleneck in performance is memory latency, i.e. time spent from generating request to memory to fulfill it. Processor caches can help with this, but only if data access pattern is right, so only a small fraction of the data set is accessed at once. CPUs are shit in handling scattered memory reads and writes, and caches help only if the data set layout in memory and access pattern is fine-tuned for performance.

Naturally, when performance is an issue, people aim to streamline algorithms for use of linear scans. CPUs are fine in handling linear scans. Incidental, commonly used object-oriented paradigm is not very compatible with this kind of optimizations. Still, Entity-Component-System approach is the king in AAA projects.

Incidentally, OOP is not a friend of performance. OOP extensively uses dynamic dispatch, i.e. indirect jumps. Indirect jumps are a killer of performance, because indirect jumb causes CPU pipeline stall and might cause cache miss.

Yet another angle for optimization is to split data sets into small chunks, fitting into CPU completely. Stellaris economic model is actually naturally hierarchic (Empires -> Systems -> Planets -> Pops), and there is nothing preventing planets being processed locally as long as all relevant data is collected in compact per-planet arrays. However, if they access some non-local resources during processing, it will be a big bottleneck.
That's all correct and it applies to a lot of games (and software in general). However, this doesn't appear to be a significant factor in Stellaris. When looking at CPU utilization the telling symptom of the problems you've described is high ratio of data cache misses and a lot of stalls in the back end. But in Stellaris it's not the case (assuming that graphics turned down enough to become a negligible cost). It's the front end that is overloaded which is a typical symptom of an interpreter. Perhaps the scripting language used by PDX is the root issue?

...or the game tries to evaluate pops with interpreted scripts and multiple (dozens of) memory redirections, and not optimized C++ code.
That is my guess too.
 

permeakra

Major
65 Badges
Nov 20, 2017
587
29
  • Tyranny - Bastards Wound
  • Crusader Kings II: Reapers Due
  • Tyranny: Archon Edition
  • Tyranny: Archon Edition
  • Tyranny: Gold Edition
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Leviathans Story Pack
  • Cities: Skylines - Natural Disasters
  • Crusader Kings II: Monks and Mystics
  • Stellaris - Path to Destruction bundle
  • Cities: Skylines - Mass Transit
  • Stellaris: Megacorp
  • Cities: Skylines Industries
  • Stellaris: Galaxy Edition
  • Cities: Skylines - Green Cities
  • Crusader Kings II: Jade Dragon
  • Stellaris: Humanoids Species Pack
  • Stellaris: Apocalypse
  • Surviving Mars: Digital Deluxe Edition
  • Cities: Skylines - Parklife Pre-Order
  • Cities: Skylines - Parklife
  • Stellaris: Distant Stars
  • Shadowrun Returns
  • Shadowrun: Dragonfall
  • Surviving Mars: First Colony Edition
  • Stellaris: Galaxy Edition
  • Crusader Kings II: Holy Fury
  • Stellaris: Lithoids
  • Crusader Kings III
  • Victoria 2
  • Cities: Skylines
  • Surviving Mars: First Colony Edition
  • Crusader Kings II: Way of Life
  • Pillars of Eternity
  • Crusader Kings II: Horse Lords
  • Cities: Skylines - After Dark
  • Crusader Kings II: Conclave
  • Cities: Skylines - Snowfall
  • Cities: Skylines - Campus
  • Crusader Kings II
  • Crusader Kings II: Legacy of Rome
  • Crusader Kings II: The Old Gods
  • Cities in Motion 2
  • Crusader Kings II: The Republic
  • Crusader Kings II: Sons of Abraham
  • Crusader Kings II: Sunset Invasion
  • Crusader Kings II: Sword of Islam
  • Majesty 2 Collection
  • Sword of the Stars
  • Tyranny - Tales from the Tiers
Perhaps the scripting language used by PDX is the root issue?
AFAIK, Stellaris scripts are not a proper scripting language. They allow to tie events to triggers and provide weights and numeric values for predefined slots. They also posted somewhere, that the event handling is properly optimized and isn't an issue.

In short, it should not be a big problem.

back end.
I dunno how you split Stellaris into back-end and front-end. I heard there are problems with GUI layer, but that they are not all that dire and are not a source of late-game lag.
 

Dëzaël

Captain
12 Badges
Nov 12, 2016
494
21
  • Stellaris: Synthetic Dawn
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Leviathans Story Pack
  • Stellaris - Path to Destruction bundle
  • Stellaris: Humanoids Species Pack
  • Stellaris: Apocalypse
  • Stellaris: Distant Stars
  • Stellaris: Megacorp
  • Stellaris: Ancient Relics
  • Stellaris: Lithoids
It's the front end that is overloaded which is a typical symptom of an interpreter. Perhaps the scripting language used by PDX is the root issue?
AFAIK, Stellaris scripts are not a proper scripting language. They allow to tie events to triggers and provide weights and numeric values for predefined slots. They also posted somewhere, that the event handling is properly optimized and isn't an issue.
There was also a video I saw with Wiz answering some questions after a presentation and he told specifically about the scripting language that a part of the loading time at game start was pre-compiling the scripts. Sure it is not as efficient as could be if made in C++ directly but it is clearly not interpreted either, and given what was dug in various threads by numerous people, I think the problems the game faces are an order of magnitude above what inefficiency those scripts could bring.

I would have given a link to the video itself but I have no clue where I saw this... 99.99% certain tho.
 

Surt

Field Marshal
29 Badges
Jan 29, 2003
6.865
227
Visit site
  • Semper Fi
  • Stellaris: Ancient Relics
  • Imperator: Rome
  • Stellaris: Megacorp
  • Stellaris: Distant Stars
  • Stellaris: Apocalypse
  • Stellaris: Humanoids Species Pack
  • Stellaris: Synthetic Dawn
  • Stellaris - Path to Destruction bundle
  • Stellaris: Leviathans Story Pack
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • 500k Club
  • Hearts of Iron II: Armageddon
  • Europa Universalis: Rome
  • Victoria: Revolutions
  • Europa Universalis III Complete
  • Europa Universalis III Complete
  • Heir to the Throne
  • Hearts of Iron III: Their Finest Hour
  • Hearts of Iron III
  • For the Motherland
  • Divine Wind
  • Europa Universalis III Complete
  • Europa Universalis III
  • Deus Vult
  • Crusader Kings II
I noticed the pop job swapping between two jobs too, it would definitely indicate that all pops are checked everyday, maybe multiple times ... if i judge from the swapping speed.
 

AlphaAsh

Miserable Git
41 Badges
Mar 16, 2015
1.199
700
www.alphastrikegames.com
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Stellaris: Megacorp
  • Stellaris: Leviathans Story Pack
  • Stellaris: Ancient Relics
  • Stellaris - Path to Destruction bundle
  • Stellaris: Lithoids
  • Stellaris: Distant Stars
  • Stellaris: Humanoids Species Pack
  • Stellaris: Synthetic Dawn
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Apocalypse
  • Shadowrun Returns
  • Shadowrun: Hong Kong
  • Shadowrun: Dragonfall
  • BATTLETECH
  • BATTLETECH - Beta Backer
  • BATTLETECH - Backer
  • BATTLETECH: Season pass
  • BATTLETECH: Flashpoint
  • Surviving Mars
  • Surviving Mars: First Colony Edition
  • Cities: Skylines Deluxe Edition
  • Cities: Skylines
  • Cities: Skylines - Parklife Pre-Order
  • Cities: Skylines - Parklife
  • Cities: Skylines - Green Cities
  • Cities: Skylines - Campus
  • Cities: Skylines - After Dark
  • Cities: Skylines - Snowfall
  • Cities: Skylines - Natural Disasters
  • Cities: Skylines - Mass Transit
  • Cities: Skylines Industries
  • Pillars of Eternity
  • Tyranny: Archon Edition
I noticed the pop job swapping between two jobs too, it would definitely indicate that all pops are checked everyday, maybe multiple times ... if i judge from the swapping speed.
I explained this earlier in the thread. The evaluation process has been derailed when a fringe case is hit.

edit - Inevitable "how do you know that?" qualifier. I wrote an AI mod that includes eliminating the fringe case. Which means during the development of the mod, I created test-cases to reproduce the issue, and to prove that it was in fact derailing. It was.

edit 2 - An example. When a version of GAI was integrated, a bunch of its scripted triggers were included. Here's one of the weight clauses for the artisan:

Code:
       modifier = {
           factor = 0
           jobs_work_minerals_goods = yes
       }
Normally the weight clause is evaluated for 25% of pops every 7 days. However this evaluation is derailed when this weight is applied, and all your artisans are sacked. Your artisans become unemployed, and immediately re-evaluate all vacant jobs - they've come off the 25%/7 days track.

And then one of those ex-artisans grabs a vacant job. Now that original weighting that got it sacked is no longer valid, so rather than being back on the 25%/7 days track, the pop evaluates that the artisan job is actually more suitable for it and swaps back.

Repeat ad infinitum until fringe case is broken by some outside factor.

A pop only gets back on the 25%/7days track if it actually holds a job long enough to get back on it.
 
Last edited:

alexti

Field Marshal
24 Badges
Jul 25, 2010
3.146
383
  • Crusader Kings II
  • Victoria 2: Heart of Darkness
  • Victoria 2: A House Divided
  • Europa Universalis IV: Res Publica
  • Europa Universalis III Complete
  • Heir to the Throne
  • Europa Universalis IV: Conquest of Paradise
  • Europa Universalis IV: Art of War
  • Divine Wind
  • Europa Universalis IV: Wealth of Nations
  • Europa Universalis III
  • Stellaris: Ancient Relics
  • Stellaris: Megacorp
  • Stellaris: Distant Stars
  • Europa Universalis IV: Cradle of Civilization
  • Europa Universalis IV: Third Rome
  • Europa Universalis IV: Mandate of Heaven
  • Stellaris: Leviathans Story Pack
  • Hearts of Iron IV: Cadet
  • Stellaris
  • 500k Club
  • Victoria 2
  • Stellaris - Path to Destruction bundle
AFAIK, Stellaris scripts are not a proper scripting language. They allow to tie events to triggers and provide weights and numeric values for predefined slots. They also posted somewhere, that the event handling is properly optimized and isn't an issue.

In short, it should not be a big problem.
That logic has to be evaluated somewhere somehow and it was often observed that large mods tend to have much worse performance than vanilla, so associating slow down with scripts seems a reasonable guess.

I dunno how you split Stellaris into back-end and front-end. I heard there are problems with GUI layer, but that they are not all that dire and are not a source of late-game lag.
CPU has hardware counters that record various events (such as instructions retiring, cache misses and a lot of other stuff), so running something like sysprof will show you where the time is being spent in CPU (front-end vs back-end and even more specifically). GUI issues in Stellaris is a separate source of slow down, but it can be practically eliminated with ticks_per_turn setting. I was referring to the apparently unavoidable slowdown (even when running with TPT 10).


There was also a video I saw with Wiz answering some questions after a presentation and he told specifically about the scripting language that a part of the loading time at game start was pre-compiling the scripts. Sure it is not as efficient as could be if made in C++ directly but it is clearly not interpreted either, and given what was dug in various threads by numerous people, I think the problems the game faces are an order of magnitude above what inefficiency those scripts could bring.

I would have given a link to the video itself but I have no clue where I saw this... 99.99% certain tho.
Interpreted and compiled are somewhat loosely defined. The point is that it is not compiled into a code CPU can run. Realistically, looking at the amount of scripts in the game there is no way it could compile all of them into reasonably efficient machine code during those 4-5 seconds it takes to load the game. If it was compiled (as from efficient C++), 10x-100x speedup would likely be enough not to have this thread at all.
 

permeakra

Major
65 Badges
Nov 20, 2017
587
29
  • Tyranny - Bastards Wound
  • Crusader Kings II: Reapers Due
  • Tyranny: Archon Edition
  • Tyranny: Archon Edition
  • Tyranny: Gold Edition
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Leviathans Story Pack
  • Cities: Skylines - Natural Disasters
  • Crusader Kings II: Monks and Mystics
  • Stellaris - Path to Destruction bundle
  • Cities: Skylines - Mass Transit
  • Stellaris: Megacorp
  • Cities: Skylines Industries
  • Stellaris: Galaxy Edition
  • Cities: Skylines - Green Cities
  • Crusader Kings II: Jade Dragon
  • Stellaris: Humanoids Species Pack
  • Stellaris: Apocalypse
  • Surviving Mars: Digital Deluxe Edition
  • Cities: Skylines - Parklife Pre-Order
  • Cities: Skylines - Parklife
  • Stellaris: Distant Stars
  • Shadowrun Returns
  • Shadowrun: Dragonfall
  • Surviving Mars: First Colony Edition
  • Stellaris: Galaxy Edition
  • Crusader Kings II: Holy Fury
  • Stellaris: Lithoids
  • Crusader Kings III
  • Victoria 2
  • Cities: Skylines
  • Surviving Mars: First Colony Edition
  • Crusader Kings II: Way of Life
  • Pillars of Eternity
  • Crusader Kings II: Horse Lords
  • Cities: Skylines - After Dark
  • Crusader Kings II: Conclave
  • Cities: Skylines - Snowfall
  • Cities: Skylines - Campus
  • Crusader Kings II
  • Crusader Kings II: Legacy of Rome
  • Crusader Kings II: The Old Gods
  • Cities in Motion 2
  • Crusader Kings II: The Republic
  • Crusader Kings II: Sons of Abraham
  • Crusader Kings II: Sunset Invasion
  • Crusader Kings II: Sword of Islam
  • Majesty 2 Collection
  • Sword of the Stars
  • Tyranny - Tales from the Tiers
CPU has hardware counters that record various events
I'm very much aware of that.
You STILL didn't clarified how you split Stellaris into back-end and front-end. Do qualify.

That said, there was dev diary N 149, that posted plots of CPU time sinks and specifically qualified that most time is spent on pops-related calculations.
 
Last edited:

AlphaAsh

Miserable Git
41 Badges
Mar 16, 2015
1.199
700
www.alphastrikegames.com
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Stellaris: Megacorp
  • Stellaris: Leviathans Story Pack
  • Stellaris: Ancient Relics
  • Stellaris - Path to Destruction bundle
  • Stellaris: Lithoids
  • Stellaris: Distant Stars
  • Stellaris: Humanoids Species Pack
  • Stellaris: Synthetic Dawn
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Apocalypse
  • Shadowrun Returns
  • Shadowrun: Hong Kong
  • Shadowrun: Dragonfall
  • BATTLETECH
  • BATTLETECH - Beta Backer
  • BATTLETECH - Backer
  • BATTLETECH: Season pass
  • BATTLETECH: Flashpoint
  • Surviving Mars
  • Surviving Mars: First Colony Edition
  • Cities: Skylines Deluxe Edition
  • Cities: Skylines
  • Cities: Skylines - Parklife Pre-Order
  • Cities: Skylines - Parklife
  • Cities: Skylines - Green Cities
  • Cities: Skylines - Campus
  • Cities: Skylines - After Dark
  • Cities: Skylines - Snowfall
  • Cities: Skylines - Natural Disasters
  • Cities: Skylines - Mass Transit
  • Cities: Skylines Industries
  • Pillars of Eternity
  • Tyranny: Archon Edition
I was reviewing that first thread I linked where @KingAlamar , @UltimateTobi and others did a very thorough examination of lag issues with the game.

Many of their findings suggest the issue is unemployment. Pairing that up with my own work on performance and eliminating job popping, when developing EDAI, I'd agree, and I think I can make the following conclusions from it:

1 - Employed pops will re-evaluate jobs on the schedule suggested in a dev post/changelog/dev diary (needs citation clarifying) for an update: approximately 25% every 7 days.
2 - Unemployed pops aren't on this schedule. They are evaluating jobs continuously, regardless of whether there are actual vacancies (presumably to see if they can displace an employed pop because they are more suitable).

It is #2 that is the cause of most performance problems coming from pops. Whether evaluation of potential clauses or any other clauses of a job, other than the job weightings, are evaluated on a different schedule for unemployed/employed, I'm not sure.

I think a considerable performance improvement would result from re-thinking the creation of unemployment, and those pops method of evaluating jobs.
 
Last edited:

Surt

Field Marshal
29 Badges
Jan 29, 2003
6.865
227
Visit site
  • Semper Fi
  • Stellaris: Ancient Relics
  • Imperator: Rome
  • Stellaris: Megacorp
  • Stellaris: Distant Stars
  • Stellaris: Apocalypse
  • Stellaris: Humanoids Species Pack
  • Stellaris: Synthetic Dawn
  • Stellaris - Path to Destruction bundle
  • Stellaris: Leviathans Story Pack
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • 500k Club
  • Hearts of Iron II: Armageddon
  • Europa Universalis: Rome
  • Victoria: Revolutions
  • Europa Universalis III Complete
  • Europa Universalis III Complete
  • Heir to the Throne
  • Hearts of Iron III: Their Finest Hour
  • Hearts of Iron III
  • For the Motherland
  • Divine Wind
  • Europa Universalis III Complete
  • Europa Universalis III
  • Deus Vult
  • Crusader Kings II

alexti

Field Marshal
24 Badges
Jul 25, 2010
3.146
383
  • Crusader Kings II
  • Victoria 2: Heart of Darkness
  • Victoria 2: A House Divided
  • Europa Universalis IV: Res Publica
  • Europa Universalis III Complete
  • Heir to the Throne
  • Europa Universalis IV: Conquest of Paradise
  • Europa Universalis IV: Art of War
  • Divine Wind
  • Europa Universalis IV: Wealth of Nations
  • Europa Universalis III
  • Stellaris: Ancient Relics
  • Stellaris: Megacorp
  • Stellaris: Distant Stars
  • Europa Universalis IV: Cradle of Civilization
  • Europa Universalis IV: Third Rome
  • Europa Universalis IV: Mandate of Heaven
  • Stellaris: Leviathans Story Pack
  • Hearts of Iron IV: Cadet
  • Stellaris
  • 500k Club
  • Victoria 2
  • Stellaris - Path to Destruction bundle
I'm very much aware of that.
You STILL didn't clarified how you split Stellaris into back-end and front-end. Do qualify.
I think you're misunderstanding. I don't split the software into back-end and front-end, only observe what it is utilizing. There are many tools that read the counters and show what hardware (inside of CPU) application is utilizing (I've used perf). In a big picture, front-end is responsible for fetching and decoding the instructions and the back-end is responsible for loading the data and executing the instructions. For the compiled algorithms it's typical to heavily load the back-end (because they may go over large data sets and contain complex instructions that take many cycles to run, but they keep running the same small set of instructions, so they are not demanding on the front-end ). The problems with the data locality you've described in your earlier post would cause processor to stall in the back-end while waiting for the data to load (which would be slow due to data cache misses). Stellaris is stalling a lot in the front-end, but not in the back-end, so it shows that the data layout is not a main issue.

That said, there was dev diary N 149, that posted plots of CPU time sinks and specifically qualified that most time is spent on pops-related calculations.
It didn't really show much details. Is it spending time running the pop-related scripts? Or some pop-related compiled code?
 

alexti

Field Marshal
24 Badges
Jul 25, 2010
3.146
383
  • Crusader Kings II
  • Victoria 2: Heart of Darkness
  • Victoria 2: A House Divided
  • Europa Universalis IV: Res Publica
  • Europa Universalis III Complete
  • Heir to the Throne
  • Europa Universalis IV: Conquest of Paradise
  • Europa Universalis IV: Art of War
  • Divine Wind
  • Europa Universalis IV: Wealth of Nations
  • Europa Universalis III
  • Stellaris: Ancient Relics
  • Stellaris: Megacorp
  • Stellaris: Distant Stars
  • Europa Universalis IV: Cradle of Civilization
  • Europa Universalis IV: Third Rome
  • Europa Universalis IV: Mandate of Heaven
  • Stellaris: Leviathans Story Pack
  • Hearts of Iron IV: Cadet
  • Stellaris
  • 500k Club
  • Victoria 2
  • Stellaris - Path to Destruction bundle
Many of their findings suggest the issue is unemployment. Pairing that up with my own work on performance and eliminating job popping, when developing EDAI, I'd agree, and I think I can make the following conclusions from it:

1 - Employed pops will re-evaluate jobs on the schedule suggested in a dev post/changelog/dev diary (needs citation clarifying) for an update: approximately 25% every 7 days.
2 - Unemployed pops aren't on this schedule. They are evaluating jobs continuously, regardless of whether there are actual vacancies (presumably to see if they can displace an employed pop because they are more suitable).

It is #2 that is the cause of most performance problems coming from pops. Whether evaluation of potential clauses or any other clauses of a job, other than the job weightings, are evaluated on a different schedule for unemployed/employed, I'm not sure.
I agree with (1), but there is more to (2). Unemployed pops don't always evaluate jobs every day. By going on slow speed it's easy to observe the situations when there is an open job, an unemployed pop and it takes that pop several days to pick the job. I am not sure if unemployment makes performance worse or not, but unfortunately (for me) I see performance degradation even when I have no unemployment (and it applies even in the case of empty galaxy with no AI). Maybe it's possible to test it by disabling all jobs and comparing performance with the case when all those jobs are enabled?
 

AlphaAsh

Miserable Git
41 Badges
Mar 16, 2015
1.199
700
www.alphastrikegames.com
  • Stellaris
  • Stellaris: Galaxy Edition
  • Stellaris: Galaxy Edition
  • Stellaris: Megacorp
  • Stellaris: Leviathans Story Pack
  • Stellaris: Ancient Relics
  • Stellaris - Path to Destruction bundle
  • Stellaris: Lithoids
  • Stellaris: Distant Stars
  • Stellaris: Humanoids Species Pack
  • Stellaris: Synthetic Dawn
  • Stellaris: Digital Anniversary Edition
  • Stellaris: Apocalypse
  • Shadowrun Returns
  • Shadowrun: Hong Kong
  • Shadowrun: Dragonfall
  • BATTLETECH
  • BATTLETECH - Beta Backer
  • BATTLETECH - Backer
  • BATTLETECH: Season pass
  • BATTLETECH: Flashpoint
  • Surviving Mars
  • Surviving Mars: First Colony Edition
  • Cities: Skylines Deluxe Edition
  • Cities: Skylines
  • Cities: Skylines - Parklife Pre-Order
  • Cities: Skylines - Parklife
  • Cities: Skylines - Green Cities
  • Cities: Skylines - Campus
  • Cities: Skylines - After Dark
  • Cities: Skylines - Snowfall
  • Cities: Skylines - Natural Disasters
  • Cities: Skylines - Mass Transit
  • Cities: Skylines Industries
  • Pillars of Eternity
  • Tyranny: Archon Edition
Maybe it's possible to test it by disabling all jobs and comparing performance with the case when all those jobs are enabled?
It is. All the jobs are almost instantly taken by any pop that meets the basic potential clause. In other words, there's no 25%/7 day evaluation. It's happening constantly for unemployed pops.

Unemployed pops don't always evaluate jobs every day.
I'll do some more testing as this surprises me. Perhaps it is a matter of perception, or possibly the amount of unemployment is a factor. Or maybe there's an internal call when job availability changes. I'll see if I can get the weight clauses to pipe a ping out to the log as well.
 
Last edited: