Why Did Late Game Pop Lag Exist in the First Place?

[IntrigueAuthoritarian]

Now that Stellaris is starting to really work on improving performance with their updates, I have to ask:

Why was Stellaris Late Game Pop Lag (patent pending) ever a problem to begin with?

Now, nobody needs to be reminded that this was (and likely will be for a few updates more), in fact, a problem that needs fixing. But I'm not here just to complain about the fact that entire playthroughs turn genocidal just so the game runs at a playable speed, or the fact that the meta favors rushing planet busters as fast as humanly possible not because of in-game reasons, but because it's the only way to keep the galaxy's population from breaking your PC. Oh, and I certainly don't need to mention that the exponential increase to Pop growth cost on a per empire basis not only does weird things to game balance and opens up some... unusual ways to cheese the system, but only delays the inevitable.

No, ranting about that now that performance is finally being worked on would be in poor taste. I'm here to talk about a solution. And while I'm no programming expert, I know enough to say with a little confidence that - unless Stellaris's pop code is REALLY wonky - this change should be easier to implement than some other changes made to the game over the years, and is definitely important enough that it's worth the effort.

And that solution can be found in the Victoria series. (Also made by Paradox.) Victoria 2 also tracks population, Victoria 2 tracks much larger populations than Stellaris - so what's the difference? Why doesn't this lag the hell out of Victoria 2 like it does Stellaris?

Because Victoria 2 internally tracks its "pops" as overall statistics, and Stellaris internally tracks pops as individuals. And to avoid any possible misunderstanding as to what I mean by this, I'm going to use an example!



Stellaris as it is now, internally tracks a planet's pops like this:

Pop 01
Species: Flibbit
Ethos: Egalitarian
Job: Miner
Stratum: Worker

Pop 02
Species: Flibbit
Ethos: Militarist
Job: Miner
Stratum: Worker

...and so on and so forth, with an entry for each pop on the planet.



If Stellaris internally tracked pops like Victoria 2, it would track a planet's pops like this:


Worker Stratum
20 Egalitarian
5 Authoritarian
7 Militarist
10 Xenophile

Miners
35 pops
21 Flibbits, 8 Biffits, 6 Blorgs.

Clerks
7 pops
7 Biffits

...with similar entries for the other stratums.



Notice how information on 42 pops was delivered using Victoria's method in about the same amount of space it took Stellaris's to elaborate upon 2? That's a lot less variables for the game to track, which means less lag.

So why doesn't Stellaris track pops this way? Well, as those who have been around for a while already know, originally pops were on a grid - with all these wacky adjacency bonus and tile based shenanigans. Back then, which pop had which ethos and was where actually mattered. And back then, a planet only held 25 pops at MOST.

Now, though? Who cares whether those 7 militarist pops are Flibbits or Blorg? Who cares if they're miners or clerks? As long as all seven are from the worker stratum, their political influence is the same - and if that information isn't important in any way shape or form, why does the game track it? This isn't The Sims or Crusader Kings, nobody is checking in on individual specific pops to see what they're up to and how they're doing!

Not to mention that the vast majority of the time, these statistics are how the player sees pop related data. I would probably just assume the game ALREADY tracked pops this way if it weren't for two things:

The fact that I can check the ethos of each individual pop - as well as some other things, like whether they're currently demoting or still upset that they used to be a slave.

And, perhaps more importantly, the very existence of pop lag to begin with. A 7 doesn't make more lag than a 4, if pops were just statistics in the game files then how many pops there were on a planet wouldn't make a difference.



But, how would this affect the players? (Aside from reducing lag, of course.) Well, it wouldn't. Only two minor changes would need to be made, and even then I can think of ways to get around them.

The first is pop demotion and similar pop behaviors - which could be easily fixed by making them demote faster, but only one at a time... or just attaching the data to the planet rather than the pop. (There are 3 Blorgs currently demoting is all the player needs to know.)

The second is ethos. The player wouldn't be able to click on individual pops to see what their ethos is anymore, but, again, who cares? In fact, why is pop ethos still tracked as solid numbers instead of percentages? I mean, usually that's how the player perceives it. I don't care how many authoritarian pops there are on this planet total, I care what portion of the planet's population they comprise - and maybe what stratum they're in.



Not to mention that a pop is definitely WAY more than just one person. Sure, the exact amount of people that make up a pop is deliberately left vague, but even if we lowball it and say one 20 Pop planet has a population of just 1 billion (less than a seventh of the population of Earth) that still puts 1 Pop at 50 million people! And you mean to tell me ALL 50 million are authoritarian? Plus, tracking the constant ethos shifting of individual pops has GOT to contribute to the lag problem. Just say that 32% of the planet's worker stratum is authoritarian and leave it at that. Heck, the total doesn't even have to add up to 100% - after all, some people could support materialist AND authoritarian policies, or some people might just not be invested in politics.



Combine this with some changes to AI to make them less zealous with creating new species, and some auto-cleaning that - every month or so - deletes every species template with no pops either actively part of the template or being converted to it... and Stellaris Late Game Pop Lag should be all but dead and gone.

The only real remaining source of lag at this point should be fleets and armies, and the performance updates Paradox already has brewing should be enough for that.

 

[ASGeek2012]

This will not work as the game mechanics still look at individual pops for things like migration where pops move from one planet to the other, or for events that wind up affecting individual pops. Once you break the connection between a pop and what they are, it makes it hard to manage pops at an individual level.

For example, say I conquered a Blorg planet. Most Blorg are pacifists, so the planet has numbers of Blorg and numbers of pacifists. But say they have other species there as well who have different ethics. Or some of the Blorg drifted to other ethics. Their numbers would be represented as an abstract as you described. So we're okay so far.

Now a Blorg needs to leave the planet. Either they're migrating because of lack of jobs, or being killed off by an event, or I'm at war with someone with raiding stance bombardment and are stealing pops. Now, I can know the pop being removed is a Blorg, but how do I know to decrement the number of pacifists? Answer: I can't, because I no longer know the association between Blorg and pacifist ethic. And the pacifist ethic may not be the right one because we have Blorg who are not pacifists.

I'm not saying what you came up with is not a good idea, it's just not suited for the way the game is currently designed.

 

[CaesarVincens]

[...]

And while I'm no programming expert, I know enough to say with a little confidence that - unless Stellaris's pop code is REALLY wonky - this change should be easier to implement than some other changes made to the game over the years, and is definitely important enough that it's worth the effort.

And that solution can be found in the Victoria series. (Also made by Paradox.) Victoria 2 also tracks population, Victoria 2 tracks much larger populations than Stellaris - so what's the difference? Why doesn't this lag the hell out of Victoria 2 like it does Stellaris?

Because Victoria 2 internally tracks its "pops" as overall statistics, and Stellaris internally tracks pops as individuals. And to avoid any possible misunderstanding as to what I mean by this, I'm going to use an example!

[...]

So why doesn't Stellaris track pops this way? Well, as those who have been around for a while already know, originally pops were on a grid - with all these wacky adjacency bonus and tile based shenanigans. Back then, which pop had which ethos and was where actually mattered. And back then, a planet only held 25 pops at MOST.

[...]

Here's your reason, pops are the integral machine of Stellaris' economy and also are tied into many events and other mechanics. Changing how they are tracked is certainly feasible, but the question is always at what cost versus to what benefit?

Changing the pop representation necessarily entails changing all interacting mechanics as well, if only in the strict sense of changing the interface/hooks in the code. It's certainly doable, but would be a rework at least on par with the 2.2 change from tiles to districts, likely more than that even.

Would I be surprised if a Victoria style system is used in an eventual sequel game? Not at all. As you outline, there are many advantages, in performance and in granularity.

 

[Silesian Burd]

Could it be that every single thing in the map screens is a 3-D model, and that the game fills up with thousands of these things in the late game, and that each of these things has its own Paradox-style (TM) movement?

Nah, it must be the spreadsheet side of the game.

The first is pop demotion and similar pop behaviors - which could be easily fixed by making them demote faster, but only one at a time... or just attaching the data to the planet rather than the pop. (There are 3 Blorgs currently demoting is all the player needs to know.)

The game already does that. The problem is that every single Pop wants to be the one to do that; or if you view them as inert values in a spreadsheet, the problem is that the game checks every single Pop for optimal validity, and then picks a rando among the pile of those available.

The second is ethos. The player wouldn't be able to click on individual pops to see what their ethos is anymore, but, again, who cares? In fact, why is pop ethos still tracked as solid numbers instead of percentages? I mean, usually that's how the player perceives it. I don't care how many authoritarian pops there are on this planet total, I care what portion of the planet's population they comprise - and maybe what stratum they're in.

In the not-too-distant past, Pops didn't stack at all. Also, Imperator Rome tried the Vicky 2 thing with Pops, and everyone hated it and that game died. I suppose you can say Paradox shouldn't have flipped a coin to decide which game would get which style of the mechanic for their revamp, but if it's any consolation, Stellaris got the Jobs end.

Not to mention that a pop is definitely WAY more than just one person. Sure, the exact amount of people that make up a pop is deliberately left vague, but even if we lowball it and say one 20 Pop planet has a population of just 1 billion (less than a seventh of the population of Earth) that still puts 1 Pop at 50 million people! And you mean to tell me ALL 50 million are authoritarian? Plus, tracking the constant ethos shifting of individual pops has GOT to contribute to the lag problem. Just say that 32% of the planet's worker stratum is authoritarian and leave it at that. Heck, the total doesn't even have to add up to 100% - after all, some people could support materialist AND authoritarian policies, or some people might just not be invested in politics.

Pops are distinct because they are people working in a specific area; a District, you can say. Before the update that replaced Tiles with Districts, Pops also LIVED where they worked, making the concept easier to understand. Comparing Pops to population counts is not ideal, considering that Stellaris made the mistake of having Earth in the game; a Pop would be 35 million people in the Late Medieval Period (start of EU4 or so), around 46 million people in the Industrial Age (around the end of EU4), 100 million people in the Machine Age (WW2), and 200-600 million people in the Early Space Age (based on our current models), implying that people only got LESS efficient with all of the technological advances.

As for Pie-Charting Ethics, you'd still end up making the calculations, you'd just be hiding them so that people have less of a clue as to why lag is picking up.

Combine this with some changes to AI to make them less zealous with creating new species, and some auto-cleaning that - every month or so - deletes every species template with no pops either actively part of the template or being converted to it... and Stellaris Late Game Pop Lag should be all but dead and gone.

I could've sworn the wiki said the AI didn't create new species outside of taking Synthetic or Psionic Ascensions, or outside of using Relics. Are you sure you're not just having a bad time with Xeno-Compatibility? Have you considered switching your Species menu to only show species in your Empire, or just not looking at those menus?

The only real remaining source of lag at this point should be fleets and armies, and the performance updates Paradox already has brewing should be enough for that.

 

[HFY]

Stellaris was built as a game with size 25 planets that held up to 25 pops, and you got fewer than 100 of these planets.

Stellaris pop modeling was based on the idea that every pop is important, and each pop has a history -- there are still temporary effects like Recently Conquered / Recently Liberated / Grateful Refugee / etc. which show the pop's reaction to transitions between some states.

That's why there's late-game lag. The engine was built for a much smaller pop count on tiles, and the 2.2 job changes which massively increased pop count broke that design without compensating for its changes.


In terms of "just use V2 pop mechanics" that would require design work which nobody has been able to do so far.

Stellaris pops have traits. They are not all humans. Games like V2 which have only one species don't have this issue, so their pop calculations can be simpler.

Stellaris pops mutate over time. They can be gene-modded or assimilated or even self-modified into having different traits. Traits impact happiness and faction attraction. Happiness itself impacts faction attraction. Having robots or other species on the planet impacts faction attraction -- modified by both the pop's traits and the xeno's traits. These are issues V2 will never need to solve, so V2 can be simpler (and indeed is simpler).

Stellaris can't just copy-paste mechanics from a simpler game with only one species.


If you can solve the issues of fitting complex Stellaris pops into V2 mechanics, please describe your solution in enough detail that the Stellaris devs can use your solution for the game.

Nobody has been able to solve this so far. Maybe it can be done, but if so it's not easy.

If you can do it, please share with the devs.

 

[ZomgK3tchup]

And that solution can be found in the Victoria series. (Also made by Paradox.) Victoria 2 also tracks population, Victoria 2 tracks much larger populations than Stellaris - so what's the difference? Why doesn't this lag the hell out of Victoria 2 like it does Stellaris?

Victoria 2 was also slower than Stellaris and had a set end date.

 

[Silesian Burd]

Stellaris was built as a game with size 25 planets that held up to 25 pops, and you got fewer than 100 of these planets.

Stellaris pop modeling was based on the idea that every pop is important, and each pop has a history -- there are still temporary effects like Recently Conquered / Recently Liberated / Grateful Refugee / etc. which show the pop's reaction to transitions between some states.

That's why there's late-game lag. The engine was built for a much smaller pop count on tiles, and the 2.2 job changes which massively increased pop count broke that design without compensating for its changes.


In terms of "just use V2 pop mechanics" that would require design work which nobody has been able to do so far.

Stellaris pops have traits. They are not all humans. Games like V2 which have only one species don't have this issue, so their pop calculations can be simpler.

Stellaris pops mutate over time. They can be gene-modded or assimilated or even self-modified into having different traits. Traits impact happiness and faction attraction. Happiness itself impacts faction attraction. Having robots or other species on the planet impacts faction attraction -- modified by both the pop's traits and the xeno's traits. These are issues V2 will never need to solve, so V2 can be simpler (and indeed is simpler).

Stellaris can't just copy-paste mechanics from a simpler game with only one species.


If you can solve the issues of fitting complex Stellaris pops into V2 mechanics, please describe your solution in enough detail that the Stellaris devs can use your solution for the game.

Nobody has been able to solve this so far. Maybe it can be done, but if so it's not easy.

If you can do it, please share with the devs.

Man, I really wish we could track the histories of Pops... or at least tag them, so we can spot them more easily while clicking through planets.

Oh well; if it didn't come up in between them switching to Districts and them massively inflating Pop counts while blowing off what that would actually mean for the setting (basically all of the in-game dialogue acts like Pops are still a billion people each, for example, as if it were still 1.0), I doubt it's ever going to come up...

 

[SectorsAreOkay]

The devs said that there were a lot of extra checks and rechecks going on for pop jobs and placement and such. It's really easy for those types of calculations to grow out of hand without noticing because many of the checks and events are dependent on other checks and events creating a complex graph of things to update. As a previous poster said above, when it was just 25 pops at most per planet and the jobs were mostly static, any inefficiencies are hidden. When you can have 100s of pops per planet with all sorts of jobs and weights it very easily can make the inefficiencies apparent. Fixing them is not simple, but thankfully they did.