Hi everyone! I am Caligula, one of Stellaris’ Content Designers, which means that I do a variety of tasks based around narrative writing and scripting - “scripting” being our term for doing things that is somewhat similar to programming, but without changing the source code. In other words, I do what modders do (though I have the significant advantage of also being able to peek into the source code and change it around when needed). Every Content Designer has their niche, and mine is that when a particularly complicated system needs to be scripted in (or, more frequently, is giving some sort of trouble - the War in Heaven still gives me nightmares...), I step into the breach.
Now, we have a lot of exciting stuff to show off in the weeks and months to come, but for today, inspired by some questions that were asked after the last dev diary, I’m going to be writing about the technical side of scripting for modders and aspiring modders, specifically with an eye on what can cause performance problems and how to avoid making bad scripts.
The Stellaris scripting language is a very powerful tool, and a lot can be done with it, but first of all, a note of caution: just because something is possible, does not mean it should be done. I can’t really stress this enough, because (and I speak from experience here) this attitude will almost certainly end up causing both performance issues and unreadable scripts that you will not be able to disentangle six months later when you realise some part of it is broken. Though it should be borne in mind that doing something in code is, by definition, faster: in code, you can check a single function and be done with it, but if you want it to be accessible through script, there’s a fair few necessary functions it has to go through before you get to checking your function (turning the line of script into a code command, checking whether it’s used in the right scope, etc etc) - hence why some things are hardcoded, and also why hacky solutions to problems can end up being quite bad. So, the first question to consider is, should I really be doing this?
But who am I kidding, I’m speaking to modders here, so of course you will do it
So without further ado...
What causes performance issues?
Every time you run a check or execute an effect, this will take a very tiny amount of your computer’s processing power. With a few exceptions that should be used sparingly (I’ll get to those later), this is totally fine and is needed to do anything at all. It is when the check is repeated often, over lots of objects, that problems happen. In practice, this usually means pops are the cause, though running something across all planets in the galaxy is also a pretty bad idea.
As a first step, when possible, it is a good idea to control when your script is run. The best way to do this is by setting where events are fired and using on_actions (or firing events from decisions and the like) wherever possible, instead of mean time to happen or, even worse, just setting an event to try and fire every day. If a degree of randomness is needed, one could also fire a hidden event via, say, a yearly pulse and then firing the actual event you want with a random delay (for an example, check out event action.220).
Of course, not everything is events, and unfortunately, in Stellaris, a lot of stuff is done with pops! Job weights and other triggers in the jobs files, in particular, have been shown to cause issues in the past. As a rule of thumb, even if you can do super cool stuff, it is a bad idea to do any too complicated script pyromania on jobs. For example, if you were to make a job weight dependent on there being no other pops on the planet that are unemployed (using
So what can be done?
Avoiding nested loops, making sure your events are fired appropriately and avoiding pops when possible will get you some way, but we can do better than that. Here is my list of advice for optimising scripts:
Always use the most appropriate scope
Say you want to check something on the leader of the current country’s federation. One could, theoretically, do it this way:
This'll run through all countries in the game and see whether they are in the same federation as you, including the space amoeba country and the vile Pasharti (and who would want them in a federation?). That’s a bunch that are definitely irrelevant. So a better check would be to do it this way:
In code terms, this means that the game going from the country to the federation and then grabbing a list of its members, excluding the current country, and checking the triggers against them. So, that’s obviously going to be fewer checks. However, the best version would be this:
That version would go straight to the federation and from there straight to its leader in the code, with as little as possible script to code conversion needed and no need to check triggers against any countries to get there. It also happens to be the most readable (readability and better performance very often correlate…).
So in this case, the game would check around 50 countries first the first version, 5 for the second and 1 for the third - not bad for some optimisations! Using a similar logic, it is always better to use something that isn’t checking all objects in the galaxy (esp. all pops or all planets) if at all possible but rather a filtered list, e.g. any_planet_within_border instead of any_planet = { solar_system.owner = { is_same_value = prevprev } } (you laugh, but I’ve seen it). And, indeed, one can almost always check any_owned_fleet instead of any_owned_ship.
Another important improvement we added in 2.6 was any_owned_species, which can replace many any_owned_pop checks (specifically the ones that check for traits and so on of the pop) and mean that way, way fewer objects have to be checked (in a xenophobic empire, it could be single figures for any_owned_species and thousands for any_owned_pop).
Sometimes you can avoid scopes completely
On a similar note, if you can check something without doing things with scopes, that’s always going to be better. So, if one wants to check whether a planet has more than two pops working as miners, one could do this two ways:
The former will check each pop on the planet and see whether it has the miner job, and then see whether the number that do is higher than 2. The latter will check a cached number that the game has already calculated and see if it is more than 2, which is much quicker to do.
Some things are just expensive
Not every check or effect is equal. Checking a flag or a value is generally pretty simple, and changing it is usually not much more complicated. If, however, the game has to recalculate stuff, then it will take longer, because it’s not just looking up a number it already knows. Creating new stuff is also more expensive, both because it’s doing something somewhat complicated (the create_species effect is, I kid you not, more than 600 lines of C++ code...), and because it’ll probably have to recalculate all sorts of values once this is done. It can be a bit tricky to know which triggers and effects are going to be bad, but as a rule, these cases are what you should look out for:
If it must be done...
Sometimes, bad things must be done. In these cases, it is best to still use the not so great things with precision. When the game is checking triggers e.g. for an event, it’ll generally stop checking them at the first point something returns false (I’m told this is called “short-circuit evaluation”), so you’ll want to do something like this:
I recently did something like this to the refugee pop effect. It was previously a little bit insane (see 01_scripted_triggers_refugees.txt for the full horror). In total, it would check a variation of the following up to eight times:
Where it varied was simply the last any_owned_planet: It would try and find a relation with a really good planet for the pop to live on, then a pretty good, then a pretty decent, and then finally settle for just any old planet. Which is, obviously, pretty inefficient, since the list of countries that welcome refugees does not change between each of the 8 times you check it. My way of avoiding this - and making the script way more readable, whilst I was at it - was to set a flag before any of the checks, like this:
The checks then simply had to be “does the country have the flag, if yes, does it have a good enough planet”:
One can also similarly use if-limits and elses (and, even better, switches when possible - those are the best for performance) in triggers to narrow down the checks down and make things far more readable whilst you are at it. I recently went through the species rights files and redid the allow triggers for sanity’s sake:
(Before)
(After)
The second version will be more efficient, since it is only checking e.g. whether the species has the mechanical trait or whether the country has the assimilator civic once instead of twice, and also, the triggers in the second custom tooltip aren’t obscenely weird anymore. (I also removed all the NANDs, because they broke my brain).
Happy New Year
I can’t write this dev diary without telling you about the “Happy New Year” bug. Basically, we were playing dev MP on a reasonably large galaxy and reached reasonably late into the game, and since we were all working remotely on wildly varying computers and internet connection speeds, the performance was perhaps a tad sluggish, but still acceptable for the most part. Then, suddenly, we noticed huge lag spikes - 20 seconds and more - on the 1st of January. So noticeable were these spikes that we began wishing each other a Happy New Year each time the game froze!
It just so happened that the onset of this lag coincided with several large empires deciding to become synthetic and starting to assimilating their empires. Now, assimilation falls into the category of things that are done in script that maybe, in hindsight, should probably not have been done that way… and works by firing an event for each assimilating country every 1st of January. This event in turn fired an event for each of their planets that selected a bunch of pops on the planet and used modify_species on each of them at least once, but sometimes up to four times. This added up to a fairly significant performance hog!
After trying various solutions, it turned out the best way to fix this was to first go through every_owned_species from the country scope, check whether this species should be assimilated, and if so use modify_species to create the species it would assimilate to, setting a species flag that pointed to the species it was being assimilated from. Then, instead of creating a new species for every pop that was assimilated, the scripts were rewritten to find the already-created species that the pop should become, and simply use change_species on it. The result is still unreadable script (I will spare your eyes and not post it here), but in my tests it reduced the yearly tick by over 50%, thanks to the complicated effect (modify_species) being run as seldom as possible.
---
That’s it for me, for now! I’m guessing this was a bit of a drier dev diary than usual for most of you, but hopefully it was interesting nonetheless As a final note, I suspect that the more intrepid among you could call me out on bits of script in the base game that don’t quite live up to these guidelines. Please, feel free to do so, because there are few things more satisfying in this profession than taking something really horrible and making something less horrible out of it!
Now, we have a lot of exciting stuff to show off in the weeks and months to come, but for today, inspired by some questions that were asked after the last dev diary, I’m going to be writing about the technical side of scripting for modders and aspiring modders, specifically with an eye on what can cause performance problems and how to avoid making bad scripts.
The Stellaris scripting language is a very powerful tool, and a lot can be done with it, but first of all, a note of caution: just because something is possible, does not mean it should be done. I can’t really stress this enough, because (and I speak from experience here) this attitude will almost certainly end up causing both performance issues and unreadable scripts that you will not be able to disentangle six months later when you realise some part of it is broken. Though it should be borne in mind that doing something in code is, by definition, faster: in code, you can check a single function and be done with it, but if you want it to be accessible through script, there’s a fair few necessary functions it has to go through before you get to checking your function (turning the line of script into a code command, checking whether it’s used in the right scope, etc etc) - hence why some things are hardcoded, and also why hacky solutions to problems can end up being quite bad. So, the first question to consider is, should I really be doing this?
But who am I kidding, I’m speaking to modders here, so of course you will do it
So without further ado...What causes performance issues?
Every time you run a check or execute an effect, this will take a very tiny amount of your computer’s processing power. With a few exceptions that should be used sparingly (I’ll get to those later), this is totally fine and is needed to do anything at all. It is when the check is repeated often, over lots of objects, that problems happen. In practice, this usually means pops are the cause, though running something across all planets in the galaxy is also a pretty bad idea.
As a first step, when possible, it is a good idea to control when your script is run. The best way to do this is by setting where events are fired and using on_actions (or firing events from decisions and the like) wherever possible, instead of mean time to happen or, even worse, just setting an event to try and fire every day. If a degree of randomness is needed, one could also fire a hidden event via, say, a yearly pulse and then firing the actual event you want with a random delay (for an example, check out event action.220).
Of course, not everything is events, and unfortunately, in Stellaris, a lot of stuff is done with pops! Job weights and other triggers in the jobs files, in particular, have been shown to cause issues in the past. As a rule of thumb, even if you can do super cool stuff, it is a bad idea to do any too complicated script pyromania on jobs. For example, if you were to make a job weight dependent on there being no other pops on the planet that are unemployed (using
planet = { any_owned_pop = { is_unemployed = yes } }), then you are doing a regular check on every pop on the planet that then checks every other pop on the planet, i.e. pops squared. Once you reach the late game, this is pretty much guaranteed to cause issues.So what can be done?
Avoiding nested loops, making sure your events are fired appropriately and avoiding pops when possible will get you some way, but we can do better than that. Here is my list of advice for optimising scripts:
Always use the most appropriate scope
Say you want to check something on the leader of the current country’s federation. One could, theoretically, do it this way:
Code:
any_country = {
is_in_federation_with = root
is_federation_leader = yes
<my_triggers_here> = yes
}This'll run through all countries in the game and see whether they are in the same federation as you, including the space amoeba country and the vile Pasharti (and who would want them in a federation?). That’s a bunch that are definitely irrelevant. So a better check would be to do it this way:
Code:
any_federation_ally = {
is_federation_leader = yes
<my_triggers_here> = yes
}In code terms, this means that the game going from the country to the federation and then grabbing a list of its members, excluding the current country, and checking the triggers against them. So, that’s obviously going to be fewer checks. However, the best version would be this:
Code:
federation.leader = {
<my_triggers_here> = yes
}That version would go straight to the federation and from there straight to its leader in the code, with as little as possible script to code conversion needed and no need to check triggers against any countries to get there. It also happens to be the most readable (readability and better performance very often correlate…).
So in this case, the game would check around 50 countries first the first version, 5 for the second and 1 for the third - not bad for some optimisations! Using a similar logic, it is always better to use something that isn’t checking all objects in the galaxy (esp. all pops or all planets) if at all possible but rather a filtered list, e.g. any_planet_within_border instead of any_planet = { solar_system.owner = { is_same_value = prevprev } } (you laugh, but I’ve seen it). And, indeed, one can almost always check any_owned_fleet instead of any_owned_ship.
Another important improvement we added in 2.6 was any_owned_species, which can replace many any_owned_pop checks (specifically the ones that check for traits and so on of the pop) and mean that way, way fewer objects have to be checked (in a xenophobic empire, it could be single figures for any_owned_species and thousands for any_owned_pop).
Sometimes you can avoid scopes completely
On a similar note, if you can check something without doing things with scopes, that’s always going to be better. So, if one wants to check whether a planet has more than two pops working as miners, one could do this two ways:
Code:
count_owned_pop = {
count > 2
limit = {
has_job = miner
}
}
Code:
num_assigned_jobs = {
job = miner
value >= 2
}The former will check each pop on the planet and see whether it has the miner job, and then see whether the number that do is higher than 2. The latter will check a cached number that the game has already calculated and see if it is more than 2, which is much quicker to do.
Some things are just expensive
Not every check or effect is equal. Checking a flag or a value is generally pretty simple, and changing it is usually not much more complicated. If, however, the game has to recalculate stuff, then it will take longer, because it’s not just looking up a number it already knows. Creating new stuff is also more expensive, both because it’s doing something somewhat complicated (the create_species effect is, I kid you not, more than 600 lines of C++ code...), and because it’ll probably have to recalculate all sorts of values once this is done. It can be a bit tricky to know which triggers and effects are going to be bad, but as a rule, these cases are what you should look out for:
- Anything where you are creating a new scope e.g. create_country, create_species, modify_species
- Anything that needs you to calculate or recalculate pathfinding (e.g. can_access_system trigger, creating new hyperlanes, especially creating new systems)
- Anything that calculates pops (changing around pop jobs on a planet, for instance)
If it must be done...
Sometimes, bad things must be done. In these cases, it is best to still use the not so great things with precision. When the game is checking triggers e.g. for an event, it’ll generally stop checking them at the first point something returns false (I’m told this is called “short-circuit evaluation”), so you’ll want to do something like this:
Code:
trigger = {
has_country_flag = flag_that_narrows_things_down_loads
<something really horrible here>
}I recently did something like this to the refugee pop effect. It was previously a little bit insane (see 01_scripted_triggers_refugees.txt for the full horror). In total, it would check a variation of the following up to eight times:
Code:
any_relation = {
is_country_type = default
has_communications = prev #relations include countries that have made first contact but not established comms
NOT = { has_policy_flag = refugees_not_allowed }
prevprev = { #this ensures Pop scope, as root will not always be pop scope
OR = {
has_citizenship_type = { type = citizenship_full country = prev }
has_citizenship_type = { type = citizenship_caste_system country = prev }
AND = {
has_citizenship_type = { type = citizenship_limited country = prev }
has_citizenship_type = { type = citizenship_caste_system_limited country = prev }
prev = { has_policy_flag = refugees_allowed }
}
}
}
any_owned_planet = {
is_under_colonization = no
is_controlled_by = owner
has_orbital_bombardment = no
}
}Where it varied was simply the last any_owned_planet: It would try and find a relation with a really good planet for the pop to live on, then a pretty good, then a pretty decent, and then finally settle for just any old planet. Which is, obviously, pretty inefficient, since the list of countries that welcome refugees does not change between each of the 8 times you check it. My way of avoiding this - and making the script way more readable, whilst I was at it - was to set a flag before any of the checks, like this:
Code:
every_relation = {
limit = {
has_any_habitability = yes #bare minimum for being a refugee destination
}
set_country_flag = valid_refugee_destination_for_@event_target:refugee_pop
}The checks then simply had to be “does the country have the flag, if yes, does it have a good enough planet”:
Code:
has_good_habitability_and_housing = {
has_country_flag = valid_refugee_destination_for_@event_target:refugee_pop
any_owned_planet = {
habitability = { who = event_target:refugee_pop value >= 0.7 }
free_housing >= 1
is_under_colonization = no
is_controlled_by = owner
has_orbital_bombardment = no
}
}One can also similarly use if-limits and elses (and, even better, switches when possible - those are the best for performance) in triggers to narrow down the checks down and make things far more readable whilst you are at it. I recently went through the species rights files and redid the allow triggers for sanity’s sake:
(Before)
Code:
custom_tooltip = {
fail_text = MACHINE_SPECIES_NOT_MACHINE
OR = {
has_trait = trait_mechanical
has_trait = trait_machine_unit
from = { has_valid_civic = civic_machine_assimilator }
}
}
custom_tooltip = {
fail_text = ASSIMILATOR_SPECIES_NOT_CYBORG
OR = {
NOT = { from = { has_valid_civic = civic_machine_assimilator } }
AND = {
OR = {
has_trait = trait_cybernetic
has_trait = trait_machine_unit
has_trait = trait_mechanical
}
from = { has_valid_civic = civic_machine_assimilator }
}
}
}(After)
Code:
if = {
limit = {
from = { NOT = { has_valid_civic = civic_machine_assimilator } }
}
custom_tooltip = {
fail_text = MACHINE_SPECIES_NOT_MACHINE
OR = {
has_trait = trait_mechanical
has_trait = trait_machine_unit
}
}
}
else = {
custom_tooltip = {
fail_text = ASSIMILATOR_SPECIES_NOT_CYBORG
OR = {
has_trait = trait_cybernetic
has_trait = trait_machine_unit
has_trait = trait_mechanical
}
}
}The second version will be more efficient, since it is only checking e.g. whether the species has the mechanical trait or whether the country has the assimilator civic once instead of twice, and also, the triggers in the second custom tooltip aren’t obscenely weird anymore. (I also removed all the NANDs, because they broke my brain).
Happy New Year
I can’t write this dev diary without telling you about the “Happy New Year” bug. Basically, we were playing dev MP on a reasonably large galaxy and reached reasonably late into the game, and since we were all working remotely on wildly varying computers and internet connection speeds, the performance was perhaps a tad sluggish, but still acceptable for the most part. Then, suddenly, we noticed huge lag spikes - 20 seconds and more - on the 1st of January. So noticeable were these spikes that we began wishing each other a Happy New Year each time the game froze!
It just so happened that the onset of this lag coincided with several large empires deciding to become synthetic and starting to assimilating their empires. Now, assimilation falls into the category of things that are done in script that maybe, in hindsight, should probably not have been done that way… and works by firing an event for each assimilating country every 1st of January. This event in turn fired an event for each of their planets that selected a bunch of pops on the planet and used modify_species on each of them at least once, but sometimes up to four times. This added up to a fairly significant performance hog!
After trying various solutions, it turned out the best way to fix this was to first go through every_owned_species from the country scope, check whether this species should be assimilated, and if so use modify_species to create the species it would assimilate to, setting a species flag that pointed to the species it was being assimilated from. Then, instead of creating a new species for every pop that was assimilated, the scripts were rewritten to find the already-created species that the pop should become, and simply use change_species on it. The result is still unreadable script (I will spare your eyes and not post it here), but in my tests it reduced the yearly tick by over 50%, thanks to the complicated effect (modify_species) being run as seldom as possible.
---
That’s it for me, for now! I’m guessing this was a bit of a drier dev diary than usual for most of you, but hopefully it was interesting nonetheless
As a final note, I suspect that the more intrepid among you could call me out on bits of script in the base game that don’t quite live up to these guidelines. Please, feel free to do so, because there are few things more satisfying in this profession than taking something really horrible and making something less horrible out of it!
