• We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.
Hi everyone! I am Caligula, one of Stellaris’ Content Designers, which means that I do a variety of tasks based around narrative writing and scripting - “scripting” being our term for doing things that is somewhat similar to programming, but without changing the source code. In other words, I do what modders do (though I have the significant advantage of also being able to peek into the source code and change it around when needed). Every Content Designer has their niche, and mine is that when a particularly complicated system needs to be scripted in (or, more frequently, is giving some sort of trouble - the War in Heaven still gives me nightmares...), I step into the breach.

Now, we have a lot of exciting stuff to show off in the weeks and months to come, but for today, inspired by some questions that were asked after the last dev diary, I’m going to be writing about the technical side of scripting for modders and aspiring modders, specifically with an eye on what can cause performance problems and how to avoid making bad scripts.

The Stellaris scripting language is a very powerful tool, and a lot can be done with it, but first of all, a note of caution: just because something is possible, does not mean it should be done. I can’t really stress this enough, because (and I speak from experience here) this attitude will almost certainly end up causing both performance issues and unreadable scripts that you will not be able to disentangle six months later when you realise some part of it is broken. Though it should be borne in mind that doing something in code is, by definition, faster: in code, you can check a single function and be done with it, but if you want it to be accessible through script, there’s a fair few necessary functions it has to go through before you get to checking your function (turning the line of script into a code command, checking whether it’s used in the right scope, etc etc) - hence why some things are hardcoded, and also why hacky solutions to problems can end up being quite bad. So, the first question to consider is, should I really be doing this?

But who am I kidding, I’m speaking to modders here, so of course you will do it :D So without further ado...

What causes performance issues?

Every time you run a check or execute an effect, this will take a very tiny amount of your computer’s processing power. With a few exceptions that should be used sparingly (I’ll get to those later), this is totally fine and is needed to do anything at all. It is when the check is repeated often, over lots of objects, that problems happen. In practice, this usually means pops are the cause, though running something across all planets in the galaxy is also a pretty bad idea.

As a first step, when possible, it is a good idea to control when your script is run. The best way to do this is by setting where events are fired and using on_actions (or firing events from decisions and the like) wherever possible, instead of mean time to happen or, even worse, just setting an event to try and fire every day. If a degree of randomness is needed, one could also fire a hidden event via, say, a yearly pulse and then firing the actual event you want with a random delay (for an example, check out event action.220).

Of course, not everything is events, and unfortunately, in Stellaris, a lot of stuff is done with pops! Job weights and other triggers in the jobs files, in particular, have been shown to cause issues in the past. As a rule of thumb, even if you can do super cool stuff, it is a bad idea to do any too complicated script pyromania on jobs. For example, if you were to make a job weight dependent on there being no other pops on the planet that are unemployed (using planet = { any_owned_pop = { is_unemployed = yes } }), then you are doing a regular check on every pop on the planet that then checks every other pop on the planet, i.e. pops squared. Once you reach the late game, this is pretty much guaranteed to cause issues.

So what can be done?

Avoiding nested loops, making sure your events are fired appropriately and avoiding pops when possible will get you some way, but we can do better than that. Here is my list of advice for optimising scripts:

Always use the most appropriate scope

Say you want to check something on the leader of the current country’s federation. One could, theoretically, do it this way:
Code:
        any_country = {
            is_in_federation_with = root
            is_federation_leader = yes
            <my_triggers_here> = yes
        }

This'll run through all countries in the game and see whether they are in the same federation as you, including the space amoeba country and the vile Pasharti (and who would want them in a federation?). That’s a bunch that are definitely irrelevant. So a better check would be to do it this way:
Code:
        any_federation_ally = {
            is_federation_leader = yes
            <my_triggers_here> = yes
        }

In code terms, this means that the game going from the country to the federation and then grabbing a list of its members, excluding the current country, and checking the triggers against them. So, that’s obviously going to be fewer checks. However, the best version would be this:
Code:
        federation.leader = {
            <my_triggers_here> = yes
        }

That version would go straight to the federation and from there straight to its leader in the code, with as little as possible script to code conversion needed and no need to check triggers against any countries to get there. It also happens to be the most readable (readability and better performance very often correlate…).

So in this case, the game would check around 50 countries first the first version, 5 for the second and 1 for the third - not bad for some optimisations! Using a similar logic, it is always better to use something that isn’t checking all objects in the galaxy (esp. all pops or all planets) if at all possible but rather a filtered list, e.g. any_planet_within_border instead of any_planet = { solar_system.owner = { is_same_value = prevprev } } (you laugh, but I’ve seen it). And, indeed, one can almost always check any_owned_fleet instead of any_owned_ship.

Another important improvement we added in 2.6 was any_owned_species, which can replace many any_owned_pop checks (specifically the ones that check for traits and so on of the pop) and mean that way, way fewer objects have to be checked (in a xenophobic empire, it could be single figures for any_owned_species and thousands for any_owned_pop).

Sometimes you can avoid scopes completely

On a similar note, if you can check something without doing things with scopes, that’s always going to be better. So, if one wants to check whether a planet has more than two pops working as miners, one could do this two ways:
Code:
        count_owned_pop = {
            count > 2
            limit = {
                has_job = miner
            }
        }
Code:
        num_assigned_jobs = {
            job = miner
            value >= 2
        }

The former will check each pop on the planet and see whether it has the miner job, and then see whether the number that do is higher than 2. The latter will check a cached number that the game has already calculated and see if it is more than 2, which is much quicker to do.

Some things are just expensive

Not every check or effect is equal. Checking a flag or a value is generally pretty simple, and changing it is usually not much more complicated. If, however, the game has to recalculate stuff, then it will take longer, because it’s not just looking up a number it already knows. Creating new stuff is also more expensive, both because it’s doing something somewhat complicated (the create_species effect is, I kid you not, more than 600 lines of C++ code...), and because it’ll probably have to recalculate all sorts of values once this is done. It can be a bit tricky to know which triggers and effects are going to be bad, but as a rule, these cases are what you should look out for:
  • Anything where you are creating a new scope e.g. create_country, create_species, modify_species
  • Anything that needs you to calculate or recalculate pathfinding (e.g. can_access_system trigger, creating new hyperlanes, especially creating new systems)
  • Anything that calculates pops (changing around pop jobs on a planet, for instance)

If it must be done...

Sometimes, bad things must be done. In these cases, it is best to still use the not so great things with precision. When the game is checking triggers e.g. for an event, it’ll generally stop checking them at the first point something returns false (I’m told this is called “short-circuit evaluation”), so you’ll want to do something like this:
Code:
    trigger = {
        has_country_flag = flag_that_narrows_things_down_loads
        <something really horrible here>
    }

I recently did something like this to the refugee pop effect. It was previously a little bit insane (see 01_scripted_triggers_refugees.txt for the full horror). In total, it would check a variation of the following up to eight times:
Code:
        any_relation = {
            is_country_type = default
            has_communications = prev #relations include countries that have made first contact but not established comms
            NOT = { has_policy_flag = refugees_not_allowed }
            prevprev = { #this ensures Pop scope, as root will not always be pop scope
                OR = { 
                    has_citizenship_type = { type = citizenship_full country = prev }
                    has_citizenship_type = { type = citizenship_caste_system country = prev }
                    AND = {
                        has_citizenship_type = { type = citizenship_limited country = prev }
                        has_citizenship_type = { type = citizenship_caste_system_limited country = prev }
                        prev = { has_policy_flag = refugees_allowed }
                    }
                }
            }
            any_owned_planet = {
                is_under_colonization = no
                is_controlled_by = owner
                has_orbital_bombardment = no
            }
        }

Where it varied was simply the last any_owned_planet: It would try and find a relation with a really good planet for the pop to live on, then a pretty good, then a pretty decent, and then finally settle for just any old planet. Which is, obviously, pretty inefficient, since the list of countries that welcome refugees does not change between each of the 8 times you check it. My way of avoiding this - and making the script way more readable, whilst I was at it - was to set a flag before any of the checks, like this:
Code:
        every_relation = {
            limit = {
                has_any_habitability = yes #bare minimum for being a refugee destination
            }
            set_country_flag = valid_refugee_destination_for_@event_target:refugee_pop
        }

The checks then simply had to be “does the country have the flag, if yes, does it have a good enough planet”:
Code:
has_good_habitability_and_housing = {
    has_country_flag = valid_refugee_destination_for_@event_target:refugee_pop
    any_owned_planet = {
        habitability = { who = event_target:refugee_pop value >= 0.7 }
        free_housing >= 1
        is_under_colonization = no
        is_controlled_by = owner
        has_orbital_bombardment = no                                
    }
}

One can also similarly use if-limits and elses (and, even better, switches when possible - those are the best for performance) in triggers to narrow down the checks down and make things far more readable whilst you are at it. I recently went through the species rights files and redid the allow triggers for sanity’s sake:

(Before)
Code:
        custom_tooltip = {
            fail_text = MACHINE_SPECIES_NOT_MACHINE
            OR = {
                has_trait = trait_mechanical
                has_trait = trait_machine_unit
                from = { has_valid_civic = civic_machine_assimilator }
            }
        }
        custom_tooltip = {
            fail_text = ASSIMILATOR_SPECIES_NOT_CYBORG
            OR = {
                NOT = { from = { has_valid_civic = civic_machine_assimilator } }
                AND = {
                    OR = {
                        has_trait = trait_cybernetic
                        has_trait = trait_machine_unit
                        has_trait = trait_mechanical
                    }
                    from = { has_valid_civic = civic_machine_assimilator }
                }
            }
        }

(After)
Code:
        if = {
            limit = {
                from = { NOT = { has_valid_civic = civic_machine_assimilator } }
            }
            custom_tooltip = {
                fail_text = MACHINE_SPECIES_NOT_MACHINE
                OR = {
                    has_trait = trait_mechanical
                    has_trait = trait_machine_unit
                }
            }
        }
        else = {
            custom_tooltip = {
                fail_text = ASSIMILATOR_SPECIES_NOT_CYBORG
                OR = {
                    has_trait = trait_cybernetic
                    has_trait = trait_machine_unit
                    has_trait = trait_mechanical
                }
            }
        }

The second version will be more efficient, since it is only checking e.g. whether the species has the mechanical trait or whether the country has the assimilator civic once instead of twice, and also, the triggers in the second custom tooltip aren’t obscenely weird anymore. (I also removed all the NANDs, because they broke my brain).

Happy New Year

I can’t write this dev diary without telling you about the “Happy New Year” bug. Basically, we were playing dev MP on a reasonably large galaxy and reached reasonably late into the game, and since we were all working remotely on wildly varying computers and internet connection speeds, the performance was perhaps a tad sluggish, but still acceptable for the most part. Then, suddenly, we noticed huge lag spikes - 20 seconds and more - on the 1st of January. So noticeable were these spikes that we began wishing each other a Happy New Year each time the game froze!

It just so happened that the onset of this lag coincided with several large empires deciding to become synthetic and starting to assimilating their empires. Now, assimilation falls into the category of things that are done in script that maybe, in hindsight, should probably not have been done that way… and works by firing an event for each assimilating country every 1st of January. This event in turn fired an event for each of their planets that selected a bunch of pops on the planet and used modify_species on each of them at least once, but sometimes up to four times. This added up to a fairly significant performance hog!

After trying various solutions, it turned out the best way to fix this was to first go through every_owned_species from the country scope, check whether this species should be assimilated, and if so use modify_species to create the species it would assimilate to, setting a species flag that pointed to the species it was being assimilated from. Then, instead of creating a new species for every pop that was assimilated, the scripts were rewritten to find the already-created species that the pop should become, and simply use change_species on it. The result is still unreadable script (I will spare your eyes and not post it here), but in my tests it reduced the yearly tick by over 50%, thanks to the complicated effect (modify_species) being run as seldom as possible.

---

That’s it for me, for now! I’m guessing this was a bit of a drier dev diary than usual for most of you, but hopefully it was interesting nonetheless :) As a final note, I suspect that the more intrepid among you could call me out on bits of script in the base game that don’t quite live up to these guidelines. Please, feel free to do so, because there are few things more satisfying in this profession than taking something really horrible and making something less horrible out of it!
 
PDX creates a 4x with pops and sell is like "it's going to be even better than victoria 2!!!, you can do anything with pops!", and then it's the fault of modders that create scripts that touch pops. Besides the product crawls down to unplayable speeds without mods.

All of this was irrelevant when planets had 25 pops. By allowing hundreds of pops on colonies without a proper system to process then, most/all scripts broke (performance wise), and now it's the fault of the scripts and not of the engine. No, the assimilation script was not to blame, the fact that you chose to throw 10.000 pops at it, and have the engine operate beyond its design specs is.

Yes, you can spend time to improve the scripts, but that will always have a marginal or minimal improvement, without solving the undelying issue.
a game engine is not to a game what a car engine is to a car, a game engine is simply the code that you start with before you build your game. An engine cannot "operate beyond its design specs" as you said unless that means not working entirely, the lag is caused on the scripting side and the game code, pops themselves are a construct of the game code and are not present in other games using the same game engine. It is easy to tell that you can fix the problems with Stellaris without changing the game engine or the amount of pops because vic 2 does handle about 10,000 pops and has the same engine

if they reduce the amount of checks made on the pop or optimize how the checks are done they will solve the problem with lag
 
  • 12
  • 1Like
  • 1
  • 1
Reactions:
Code:
#Scope type varies depending on what is selected
# This = selected object or player country
# From = player country

example_button_effect = {
    potential = {
        always = yes
    }
    allow = {
        always = yes
    }
    
    effect = {
        add_monthly_resource_mult = {
            resource = minerals
            value = 5
            min = 5
            max = 500
        }       
    }
}
# This = selected object or player country
Fix the button effect bug?
 
As someone who occasionally has to write mods (only way to add new portraits, for example), I like reading updates like this one. And oh yes, I do know about "you can do this, but shouldn't". One time, I wrote a tiny mod to change my starting date to a custom one, since it kind of looked silly that every alien species under the sun should coincidentally decide to use the same calendar. For a while, everything was well. But then loads of weird bugs, like all my factions disappearing, would inevitably cripple my game.

Eventually, I realized that changing the in-game date was causing the problems, because for some reason a lot of triggers in Stellaris seem to be coded to assume the startdate is always 2200. Change that to 800 for your own weird space lizard civ, and all hell will break loose. A nice lesson into what you can't do when modding Stellaris.
 
  • 2
  • 1Like
  • 1
Reactions:
# This = selected object or player country
Fix the button effect bug?
that is just a comment to tell you what "This" does, it is not a part of the script and the computer never reads it as it has a "#" at the front which denotes that it is for humans and not the computer
 
  • 2
Reactions:
As someone who occasionally has to write mods (only way to add new portraits, for example), I like reading updates like this one. And oh yes, I do know about "you can do this, but shouldn't". One time, I wrote a tiny mod to change my starting date to a custom one, since it kind of looked silly that every alien species under the sun should coincidentally decide to use the same calendar. For a while, everything was well. But then loads of weird bugs, like all my factions disappearing, would inevitably cripple my game.

Eventually, I realized that changing the in-game date was causing the problems, because for some reason a lot of triggers in Stellaris seem to be coded to assume the startdate is always 2200. Change that to 800 for your own weird space lizard civ, and all hell will break loose. A nice lesson into what you can't do when modding Stellaris.

Weird. I set my start date to 1 and have since 1.9 without ill effect.
 
Lua would be the conventional choice for this purpose; it's widely used in game modding. I've also seen games (including another space 4x, from 2006) that have what is technically their own unique programming language, but at least use something that is both clearly Turing-complete and actually compile it to binary.

Mind you, the weird Clausewitz scripting language is actually easier for people to learn than you might think, making accessibility of the modding tools much greater than if you needed a full application programming language. It's also just simpler, in some ways.

PDX scripting language and generally DSL is a dead idea.

Regular programming/scripting languages such as Lua are better because of globally developed ecosystem for said programming/scripting languages:
  1. IDE
  2. Debugger (PDX scripting language has none)
  3. Profiler (PDX scripting language has none)
  4. JIT compiler (PDX scripting language has none)
  5. You can read books on Lua, watch YouTube videos about Lua, ask questions about Lua on StackOverflow
  6. You can get a programmer job once you're done modding in Lua
There is nothing special in PDX scripting that can't be expressed in regular programming language. For example:
Code:
    count_owned_pop = {
            count > 2
            limit = {
                has_job = miner
            }
        }
can be rewritten in Java like:
Code:
    pops.stream().filter(pop -> pop.job == Jobs.MINER).count() > 2
etc...
 
Last edited:
  • 3Like
  • 1Haha
  • 1
  • 1
Reactions:
PDX scripting language and generally DSL is a dead idea.

Regular programming/scripting languages such as Lua are better because of globally developed ecosystem for said programming/scripting languages:
  1. IDE
  2. Debugger (PDX scripting language has none)
  3. Profiler (PDX scripting language has none)
  4. JIT compiler (PDX scripting language has none)
  5. You can read books on Lua, watch YouTube videos about Lua, ask questions about Lua on StackOverflow
  6. You can get a programmer job once you're done modding in Lua
There is nothing special in PDX scripting that can't be expressed in regular programming language. For example:
Code:
    count_owned_pop = {
            count > 2
            limit = {
                has_job = miner
            }
        }
can be rewritten in Java like:
Code:
    pops.stream().filter(pop -> pop.job == Jobs.MINER).count() > 2
etc...
true, but changing the language will not change the actual performance issues, and it would just be massive work to change to Lua, not just for PDX but also for Modders who have to change all theyr mods completely to the "new" language...exapt someone is creating a converter for this... but yah, still the performance problems would remain with another language...
 
  • 4
  • 1
Reactions:
PDX scripting language and generally DSL is a dead idea.

Regular programming/scripting languages such as Lua are better because of globally developed ecosystem for said programming/scripting languages:
  1. IDE
  2. Debugger (PDX scripting language has none)
  3. Profiler (PDX scripting language has none)
  4. JIT compiler (PDX scripting language has none)
  5. You can read books on Lua, watch YouTube videos about Lua, ask questions about Lua on StackOverflow
  6. You can get a programmer job once you're done modding in Lua
There is nothing special in PDX scripting that can't be expressed in regular programming language. For example:
Code:
    count_owned_pop = {
            count > 2
            limit = {
                has_job = miner
            }
        }
can be rewritten in Java like:
Code:
    pops.stream().filter(pop -> pop.job == Jobs.MINER).count() > 2
etc...
I wouldn't even have considered starting Stellaris modding if it looked like what you are suggesting, just like most people. Paradox script is simple to learn, that's why Stellaris has 18,734 and Hearts of Iron IV 28,851 mods.

Also, Stellaris PDX code does have code validator tool.
 
  • 7
Reactions:
true, but changing the language will not change the actual performance issues, and it would just be massive work to change to Lua, not just for PDX but also for Modders who have to change all theyr mods completely to the "new" language...exapt someone is creating a converter for this... but yah, still the performance problems would remain with another language...

Changing one interpreted language to another interpreted language indeed won't make a difference.

However Lua has a JIT compiler (https://en.wikipedia.org/wiki/Just-in-time_compilation) which offers order of magnitude performance improvements:
"The C program runs three times faster than the LuaJIT program. The LuaJIT program runs almost 25-times faster than the ordinary Lua program."
https://eklausmeier.wordpress.com/2016/04/05/performance-comparison-c-vs-lua-vs-luajit-vs-java

So, if PDX would implement JIT for their scripts, there will be a performance boost...

Alternatively, PDX could implement PDX script to C++ transpiler, convert vanilla scripts to C++ and link main EXE to the generated C++ code. Mods would still be executed by the interpreter (slow as they are now).
 
  • 2
Reactions:
Changing one interpreted language to another interpreted language indeed won't make a difference.

However Lua has a JIT compiler (https://en.wikipedia.org/wiki/Just-in-time_compilation) which offers order of magnitude performance improvements:


So, if PDX would implement JIT for their scripts, there will be a performance boost...

Alternatively, PDX could implement PDX script to C++ transpiler, convert vanilla scripts to C++ and link main EXE to the generated C++ code. Mods would still be executed by the interpreter (slow as they are now).

Do we know for sure that the PDX script is 100% interpreted? Rather sad if so.

As you say, it's not like they even need to switch to a different language. On top of what you suggest, they could just write a PDX script to Lau converter, for example. Or PDX script to java byte-code converter. For sure they could pre-compile the standard scripts and have the interpreter for mods. Plenty of options.
 
  • 1
Reactions:
The result is still unreadable script (I will spare your eyes and not post it here), but in my tests it reduced the yearly tick by over 50%, thanks to the complicated effect (modify_species) being run as seldom as possible.
This part is really good news. I often suffer from that very same lag, when I decide to turn thousands of pops into toasters at once.
 
  • 1
Reactions:
"The Perils of Scripting and How to Avoid Them"

Sentence like these, usually show that there is an issue. A underlying asumption that is a dead end: "It works fine, as long as nobody ever makes a mistake."

The asumption that nobody makes a mistake, does not survive success. The easier to use something is or the more popular, the more programmers you get. And the more likely someone will make a mistake. Another rule of programming takes over: Everything that can happen, will happen if something is just used often enough.

"Nobody makes a mistake" is what killed Cooperative Multitasking. It made programmer easier/more accessible, meaning more people made a mistake.
"Nobody makes a mistake" is what killed handling Naked Pointers. And ended up with us having to invent Memory Protection and workarounds to handling Naked Pointers.
I do not think it is going to work here either.

It should not be possible to make a mistake, or at least you should have to go out of your way:
- Taking a worst case scenario like itterating over all empires, every pop: You may want to simply disallow those two to be in the same event. It forces the programmer to fire off a seperate event for each country, wich should allow decent multitasking. Or get's the user not to try that in the first place and look for something more efficient
- You could enforce that flags have to be checked before any other condition in every scope, to allow the more efficient short-circuiting to hit first
- you could disallow the "has_job" limit for count pops scope
- stuff that uses a loop could be capped to 10k itterations or something like that

It may feel a bit restrictive at first. But in the end, 90% of all programming language is there to prevent the programmer (both current and any future one) from making a mistake.
 
  • 5
  • 3Like
  • 3
Reactions:
if there was code to prevent the devs from making mistakes the 2.2 fustercluck wouldn't have happened
 
  • 4Like
Reactions:
if there was code to prevent the devs from making mistakes the 2.2 fustercluck wouldn't have happened
No code is able to prevent bugs.

But we can atleat avoid doing the things we know are stupid ideas from performance point of view.
 
  • 1Like
  • 1
Reactions: