You have already seen the dev diary in which it is described how AI objectives work. There is a more significant change from conventional AI in Imperator that is necessary to know about in order to mod AI behavior throughout the game, so I thought I should spend an hour of my evening writing about it here.
Background
First, the model for the AI in Imperator was Europa Universalis 4 and it serves as a reference during development even though almost all code is new.
If you know EU4, you probably know that AI there are a lot of actions (choices) in the game the AI is taking independently. Quite often, the AI of those choices is hardcoded, but there are also a lot of scripted AI choices.
In Imperator, a far larger part of the game is scripted. What this means is coding AI outside script is more difficult. Another thing you have undoubtedly noticed is that Imperator makes heavy use of modifiers as a common interface for gameplay elements to impact each other.
The way that was solved in EU4 was to have an ai_will_do weighted trigger on every choice that gives an modifier or otherwise has scripted effects. In AI you could call this utility value maximization. This is pretty much the standard in Paradox games. Specifying exactly what the AI should be doing is surely great, or could there be any downsides?
It turns out to be a maintenance nightmare. If you have instructed the AI to do something, it will not change without you editing the script. Moreover, if you want the AI to pick a different modifier, you have to change the scripts for many (or all) the choices in the game that give that modifier. You want the AI to place higher emphasis on heavy infantry discipline? Congratulations, find all the places that give this modifier.
There are additional problems with the basic weighted trigger model. Typically, the weighted trigger knows nothing about the potential for making other choices other than by looking at the game state. Other choices can’t know that the AI is going to be making a choice at some point in the future, so they will quite frequently do conflicting things. In particular, this can involve e.g. spending gold in one choice that ends up blocking a different more important choice.
One more thing worth mentioning about the basic weighed trigger model is that is a victim of circumstances. In algorithmic terms, it is greedy and it is shallow with no hypothesising of what might be.
Lastly, it is not that certain that you know what is actually the best way for the AI to achieve something. Scripts give the illusion of control but when the strategic context is too complex to take isolated choices in the result is necessarily mediocre AI.
Action Plans
Imperator’s attempt at solving the above comes with a different kind of AI here termed an action plan that:
The subject of action planning is a greatly studied problem in literature and Imperator’s solution is not particularly advanced. However, it cannot be given that the space of possible plans is exponential and there is no general algorithm with less than exponential time complexity on the number of actions (plan length), and meanwhile we have nearly a thousand countries requiring planning in real time.
So what is an action plan? It is just a list of actions to be taken, ideally in the order defined by the list. To see how this looks in game, type “aiview” in console and hover the country flag in the left top corner:
The action plan has a (utility) value that specifies how valuable the resulting situation is to the AI after the plan’s execution. It also has a latency which is the sum of delays on the actions in the plan by having to wait for their execution - this measure allows trying to optimize the action order for faster solutions.
How is the plan generated? Initially, there was a complicated evolutionary algorithm attempting to create a good plan from random seed plans, but that approach didn’t work well because of performance requirements and instability in found solutions. Instead, now AI code simply tries to ensure that actions of various kinds have been appended to the plan where appropriate. This very much reduces the ability to hypothesize on plan variations, but it is a necessary limitation.
The plan order however is periodically optimized (especially for larger countries) to reduce latency and improve the utility value of the plan.
The following game actions currently are governed by the action planning system:
Sounds good so far, so what’s the catch? The usefulness of a plan is only as good as the definition of the utility value function. Simulating all the resulting game worlds is impossible for performance reasons, so a very compact simplified representation has to be used where only some attributes of the country in question are simulated. Developing that representation in a way that doesn’t blow up performance or take way too much time has been quite a challenge. Fortunately, that is not something that impacts you as a modder.
Modding the Plan Value (Utility Function)
If you look in the game common data directory, you will see a folder called ai_plan_goals. Open it, and you shall find something like this:
Each of these clauses is called a goal. Goals are updated once a year or on war status or ruler change. All goals with triggers evaluating to true will have their components added together. The components are then used as parameters for the AI, and in the case of modifiers explicitly stating to the AI how good they are, providing a way of doing so even in (the plentiful) cases the limited internal simulation doesn’t adequately cover the modifiers.
Once you’ve modded the utility function, you can see what it does on many action as long as the aiview console command is active by hovering over actions in tooltips:
This displays the value of taking the action on the current game state and is useful when understanding the plan value. For many types of actions, AI will always pick the highest valued choice, however it can also factor in latency in some cases or pick randomly among the best 3 choices. Note that “pick” here quite often translates to “add to plan and execute later”, although for some other actions it’s instantaneous as it was more performant that way.
A cool (if not very useful aspect due to the amount of work necessary on the developer end coupled with an AI second design process) is that the AI understands some triggers and effects as long as it has been coded to evaluate the impact on the plan. This means that for some simple events, even if you do not specify AI chances, the AI automatically understands the event choices and acts accordingly.
Quite often there are also ai_will_do sections or similar in the script files giving you conventional fine grained control where you need it.
Conclusion
Does it work? Arguably so, but the nature of a game project’s development is you only get one shot at a solution and can’t compare alternative solutions. Many limits are related to performance. If every player had an infinitely fast serial computer, the entirety of AWS, or a quantum computer of appreciable memory capacity it would help a lot. A more relevant question is, was it worth it? Time will tell, this system does take more effort to code on the developer’s end. I am always trying to improve the way of doing things and if we didn’t try, we wouldn’t get anywhere.
An interesting area of future work Master projects would be to use machine learning to learn the internal simulation (as in, how actions impact country attributes) instead of manually having to develop and maintain this. The reason why this is not so easy is that of temporal credit assignment. It is fine to code (and mod the utility function) because even if it may be inaccurate, hand designing it keeps the amount of strange things the AI might do under control. Then again, given the progress of DeepMind on e.g. AlphaStar this might be a dead end, human designability objections aside.
Work always continues, especially since the ability of designers to generate new AI work always outpaces AI implementation efforts. At least with this system modders should have a very solid way of influencing AI decision making across the game from a single file and can change things as they see fit.
Background
First, the model for the AI in Imperator was Europa Universalis 4 and it serves as a reference during development even though almost all code is new.
If you know EU4, you probably know that AI there are a lot of actions (choices) in the game the AI is taking independently. Quite often, the AI of those choices is hardcoded, but there are also a lot of scripted AI choices.
In Imperator, a far larger part of the game is scripted. What this means is coding AI outside script is more difficult. Another thing you have undoubtedly noticed is that Imperator makes heavy use of modifiers as a common interface for gameplay elements to impact each other.
The way that was solved in EU4 was to have an ai_will_do weighted trigger on every choice that gives an modifier or otherwise has scripted effects. In AI you could call this utility value maximization. This is pretty much the standard in Paradox games. Specifying exactly what the AI should be doing is surely great, or could there be any downsides?
It turns out to be a maintenance nightmare. If you have instructed the AI to do something, it will not change without you editing the script. Moreover, if you want the AI to pick a different modifier, you have to change the scripts for many (or all) the choices in the game that give that modifier. You want the AI to place higher emphasis on heavy infantry discipline? Congratulations, find all the places that give this modifier.
There are additional problems with the basic weighted trigger model. Typically, the weighted trigger knows nothing about the potential for making other choices other than by looking at the game state. Other choices can’t know that the AI is going to be making a choice at some point in the future, so they will quite frequently do conflicting things. In particular, this can involve e.g. spending gold in one choice that ends up blocking a different more important choice.
One more thing worth mentioning about the basic weighed trigger model is that is a victim of circumstances. In algorithmic terms, it is greedy and it is shallow with no hypothesising of what might be.
Lastly, it is not that certain that you know what is actually the best way for the AI to achieve something. Scripts give the illusion of control but when the strategic context is too complex to take isolated choices in the result is necessarily mediocre AI.
Action Plans
Imperator’s attempt at solving the above comes with a different kind of AI here termed an action plan that:
- Separates the concerns of executing an action (making a choice) from planning why it should be executed.
- Represents the AI intent to execute an action ahead of that action’s execution.
- Permits hypothesizing on the consequences of a particular ordering of actions.
The subject of action planning is a greatly studied problem in literature and Imperator’s solution is not particularly advanced. However, it cannot be given that the space of possible plans is exponential and there is no general algorithm with less than exponential time complexity on the number of actions (plan length), and meanwhile we have nearly a thousand countries requiring planning in real time.
So what is an action plan? It is just a list of actions to be taken, ideally in the order defined by the list. To see how this looks in game, type “aiview” in console and hover the country flag in the left top corner:
The action plan has a (utility) value that specifies how valuable the resulting situation is to the AI after the plan’s execution. It also has a latency which is the sum of delays on the actions in the plan by having to wait for their execution - this measure allows trying to optimize the action order for faster solutions.
How is the plan generated? Initially, there was a complicated evolutionary algorithm attempting to create a good plan from random seed plans, but that approach didn’t work well because of performance requirements and instability in found solutions. Instead, now AI code simply tries to ensure that actions of various kinds have been appended to the plan where appropriate. This very much reduces the ability to hypothesize on plan variations, but it is a necessary limitation.
The plan order however is periodically optimized (especially for larger countries) to reduce latency and improve the utility value of the plan.
The following game actions currently are governed by the action planning system:
- Inventions
- Ideas
- Traditions
- Economic Policies (heavily scripted though)
- Omens
- Building constructions
- Pop promotion
- Trade route creation
- Unit abilities (scripts are restrained by resource expenditure)
- Colonization
- War exhaustion
- Stability
- 1.1: Diplomatic stances.
- 1.1: Assimilate Culture, Convert Religion.
Sounds good so far, so what’s the catch? The usefulness of a plan is only as good as the definition of the utility value function. Simulating all the resulting game worlds is impossible for performance reasons, so a very compact simplified representation has to be used where only some attributes of the country in question are simulated. Developing that representation in a way that doesn’t blow up performance or take way too much time has been quite a challenge. Fortunately, that is not something that impacts you as a modder.
Modding the Plan Value (Utility Function)
If you look in the game common data directory, you will see a folder called ai_plan_goals. Open it, and you shall find something like this:
Code:
these_are_always_added = {
trigger = {
always = yes
}
#Economy parameters:
economy_exponent = 1
research_exponent = 1
research_amplifier = 3
army_offensive_exponent = 1
army_offensive_amplifier = 1.0
...
#Modifier goal coefficients:
global_unrest = -1
#Personality parameters:
aggressive = 50 #100 = Machiavelli, 0 = pacifist.
trustworthy = 50 #100 = Bamse, 0 = Machiavelli.
}
is_landlocked_aimod = {
trigger = {
has_coasts = no
}
# navals
naval_morale = -1000
naval_morale_modifier = -1000
naval_unit_attrition = -1000
naval_morale_recovery = -1000
naval_raid_cost_modifier = 1000
…
}
is_rome_aimod = {
trigger = {
tag = ROM
}
aggressive = 60
}
is_tribal_aimod = {
trigger = {
is_tribal = yes
}
aggressive = -25
}
is_lunatic_aimod = {
trigger = {
current_ruler = {
has_trait = lunatic
}
}
aggressive = 500
trustworthy = -500
economy_exponent = -2 #Should do interesting things with economy.
global_unrest = 2 #Unrest => good seems right for a lunatic.
}
Each of these clauses is called a goal. Goals are updated once a year or on war status or ruler change. All goals with triggers evaluating to true will have their components added together. The components are then used as parameters for the AI, and in the case of modifiers explicitly stating to the AI how good they are, providing a way of doing so even in (the plentiful) cases the limited internal simulation doesn’t adequately cover the modifiers.
Once you’ve modded the utility function, you can see what it does on many action as long as the aiview console command is active by hovering over actions in tooltips:
This displays the value of taking the action on the current game state and is useful when understanding the plan value. For many types of actions, AI will always pick the highest valued choice, however it can also factor in latency in some cases or pick randomly among the best 3 choices. Note that “pick” here quite often translates to “add to plan and execute later”, although for some other actions it’s instantaneous as it was more performant that way.
A cool (if not very useful aspect due to the amount of work necessary on the developer end coupled with an AI second design process) is that the AI understands some triggers and effects as long as it has been coded to evaluate the impact on the plan. This means that for some simple events, even if you do not specify AI chances, the AI automatically understands the event choices and acts accordingly.
Quite often there are also ai_will_do sections or similar in the script files giving you conventional fine grained control where you need it.
Conclusion
Does it work? Arguably so, but the nature of a game project’s development is you only get one shot at a solution and can’t compare alternative solutions. Many limits are related to performance. If every player had an infinitely fast serial computer, the entirety of AWS, or a quantum computer of appreciable memory capacity it would help a lot. A more relevant question is, was it worth it? Time will tell, this system does take more effort to code on the developer’s end. I am always trying to improve the way of doing things and if we didn’t try, we wouldn’t get anywhere.
An interesting area of future work Master projects would be to use machine learning to learn the internal simulation (as in, how actions impact country attributes) instead of manually having to develop and maintain this. The reason why this is not so easy is that of temporal credit assignment. It is fine to code (and mod the utility function) because even if it may be inaccurate, hand designing it keeps the amount of strange things the AI might do under control. Then again, given the progress of DeepMind on e.g. AlphaStar this might be a dead end, human designability objections aside.
Work always continues, especially since the ability of designers to generate new AI work always outpaces AI implementation efforts. At least with this system modders should have a very solid way of influencing AI decision making across the game from a single file and can change things as they see fit.
- 1