Modding AI in Imperator: Action Planning

  • We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.

Chaingun

Field Marshal
47 Badges
Jul 15, 2002
3.796
2.513
  • Knights of Honor
  • 500k Club
  • Europa Universalis III: Collection
  • Europa Universalis IV: El Dorado
  • Magicka: Wizard Wars Founder Wizard
  • Mount & Blade: Warband
  • Mount & Blade: With Fire and Sword
  • Europa Universalis IV: Common Sense
  • Europa Universalis IV: Cossacks
  • Europa Universalis IV: Mare Nostrum
  • Europa Universalis IV: Rights of Man
  • Steel Division: Normandy 44
  • War of the Vikings
  • Europa Universalis IV: Third Rome
  • BATTLETECH
  • Surviving Mars
  • Age of Wonders III
  • Europa Universalis IV: Rule Britannia
  • Europa Universalis IV: Dharma
  • Imperator: Rome Deluxe Edition
  • Imperator: Rome
  • Prison Architect
  • Imperator: Rome - Magna Graecia
  • Crusader Kings III
  • Europa Universalis III Complete
  • Europa Universalis III
  • Europa Universalis III: Chronicles
  • Europa Universalis III Complete
  • Divine Wind
  • Europa Universalis IV
  • Europa Universalis IV: Art of War
  • Europa Universalis IV: Conquest of Paradise
  • Europa Universalis IV: Wealth of Nations
  • Europa Universalis IV: Call to arms event
  • Hearts of Iron III
  • Heir to the Throne
  • Crusader Kings II
  • Magicka
  • Europa Universalis III Complete
  • Europa Universalis IV: Res Publica
  • Victoria: Revolutions
  • Rome Gold
  • Supreme Ruler 2020
  • Victoria 2
  • Victoria 2: A House Divided
  • Victoria 2: Heart of Darkness
  • Rome: Vae Victis
You have already seen the dev diary in which it is described how AI objectives work. There is a more significant change from conventional AI in Imperator that is necessary to know about in order to mod AI behavior throughout the game, so I thought I should spend an hour of my evening writing about it here.

Background

First, the model for the AI in Imperator was Europa Universalis 4 and it serves as a reference during development even though almost all code is new.

If you know EU4, you probably know that AI there are a lot of actions (choices) in the game the AI is taking independently. Quite often, the AI of those choices is hardcoded, but there are also a lot of scripted AI choices.

In Imperator, a far larger part of the game is scripted. What this means is coding AI outside script is more difficult. Another thing you have undoubtedly noticed is that Imperator makes heavy use of modifiers as a common interface for gameplay elements to impact each other.

The way that was solved in EU4 was to have an ai_will_do weighted trigger on every choice that gives an modifier or otherwise has scripted effects. In AI you could call this utility value maximization. This is pretty much the standard in Paradox games. Specifying exactly what the AI should be doing is surely great, or could there be any downsides?

It turns out to be a maintenance nightmare. If you have instructed the AI to do something, it will not change without you editing the script. Moreover, if you want the AI to pick a different modifier, you have to change the scripts for many (or all) the choices in the game that give that modifier. You want the AI to place higher emphasis on heavy infantry discipline? Congratulations, find all the places that give this modifier.

There are additional problems with the basic weighted trigger model. Typically, the weighted trigger knows nothing about the potential for making other choices other than by looking at the game state. Other choices can’t know that the AI is going to be making a choice at some point in the future, so they will quite frequently do conflicting things. In particular, this can involve e.g. spending gold in one choice that ends up blocking a different more important choice.

One more thing worth mentioning about the basic weighed trigger model is that is a victim of circumstances. In algorithmic terms, it is greedy and it is shallow with no hypothesising of what might be.

Lastly, it is not that certain that you know what is actually the best way for the AI to achieve something. Scripts give the illusion of control but when the strategic context is too complex to take isolated choices in the result is necessarily mediocre AI.

Action Plans

Imperator’s attempt at solving the above comes with a different kind of AI here termed an action plan that:

  • Separates the concerns of executing an action (making a choice) from planning why it should be executed.
  • Represents the AI intent to execute an action ahead of that action’s execution.
  • Permits hypothesizing on the consequences of a particular ordering of actions.

The subject of action planning is a greatly studied problem in literature and Imperator’s solution is not particularly advanced. However, it cannot be given that the space of possible plans is exponential and there is no general algorithm with less than exponential time complexity on the number of actions (plan length), and meanwhile we have nearly a thousand countries requiring planning in real time.

So what is an action plan? It is just a list of actions to be taken, ideally in the order defined by the list. To see how this looks in game, type “aiview” in console and hover the country flag in the left top corner:

pP7SHV9Bt9Z2Cg6wd0DRme8yiKByPm85KOqSVCZR6nt2HzF9i-2BYLifI86WRT1aKHKIfh8R0TIoZbA79pS9pVbuaiwBEy0LKOueQ4__tVBR9GOqh_6mc9n4V8_GEb2wcgg5332A


The action plan has a (utility) value that specifies how valuable the resulting situation is to the AI after the plan’s execution. It also has a latency which is the sum of delays on the actions in the plan by having to wait for their execution - this measure allows trying to optimize the action order for faster solutions.

How is the plan generated? Initially, there was a complicated evolutionary algorithm attempting to create a good plan from random seed plans, but that approach didn’t work well because of performance requirements and instability in found solutions. Instead, now AI code simply tries to ensure that actions of various kinds have been appended to the plan where appropriate. This very much reduces the ability to hypothesize on plan variations, but it is a necessary limitation.

The plan order however is periodically optimized (especially for larger countries) to reduce latency and improve the utility value of the plan.

The following game actions currently are governed by the action planning system:

  • Inventions
  • Ideas
  • Traditions
  • Economic Policies (heavily scripted though)
  • Omens
  • Building constructions
  • Pop promotion
  • Trade route creation
  • Unit abilities (scripts are restrained by resource expenditure)
  • Colonization
  • War exhaustion
  • Stability
  • 1.1: Diplomatic stances.
  • 1.1: Assimilate Culture, Convert Religion.

Sounds good so far, so what’s the catch? The usefulness of a plan is only as good as the definition of the utility value function. Simulating all the resulting game worlds is impossible for performance reasons, so a very compact simplified representation has to be used where only some attributes of the country in question are simulated. Developing that representation in a way that doesn’t blow up performance or take way too much time has been quite a challenge. Fortunately, that is not something that impacts you as a modder.

Modding the Plan Value (Utility Function)

If you look in the game common data directory, you will see a folder called ai_plan_goals. Open it, and you shall find something like this:

Code:
these_are_always_added = {

    trigger = {

        always = yes

    }

   

    #Economy parameters:

    economy_exponent = 1

    research_exponent = 1

    research_amplifier = 3

    army_offensive_exponent = 1

    army_offensive_amplifier = 1.0

    ...

   

    #Modifier goal coefficients:

    global_unrest = -1

   

    #Personality parameters:

    aggressive = 50 #100 = Machiavelli, 0 = pacifist.

    trustworthy = 50 #100 = Bamse, 0 = Machiavelli.

}


is_landlocked_aimod = {

    trigger = {

        has_coasts = no

    }


    # navals

    naval_morale = -1000

    naval_morale_modifier = -1000

    naval_unit_attrition = -1000

    naval_morale_recovery = -1000

    naval_raid_cost_modifier = 1000

    …

}


is_rome_aimod = {

    trigger = {

        tag = ROM

    }

   

    aggressive = 60

}


is_tribal_aimod = {

    trigger = {

        is_tribal = yes

    }

    aggressive = -25

}


is_lunatic_aimod = {

    trigger = {

        current_ruler = {

            has_trait = lunatic

        }

    }

   

    aggressive = 500

    trustworthy = -500

    economy_exponent = -2 #Should do interesting things with economy.

    global_unrest = 2 #Unrest => good seems right for a lunatic.

}

Each of these clauses is called a goal. Goals are updated once a year or on war status or ruler change. All goals with triggers evaluating to true will have their components added together. The components are then used as parameters for the AI, and in the case of modifiers explicitly stating to the AI how good they are, providing a way of doing so even in (the plentiful) cases the limited internal simulation doesn’t adequately cover the modifiers.

Once you’ve modded the utility function, you can see what it does on many action as long as the aiview console command is active by hovering over actions in tooltips:

g743XlLWH2A2GAduitu_pSxp5j3FPfcTo5DRk49Eogcv4UY_JWycCjixIyVA59Ux12lyuSZqRxl_jOBI0ChATGwY0AayQ78YjYr0Da8_uPjzWsMwCf-j7-u3AbVm0oqiY6Vy6DXc


This displays the value of taking the action on the current game state and is useful when understanding the plan value. For many types of actions, AI will always pick the highest valued choice, however it can also factor in latency in some cases or pick randomly among the best 3 choices. Note that “pick” here quite often translates to “add to plan and execute later”, although for some other actions it’s instantaneous as it was more performant that way.

A cool (if not very useful aspect due to the amount of work necessary on the developer end coupled with an AI second design process) is that the AI understands some triggers and effects as long as it has been coded to evaluate the impact on the plan. This means that for some simple events, even if you do not specify AI chances, the AI automatically understands the event choices and acts accordingly.

Quite often there are also ai_will_do sections or similar in the script files giving you conventional fine grained control where you need it.

Conclusion

Does it work? Arguably so, but the nature of a game project’s development is you only get one shot at a solution and can’t compare alternative solutions. Many limits are related to performance. If every player had an infinitely fast serial computer, the entirety of AWS, or a quantum computer of appreciable memory capacity it would help a lot. A more relevant question is, was it worth it? Time will tell, this system does take more effort to code on the developer’s end. I am always trying to improve the way of doing things and if we didn’t try, we wouldn’t get anywhere.

An interesting area of future work Master projects would be to use machine learning to learn the internal simulation (as in, how actions impact country attributes) instead of manually having to develop and maintain this. The reason why this is not so easy is that of temporal credit assignment. It is fine to code (and mod the utility function) because even if it may be inaccurate, hand designing it keeps the amount of strange things the AI might do under control. Then again, given the progress of DeepMind on e.g. AlphaStar this might be a dead end, human designability objections aside.

Work always continues, especially since the ability of designers to generate new AI work always outpaces AI implementation efforts. At least with this system modders should have a very solid way of influencing AI decision making across the game from a single file and can change things as they see fit.
 
  • 1
Reactions:
This seems like a better approach than the old weight system. How good is the AI at changing its mind?
 
Thx a lot that was very interesting! i really enjoyed reading it.

Just one question related to AI (but I know it may also a question of other game mechanics) but what i have seen on Youtube the AI (for example if playing as Rome) is not trying any invasions? that wouldnt be so bad if its EU4 but in an ancient game where everything interesting to conquer is close to the mediterranean sea but not behind the alps, its quite strange.

Are there good chances too improve this behavior?
 
Thanks for this totally-not-dev-diary on a Sunday! :D
 
Thx a lot that was very interesting! i really enjoyed reading it.

Just one question related to AI (but I know it may also a question of other game mechanics) but what i have seen on Youtube the AI (for example if playing as Rome) is not trying any invasions? that wouldnt be so bad if its EU4 but in an ancient game where everything interesting to conquer is close to the mediterranean sea but not behind the alps, its quite strange.

Are there good chances too improve this behavior?

Though unrelated to thread topic I will answer: There are bugs in AI's naval invasions currently (all will probably always be due to degree of subjectivity), when they work they can be quite nice. There's a lot of reasons why they might not as it's one of the most complicated systems.

Some of them are attempted fixed in 1.0.1 others secretpatch. I say attempted because it's always difficult to know if fixed in if a particular situation a player will encounter.
 
Last edited:
This is great!
Thank you for taking the time to write this. It's great that AI's approach to these mechanics is also moddable.

One of the streamers I watch made a pretty interesting observation. He said that generally speaking, sometimes AI tends to abandon on-going sieges to chase enemy stacks and it ends up having a hard time finishing wars in an efficient manner.

But I'm not sure that making the AI (as in, AI countries) focus more on finishing sieges would work either since I haven't tested it. Is such thing moddable? If not, I suggest you guys look into finding a balance for the AI to not abandon sieges very easily.

Also I know player-AI objectives can be moddable, but I hope that someday it'll be possible to make an AI objective dedicated to sieging, so that when you assign armies to such objective they focus solely on sieging (unless endangered).
 
This is great!
Thank you for taking the time to write this. It's great that AI's approach to these mechanics is also moddable.

One of the streamers I watch made a pretty interesting observation. He said that generally speaking, sometimes AI tends to abandon on-going sieges to chase enemy stacks and it ends up having a hard time finishing wars in an efficient manner.

But I'm not sure that making the AI (as in, AI countries) focus more on finishing sieges would work either since I haven't tested it. Is such thing moddable? If not, I suggest you guys look into finding a balance for the AI to not abandon sieges very easily.

Also I know player-AI objectives can be moddable, but I hope that someday it'll be possible to make an AI objective dedicated to sieging, so that when you assign armies to such objective they focus solely on sieging (unless endangered).

I will be making extensive changes to unit AI, the issue you describe is known and being worked on.

It should be possible to make such a siege only objective already today (set weight on cities with enemy armies to a large value such as 1 million), but I haven't tried doing it with the triggers available.
 
I will be making extensive changes to unit AI, the issue you describe is known and being worked on.

It should be possible to make such a siege only objective already today (set weight to a large value such as 1 million), but I haven't tried doing it with the triggers available.

Great! Will try that.
 
Do you plan on implementing machine learning in a future title? I imagine it would be easier to use it in a small feature/function as opposed to a large-scale implementation like in AlphaStar.
 
Hey since you're reading this:

The BIGGEST PROBLEM in Imperator right now is the fact the AI CONSTANTLY gives up sieges to go attack absolutely useless stacks. The reason you see AI struggling so hard (even Rome which is massively overpowered) to kill tribes is it will CONSTANTLY give up sieges to go attack a 0 strength tribal stack that just spawned.
 
Hey since you're reading this:

The BIGGEST PROBLEM in Imperator right now is the fact the AI CONSTANTLY gives up sieges to go attack absolutely useless stacks. The reason you see AI struggling so hard (even Rome which is massively overpowered) to kill tribes is it will CONSTANTLY give up sieges to go attack a 0 strength tribal stack that just spawned.

Known issue, stuff changing in game close before release messing AI up, e.g. 0 men retinue respawns are fairly new. Will probably be better on 1.1.
 
Do you plan on implementing machine learning in a future title? I imagine it would be easier to use it in a small feature/function as opposed to a large-scale implementation like in AlphaStar.

There is limited ML in Imperator 1.0 already (adaptive combat prediction) but that is drowned out by AI suiciding for other reasons. I should stress ML is basically a fancy term which doesn't automatically improve anything and there are quite few small cases where the implementation overhead is worth it in this application. Even if you're doing the most basic ML possible such as linear regression, you will have to collect relevantly formatted data samples which adds a bunch of dev time, with the risk of being useless.
 
Last edited:
Some of them are attempted fixed in 1.0.1 others secretpatch. I say attempted because it's always difficult to know if fixed in if a particular situation a player will encounter.
1.0.1 is that the day 0 patch?
 
Thanks for this info :)

Does this mean that declaring war is not part of the AI action system? How does the AI decide when to declare on u as a player?

Just abit concerned that the AI will not declare war on you as the player, in a lot of the play throughout I’ve seen I don’t think I’ve seen such a scenario :)
 
Thanks for this info :)

Does this mean that declaring war is not part of the AI action system? How does the AI decide when to declare on u as a player?

Just abit concerned that the AI will not declare war on you as the player, in a lot of the play throughout I’ve seen I don’t think I’ve seen such a scenario :)

The logic is the same for humans as AI targets. AI is a lot more peaceful now than it used to be for various tangential reasons, including manpower being perpetually low on current version, but also since the alliance system was nerfed to dust a while back in favor of guarantees AI nations have a harder time achieving superiority through diplomacy. AI should probably start communicating and coordinate their attacks against dangerous targets despite not being allied because currently there is no in-game mechanic that lets them coordinate aggression.
 
Last edited: