Using Modern A.I. Paradigms To Improve Paradox A.I.

[Chaingun]

I'm no AI scientist, but if this stuff was efficient and useful for game design, someone would be using it already.

Indeed.

Not going to spend much time on rehashing since this kind of thread pops up every few months, but to summarize:

• it's not efficient to play hundreds of thousands of campaigns of a PDS game for training (this would require an enormous server farm or equivalent cloud compute costs).
• Pure black box approaches (just sending in e.g. video frames) and entirely too inefficient, so inputs (features) have to be carefully chosen by AI dev. So do outputs. A database approach implies all of the game implementation is exposed to AI, which is massively constraining for gameplay programmers.
• The game changes all the time during development except the very end, breaking AI which is working several times. While an ML algorithm is trivially retrained on new gameplay, as the inputs have to be carefully chosen (if nothing else because computing them for all agents is costly).
• Defining the objective function is impossible (no, it's not only about winning), in essence it's all about making as few players pissed off as possible that have wildly differing individual expectations. Players assume the AI is bugged if they do not understand its reasoning.
• On a technical note, reinforcement learning which is far more tricky to get robustly working than supervised learning is necessary.

We are at least a decade away from commercial solutions that are viable in terms of actually solving our AI problems within required constraints. In general, choosing any algorithm that isn't super straightforward will likely make you a sad AI developer (still we try).

 

[Chaingun]

Yes, if I had my entire life (or rather, as long as I desired before calling it done) to write the AI for a Paradox game, while the game didn't change, I'm pretty sure it would turn out great.

Production and Logistics Modules could learn when to change production lines up and how best to train troops. Do most players skip producing infantry weapons 2? Do they fully train their troops before deploying them or push them out green and have them exercise?​

And then it would still get owned by experienced players because you taught it to follow a fixed strategy and because it is also "independent" from your other modules, resulting in decision conflicts.

 

[Chaingun]

Well Europa isn't changing The larger royalty cut from your individual project would be worth the risk.

News to me, then again I don't follow EU too closely. I thought there was a DLC recently?

and there is plenty of backend AI coding that could be improved.

Agreed. Perfection won't happen.

The larger royalty cut from your individual project would be worth the risk.

Unsure where this is going, what royalty and for whom?

 

[Chaingun]

Since most of AI reasoning never document and public to players by PDX, so that point does not really matters right?

Why are there additive opinions, aggressive expansion, vital/interest of territory, and a host of other systems allowing the player to understand (and to game) the AI then?

AI acting to very simple mechanical rules is more easily understandable by player, even if those aren't very competent. Another example from EU4: Players request that AI should only take sieges in preference to fighting enemies armies, or the reverse, even those both are very exploitable strategies. But they're very intuitive (and easy to cooperate with, if you happen to be the AI's ally in a war.)

"Documenting everything" on AI is practically impossible without extending development time or limiting the scope of the AI. Nor is it even desirable as it then would be little to figure out and it is just another aspect of incomplete information in the same way as it can be an improvement to have Fog of War. Still, there's usually a conflict in making black box AI intuitive versus making it competent.

 

[Chaingun]

If gaming the AI is part of the game, then there is no reason to obstruct such information.

Entirely subjective. I don't like it. Some people on my team do. I don't like always winning.

FoW is a bad comparison, hiding AI benhavior is more like obscuring the formula for trade steering or hiding random chance.

It's really not for adversaries. My enemy rarely likes to inform me what he's going to do.

If you design the AI event without any document for the expecting behaviors and the line of reasoning,then how people of other part of the team know how their tweaks on the game mechanics gonna affect the AI?

That's an interesting observation and one discussion we had internally recently where QA desired more clear internal documentation. Time spent documenting is not time spent coding though.

Now if you decided to stick with the partially obscure AI, then that part can be black box which out any change in player's understanding of your AI.

You will need to reformulate this for me to understand what you're saying.

 

[Chaingun]

That is exactly how it would NOT work. Yes a base strategy has to be installed at first, but as the AI learned from all of the player playthroughs it would learn and adapt that strategy. As more people beat the AI it will learn from the playthroughs what the players are doing and adapt its strategy. The only people that would consistently beat the AI are ones that are always adapting their strategy to what the AI is doing. That would then teach the AI new strategies and it would adapt. that is the point of a learning AI.

Also the "independent" part has nothing to do with decision making. Modules feed each other tasks and spit out results to the other modules. The internal mechanics of the modules are independent from each other to allow them to be tweaked, updated, or modded without changing the entire AI.

Think of this like a real life situation. The president says we are invading Canada. The admirals and generals lay out the battle plans. The logistics people move troops and material around. The arms manufacturers build up more weapons to maintain or increase stockpiles. If a general gets taken out by a Canadian assassin he will be replaced. The new generals strategy might be a little different but he still has the same ultimate goal of invading Canada.

First, apologies for misconstruing what you said. However, you're describing a very best case. Please implement it in a non-trivial test game and see if it works as well in practice.

By the way, an (albeit trivial) game that can be a humbling experience when it comes to illustrating how far you can go in messing with an opponent that shows any sign of predictable behavior: http://www.rpscontest.com/ Maybe you can learn a reactive enough policy to avoid this problem with offline supervised learning, I have my doubts. And after that, the more general learning algorithm you use, the longer time it takes and the higher variance in the results. Reinforcement learning and very general evolutionary algorithms ARE tricky to get working and are far from reliable safe workhorses at present we can commit to. Supervised learning is, but it is not suitable due to static policies.

Unrelated to above points but about decision conflicts: a lot of real life effort goes into dealing with situations where code somewhere wants to do something that code somewhere else wants to do differently. This severely restricts how independent systems can be. What you describe sounds great and everyone has this layered approach at some point. We are in fact already implementing independent subsystems/layered approaches where possible (maybe I already mentioned it's easier, right). AI code in the end becomes pretty terribly ugly for a real game due to all obscure dependencies that need to be accounted for. Saying ahead of time you will have well defined modules with well defined inputs/outputs is cool, now the question is what what extent you will be able to keep that once the "idiot AI" bugs start stacking up.

We're in the business of writing games, not advancing towards the singularity, and priorities/costs are reflected as such in the organization. (Ok I am a consultant nowadays and can do fun things on the side, but w/e.)