Using Modern A.I. Paradigms To Improve Paradox A.I.

[xsmilingbanditx]

Just to put that into perspective again: DOTA, Starcraft, Chess, Go or whatever are by far more trivial that HoI4 in regards to AI.

Fit example, in DOTA or SC the AI know the exact number of Hitpoints any unit has with the exact number of damage it does and the exact timing it has with attacks. With this ruleset, it is absolutely trivial to calculate a path through a tree of possible decisions for the optimal outcome resulting in perfect last-hitting or "microing" the optimum number of units to oneshot another.

These rulesets do not exist in HoI4 because Dijkstra/Prim kind of solution (take the optimal next step every time to reach optimal outcome) won't really work in a deterministic way of thinking.

In a sidenote, the GO algorithms got so good because they played thousands of games against themselfes. If you'd remove a Column from the Board, the "AI" would need to learn all over again. Everything. Because the ruleset changed.

Current AI is just a glorified decision-tree with statistics mixed in based on huge amount of data.
In the end, it still comes down to things like "If lastsuccess > 0.5 then try again and modify succesrate when finished".

TL;DR: It's about complexity of rulesets und available data including processingpower of said data. Mark my words: Cars will drive themselfes better than humans way before you get a competitive, non-cheating AI in games like HoI4. That's because the ruleset and availability/mass of data is trivial in comparison.

 

[PrinceOfMacedon]

You may have missed the part where I mentioned some of these systems are open source. It isn't even necessary for Paradox, or any company, to hire a bunch of PhDs to construct a modern AI engine. All Paradox needs to do is to make it possible for the open source community to build one. This would, ofc, involve some internal changes to the game itself to make human generated move sets and results available, and an API to allow communication between the open source AI and the client.

Open source means you can copy the code and rebuild another AI to play chess. If you want to build an AI that play HOI4, then you have to copy the code and then fix 90% of them to fit HOI4 environment, and then BUILD the actual AI.

And by rebuilding, I mean, you have to train the AI to play hoi4. The code is just how you can train the AI, you need to find a teacher to train the AI. By teacher, I mean real people play real match so that we have the data to train the AI, that why data have valuable. It just not simple, even when you have 100 millions of dollar, you need real player play high quality match.

 

[aokiryu]

A few years ago I posted to this forum about using a dynamic database of human activity in HOI3 to create very good A.I. for the Clausewitz engine. That post was largely ignored. Now, A.I. paradigms such as those hosted by DeepMind and AlphaZero are doing just that with games like Go, and Chess. Some of them are open source.

The principle is extensible into any type of gaming environment. It is not necessary to create a complex and long code base in which the AI determines "best moves" based on certain criteria, values, and predictives. Instead, each and every game which is played by a human is incorporated into the database, and measured according to its relative success. This move-set is then used recursively until a better one is discovered by another human playing the game somewhere else and incorporated into the database. Additionally, move sets can be generated and compared internally by the program with other move sets, with the more effective ones replacing the less effective over time. An even more sophisticated approach also allows the AI engine to add moves to the move set (based on simple criteria), and then "compared" (i.e. played one AI move set vs another or vs a human move set already flagged as effective), with the same process of elimination then applied.

This approach is essentially like evolution. The database is constantly populated with move sets from all players who allow their data to be parsed up to the AI engine, where the comparative and iterative process is then applied by the engine. Players could also permit the AI against which they play in the game to be the traditional one (which has known weaknesses and flaws) or against the latest iterated database of move sets from the AI engine.

It might be of some interest in the community if those who are capable of creating an open source AI for HOI based on this AI paradigm did so - but this would of course require capability for data exchange between the game and the AI.

This is essentially the future of gaming AI, whether anyone at Paradox or in the forum realizes it. Better to be ahead of the curve.

https://www.newyorker.com/science/e...e=twitter&utm_social-type=owned&utm_brand=tny

I liked your suggestion. But it is the paradox development team who have the knowledge to judge if it is possible to do something similar to this in HOI4 or not.

 

[xustc]

It's much easier to implement an advanced AI on a regular grid with limited size, or using turn based rules. Weiqi (19x19=361) and chess (8x8=64) use the rectangle grid, which is the simplest structure. So far I haven't seen applications of deep learning AI on hex grid; it would be more difficult than rectangle grid, but still feasible.

However, with the irregular grid, the sheer size of the map, as well as the semi-rts nature of HOI, no current AI models can be used directly. And I won't expect Paradox to have enough resources to develop a new specialized AI engine for its games.

 

[Giob]

Both Chess and Dota are more complex for AI than any PDX game. The production and politic is the easiest thing for AI to optimize since everything are extremely linear in PDX game. There are simpy two reason why PDX cannot have good AI. First is performance issue, it you make AI scan the whole map to make holistic decision, it may take an hours too finish a single game loop. Secondly, an optimize AI would be no fun to play against at all, as an AI gridlock is almost guaranteed.

In Chess and Dota, AI must compete with human in every step to gain any kind of advantage, and one small mistake by AI can be exploit by human player to snowball the game into victory. In PDX game, it is so clear cut between good and bad outcome, and a lot of complexity is superficial. Also, the game is much more forgiving, as even if one AI making a mistake, there still 199 other AI who can punish human mistake at anytime.

Chess and DOTA have nowhere near the level of complexity of a PDX game. Chess has a rigid and extremely diminutive structure, DOTA an extremely small number of variables. Production alone is more complex than both the two games combined and a Neural Network would get months to be traained to obtain acceptable results and God only knows how much to actually produce output ingame. Now multiply this for the other aspects of country running (Politics, Diplomacy, etc...) and the multiply again by the numberof countries. Once you have done this, erase any mod which adds anything new to the game since it won't be compatible with the new AI anymore.

 

[Alspego]

Both Chess and Dota are more complex for AI than any PDX game.

I would distinguish between complexity and depth. Complexity would be related to the number of possibilities while depth is related to the number of meaningful (or non-trivial) choices. I would say, that chess has more depth than any PDX game which is shown by the fact that there are many people playing and studying the game for decades without fully master it. But any PDX game is much more complex.
Humans are really good at pattern recognition and therefore really good at extracting essential features. Because of that, we can deal with very complex problems. On the other side, we are quite bad in dealing with depth since it takes a lot of time to compare multiple different options and handling large "data sets" necessary for an accurate comparison. A computer has fewer problems with that, but large problems in dealing with high complexity.

 

[Giob]

I would distinguish between complexity and depth. Complexity would be related to the number of possibilities while depth is related to the number of meaningful (or non-trivial) choices. I would say, that chess has more depth than any PDX game which is shown by the fact that there are many people playing and studying the game for decades without fully master it. But any PDX game is much more complex.

No, PDX games have both an enormously bigger "complexity" and "depth" (going by your definition). Just think at the amount of moves on the tactical level you may have in HoI4, or the myriad of different choices with possibly far-reaching consequences there are in other games (like creating a vassal state, activating a Golden Age, aiming for a specific mission et cetera onEU4).

Humans are really good at pattern recognition and therefore really good at extracting essential features. Because of that, we can deal with very complex problems. On the other side, we are quite bad in dealing with depth since it takes a lot of time to compare multiple different options and handling large "data sets" necessary for an accurate comparison. A computer has fewer problems with that, but large problems in dealing with high complexity.

That's not the issue. Computers can get EXTREMELY good at both deep and complex tasks. The problem is that they are rigid, especially when using an NN like in OP's examples. An AI can become unbeatable in chess, but change the rules so that the pawns can move backwards and it is as good as dead. Go further and change the amount of rows and columns or even add a new dimension to the board and it will be incapable of solving the problem. Considering the billions of different scenarions that may arise in PDX games (especially in something like Stellaris where literally no game starts the same as another) you can proably see why these systems, as they are now, can't be made to work with something as big as a GSG.

 

[D3m0]

Something nobody seems to have addressed yet is that a functioning neural network AI for HoI would not just be very good, but also very gamey.
From what I've read deep learning AI tends to abuse game systems in directions that don't ever occur to any human players, the result would be an AI that is very good at playing the game but in ways that would feel far removed from a WW2 experience. Imagine every country was played by some YouTuber doing WCs with Luxembourg through extreme metagaming.

 

[Daelyn75]

Something nobody seems to have addressed yet is that a functioning neural network AI for HoI would not just be very good, but also very gamey.
From what I've read deep learning AI tends to abuse game systems in directions that don't ever occur to any human players, the result would be an AI that is very good at playing the game but in ways that would feel far removed from a WW2 experience. Imagine every country was played by some YouTuber doing WCs with Luxembourg through extreme metagaming.

I can easily see this happening. The majority of us want a RolePlaying AI, as in playing how those nations would have fought back then with the tools given to them within the game. Otherwise it's all CAS and Fighter spam, and heaven forbid if 700 submarines sink everything in the ocean, well the AI would go that route too.

We also want to see the AI make mistakes. If it just played by any means to beat you, then it would start ticking off players and we'd have the opposite problem. We need something in the middle.

 

[Chrysostom15]

This time, you are obviously not being ignored .

 

[DGuller]

And by rebuilding, I mean, you have to train the AI to play hoi4. The code is just how you can train the AI, you need to find a teacher to train the AI. By teacher, I mean real people play real match so that we have the data to train the AI, that why data have valuable. It just not simple, even when you have 100 millions of dollar, you need real player play high quality match.

I don't think that's true. Some, if not all, of the modern AI systems learn by playing against themselves.

 

[mccarty.geoff]

Programming is just smoke and mirrors. Artificial intelligence is a misnomer. The GUI is nothing like what is actually running in the program modules. In order to keep a PDS spreadsheet sim stable the gaming interfaces have to be autonomous of each other. Trade, diplomacy, war planning are all disconnected until the scripting medium posts stacks between them. Programming an """AI""" that understands non-machine language scripting triggers and commands would be the first step to SKYNET. PDS probably does have sophisticated debugging macros that test the individual modules but, there is no inter-relation of the sub-processes without a sentient authors influence. When things are coded that actually work it is like everything we conscious automatons do; a happy accident.

 

[xsmilingbanditx]

I don't think that's true. Some, if not all, of the modern AI systems learn by playing against themselves.

Yes they do, see my post able if you like one of the problems is, there is no well definied winning condition for each and every nation. In games based in "simple" trees like tic-tac-toe, chess or go this is easy. But certainly, the winning condition for Luxembourg would be different from germany. A well designed AI might come to the conclusion that waging war ja totally stupid and not worth it. Because it depends on the goals you set - because there is no easy solution for the lack of Motivation and "drive".
All in all, you would need several "AI" or...well whatever you would call it because there is no single valid definition for what an AI is supposed to be. One AI for each subsystem plus one observant AI to make the overall decisions (e.g. trade resource or build faster, weightened by strategic demands).

 

[BaddoSpirito]

This is practically impossible with current technology. For one thing, noone has successfully used an AI based on reinforcement learning for a game like HOI4 to my knowledge. The architectures for chess and Go will not apply to hoi4 due to multiple countries and the fact that it is unclear how to measure success and may be different for each country. I think you would probably have to train a different network for different countries. Since it is unclear how to apply the concept to the problem, this requires significant AI and mathematics expertise. Paradox cannot even fix simple bugs for months and you expect them to successfully implement a revolutionary AI? The development of such an AI would take years even for very experienced AI research teams and I highly doubt it is computationally feasible with current hardware. Even on Go, you need to dedicate your GPU to AI for it to make good moves in a reasonable time frame. In Go and chess, players take turns and make only one move per turn. In HOI4, each "turn" is one hour in game time and players "make a move" at the same time. And "make a move" entails multiple actions unlike Go and chess. So basically, this requires a revolutionary AI both in terms of theory and architecture AND it also needs to be fast enough to make good decisions for every country for every hour in fractions of a second. Don't expect this to happen in the next 10 years.

What COULD be practically possible is to train a learning-based AI for more high-level strategy planning and leave the details to a traditional simpler program. But I doubt Paradox would think it's worth the investment. They would need to hire AI experts which aren't cheap and most casual players aren't even good enough to beat a stupid AI like the current one. If you had a proper AI even only at a strategic and template creation level, 99.9% of single player people would get destroyed so is it really worth investing in proper AI for the 0.1% of customers?

 

[Giob]

I don't think that's true. Some, if not all, of the modern AI systems learn by playing against themselves.

No. AIs need a working teacher to learn. Even if you train them against themselves (reinforcement learning), you have to have set up a function to mathematically quantify how well the AI is performing (which is something which doesn't exist and is probably unimaginably difficult to create), and then still give them enough positive examples to steer them towards the correct choices. And again, once you've done this, you have to retrain (or even recreate) a new "AI" every time a new patch comes out, because there is going to be new mechanics the AI is not capable of using.

 

[xsmilingbanditx]

No. AIs need a working teacher to learn. Even if you train them against themselves (reinforcement learning), you have to have set up a function to mathematically quantify how well the AI is performing (which is something which doesn't exist and is probably unimaginably difficult to create), and then still give them enough positive examples to steer them towards the correct choices. And again, once you've done this, you have to retrain (or even recreate) a new "AI" every time a new patch comes out, because there is going to be new mechanics the AI is not capable of using.

Depends. "AI" for Go and Chess (the good ones) do this, because they just play thousands of games simultaniously. Which you can never ever do otherwise.
Yet again, this is only possible due to the nature of those games: 1v1 AND a simple enough definition of Win, Draw and Loss (which basically is NOT WIN). So you only have a Maximum of two well defined states as a goal.

 

[wukk]

Also the mentioned AI for Go and Chess are not playing a real time game. While the AI of AlphaGo can make decisions fast, for it to be competitive, according to my googling it should be allowed to used in excess of one minute to decide what it does next - per turn. Now, consider how much more complex of a game HOI is compared to the examples and how much more the gameplay changes in a few seconds (think of how many "turns" of Go would in HOI pass in one game day alone) and you'll need a supercomputer for the game to progress in an hour what it does in a second now with a home computer (or something close to that scale). Not really that feasible.

 

[FrancescoT]

For a commercial product is kinda pointless to talk about the best AI you could code when the limiting factor is the best AI you can RUN in "real-time" on an average commercial gaming platform.. if you don't want to go back to a pure turn based approach where you click next and go to grab a beer, coming back minutes later.

 

[Gort11]

I'm no AI scientist, but if this stuff was efficient and useful for game design, someone would be using it already.

 

[Chaingun]

I'm no AI scientist, but if this stuff was efficient and useful for game design, someone would be using it already.

Indeed.

Not going to spend much time on rehashing since this kind of thread pops up every few months, but to summarize:

• it's not efficient to play hundreds of thousands of campaigns of a PDS game for training (this would require an enormous server farm or equivalent cloud compute costs).
• Pure black box approaches (just sending in e.g. video frames) and entirely too inefficient, so inputs (features) have to be carefully chosen by AI dev. So do outputs. A database approach implies all of the game implementation is exposed to AI, which is massively constraining for gameplay programmers.
• The game changes all the time during development except the very end, breaking AI which is working several times. While an ML algorithm is trivially retrained on new gameplay, as the inputs have to be carefully chosen (if nothing else because computing them for all agents is costly).
• Defining the objective function is impossible (no, it's not only about winning), in essence it's all about making as few players pissed off as possible that have wildly differing individual expectations. Players assume the AI is bugged if they do not understand its reasoning.
• On a technical note, reinforcement learning which is far more tricky to get robustly working than supervised learning is necessary.

We are at least a decade away from commercial solutions that are viable in terms of actually solving our AI problems within required constraints. In general, choosing any algorithm that isn't super straightforward will likely make you a sad AI developer (still we try).