Using Modern A.I. Paradigms To Improve Paradox A.I.

[DaveyDave]

A few years ago I posted to this forum about using a dynamic database of human activity in HOI3 to create very good A.I. for the Clausewitz engine. That post was largely ignored. Now, A.I. paradigms such as those hosted by DeepMind and AlphaZero are doing just that with games like Go, and Chess. Some of them are open source.

The principle is extensible into any type of gaming environment. It is not necessary to create a complex and long code base in which the AI determines "best moves" based on certain criteria, values, and predictives. Instead, each and every game which is played by a human is incorporated into the database, and measured according to its relative success. This move-set is then used recursively until a better one is discovered by another human playing the game somewhere else and incorporated into the database. Additionally, move sets can be generated and compared internally by the program with other move sets, with the more effective ones replacing the less effective over time. An even more sophisticated approach also allows the AI engine to add moves to the move set (based on simple criteria), and then "compared" (i.e. played one AI move set vs another or vs a human move set already flagged as effective), with the same process of elimination then applied.

This approach is essentially like evolution. The database is constantly populated with move sets from all players who allow their data to be parsed up to the AI engine, where the comparative and iterative process is then applied by the engine. Players could also permit the AI against which they play in the game to be the traditional one (which has known weaknesses and flaws) or against the latest iterated database of move sets from the AI engine.

It might be of some interest in the community if those who are capable of creating an open source AI for HOI based on this AI paradigm did so - but this would of course require capability for data exchange between the game and the AI.

This is essentially the future of gaming AI, whether anyone at Paradox or in the forum realizes it. Better to be ahead of the curve.

https://www.newyorker.com/science/e...e=twitter&utm_social-type=owned&utm_brand=tny

 

[Michi2]

1. One thing about your source: It explicitly speaks of two-player games and paradox games are no two-player games.

2. As far as I know (and I am no expert by any stretch of the imagination) all those AIs were used in games like chess. Chess has a limited number of potential possibilities at any point. I am not sure, whether results from chess can be used with regards to paradox games.

 

[SeekTruthFromFx]

Instead, each and every game which is played by a human is incorporated into the database, and measured according to its relative success. This move-set is then used recursively until a better one is discovered by another human playing the game somewhere else and incorporated into the database.

Could anyone please define "better" in the context of HoI IV? It's far from clear to me that there is a machine-readable definition of success.

 

[PeterCorless]

I agree. What would be required is some sort of logfile of political decisions, production and combat info from the players' games (both player choices as well as AI nations), which would then be batch uploaded to a central repository for analytics or real-time streaming from players. These batched or streaming logs could then be parsed into different national ML repositories.

What is common to all Germany run-throughs? What decisions lead most often to France's survival, and which lead to its most immediate defeat?

You could also segregate it by combat types: what are the air combats we see, and how do they resolve? What are the land combats and how do they resolve? Sea battles too.


However, the expertise for this sort of analytics doesn't come for free, and the systems to store and process it are all also non-trivial. How Paradox might afford it, and what sort of benefits it would give back to the game and the player base is as-yet unproven.

Until you could build a business case around it (sold as some sort of DLC, crowdsource-funded, etc., and how does it pay for itself then make a profit), it will remain theoretical.

 

[Gargtholomew]

the reality of the matter is until deep/machine learning is almost perfected AI will always consist of varying levels of complex IF THEN statments

 

[SeekTruthFromFx]

I agree. What would be required is some sort of logfile of political decisions, production and combat info from the players' games (both player choices as well as AI nations), which would then be batch uploaded to a central repository for analytics or real-time streaming from players. These batched or streaming logs could then be parsed into different national ML repositories.

What is common to all Germany run-throughs? What decisions lead most often to France's survival, and which lead to its most immediate defeat?

You could also segregate it by combat types: what are the air combats we see, and how do they resolve? What are the land combats and how do they resolve? Sea battles too.


However, the expertise for this sort of analytics doesn't come for free, and the systems to store and process it are all also non-trivial. How Paradox might afford it, and what sort of benefits it would give back to the game and the player base is as-yet unproven.

Until you could build a business case around it (sold as some sort of DLC, crowdsource-funded, etc., and how does it pay for itself then make a profit), it will remain theoretical.

This already happens. If the player has given permission, all recent PDX games feed back telemetry to Stockholm covering exactly these items. That's why, for example, the WTT DLC featured German alt-history NFs and China NF trees: the data showed that alt-history was far more popular than PDX imagined and that China was a very popular country to play.

 

[DukofDeth]

What if the games we play - whether multiplayer or single-player - were catalogued by way of the internet? Give the initial build of this AI - should it get built - some time to learn, and then release it? Data would be collected locally - on one's hard drive - and uploaded at a game's completion, both to keep the data separate from unfinished games, and also to not annoy players by lagging every time the game tried to update the database. Once released, the AI would stick to the same plan, so within a game the AI would only evolve as well as it could based on the player it was up against, but once data was sent to the database, it could become much stronger. Of course, then the AI would be coming up with the same gamey solutions players do - unless it finds a way to combat that method.

 

[Daddl]

Go and Chess are less complex than HoI by a ridiculously large factor. Same goes for Starcraft and Dota or whichever games someone already trained neural networks for.

Even if you ignore production and politics and just go for pure battlefield manouvering with given armies, greatly reducing the complexity, I'm quite sure it would not work out. Too many divisions, too many provinces, too many players, too many inputs. Too many combinations for a pattern to evolve.

 

[bwhite9]

There are a few major issues your idea.

1) The biggest one is that the teams that made DeepMind and AlphaZero probably had more people with PHDs then the HOI4 team has devs. I am not 100% sure but its at least pretty close.

2) Chess and Go are much simpler games. They have uniform grids and a pretty finite number of pieces.

3) As some other people have pointed out HOI4 isn't a 2 player game and that is a pretty big constraint that AI developers are working with.

4) Paradox doesn't want a perfect AI that is the best player in the world. They want an AI that is fun to play against and one that can do a bit of role play.

IIRC some of the devs have said they have poked around with some of this stuff and they just don't have anything that really helps them make the AI at this time. So at this point, it just isn't that feasible Modern A.I. Paradigms to work on the AI at this time and likely won't ever be possible to fully do the kind of AI that paradox wants with it.

 

[DaveyDave]

There are a few major issues your idea.

1) The biggest one is that the teams that made DeepMind and AlphaZero probably had more people with PHDs then the HOI4 team has devs. I am not 100% sure but its at least pretty close.

snip

You may have missed the part where I mentioned some of these systems are open source. It isn't even necessary for Paradox, or any company, to hire a bunch of PhDs to construct a modern AI engine. All Paradox needs to do is to make it possible for the open source community to build one. This would, ofc, involve some internal changes to the game itself to make human generated move sets and results available, and an API to allow communication between the open source AI and the client.

Your other points have elements of self-limiting logic. My OP never said that the move-sets have to be from two-player games, and they need not be. Move-sets can be generated from purely AI-played games, one-player, or multi-player games. Ideally, the engine would evaluate all move sets for optimal results, and continually experiment with itself for additional optimization. What "Paradox doesn't want" is to go bankrupt, or be left behind the curve when it comes to AI.

Chess and Go are much more complex than Tic-Tac-Toe, and for a time there were people who said that AI for the former two could never match human intuition. They were wrong too.

 

[Meglok]

You may have missed the part where I mentioned some of these systems are open source.

This is where your suggestions crashes. It is very, highly, and extremely unlikely Paradox will open source their engine or code. Beyond that, NDAs reportedly come into play.

I also concur that the ai code for a large number of variable options involved in each move of HOI4 is far beyond anything required for a chess ai. Right now there are a lot of stacked and sequenced if/then statements and ratio RNG options.

 

[Daelyn75]

This is essentially the future of gaming AI, whether anyone at Paradox or in the forum realizes it. Better to be ahead of the curve.

Interesting. I'm always interested to hear where the level of AI is in our technological advancement. I wish there was an easy solution to the AI in strategy games, and with HOI 4 I think with production, the complication of making a competent AI is made much worse.

Maybe in the near future there will be a solution to getting a decent AI up. Maybe even a really good AI could challenge a team of players to see which faction will win, and it might not always be the humans. That I'd love to watch.

 

[bwhite9]

You may have missed the part where I mentioned some of these systems are open source.

It doesn't seem like you understand what open source means. All it means is that paradox can copy and use the code that for there own purposes. And that code is nowhere near ready to deal with HOI4. So then it would be up to paradox to make it work for them and that would take a large team to make it do anything useful. Making what you are suggesting is more advanced then what google as for far made. So how is Paradox supposed to make something far more advanced without the same kind of resources and personnel that Google is using?

All Paradox needs to do is to make it possible for the open source community to build one. This would, ofc, involve some internal changes to the game itself to make human generated move sets and results available, and an API to allow communication between the open source AI and the client.

I don't really think you understand that you are suggesting. What you are suggesting is a far larger and more complex then what Google as done and they have infinite resources and a team large team of not just PhDs but general programmers behind there project. AI isn't a magic wand you can just wave and it fixes problems it takes a lot of work and time and at this point it Paradox doesn't have the resources or time to make a project like that viable. It makes much more sense to keep making the game and not try something that the world leader in this field hasn't even pulled off yet.

Ideally, the engine would evaluate all move sets for optimal results

This here proves that you don't know what you are talking about. This isn't how these modern AI work. They don't evaluate all possible options then pick the one that is best. That's not AI that's a decision tree.

Also, you keep on talking about "move sets" and it just makes it seem like you haven't ever played HOI4 or any paradox game or even looked at what they look like. What would these move sets even be? Its pretty simple to see what they would be in chess or go but in HOI4? It just makes it seem like you don't understand how these AI work or how HOI4 works.

What "Paradox doesn't want" is to go bankrupt, or be left behind the curve when it comes to AI.

I think you over assume the effect of AI of there games on there business. There are much more important factors that to the success of paradox. Adding new features and updating the game are much more important then good the AI is.

Chess and Go are much more complex than Tic-Tac-Toe, and for a time there were people who said that AI for the former two could never match human intuition. They were wrong too.

I have no doubt that what you are suggesting will eventually be used to make AI for games like HOI4, but that doesn't mean it is ready to happen today. HOI4 will likely have ceased in development by the time its viable for Paradox to do this kind of thing. My arguments have nothing to do with what the future will hold but until there is a monumental development in AI programming they are kind of just stuck doing what they are doing.

I do think it's important to point out that we don't disagree about whether or not this will be useful. It will be and it will be important in the future. Just at this time it doesn't make sense for them to be investing more time then they already do into this kind of thing. It's just not far along enough to be used by them at this time.

The devs have talked about this publicly and they do pay attention to changes in the AI field and what they have said is that the field just isn't far enough allong to be of any use yet.

 

[DaveyDave]

Also, you keep on talking about "move sets" and it just makes it seem like you haven't ever played HOI4 or any paradox game or even looked at what they look like.

I bought HOI1 on release day, 2001 or 2002 I don't remember, and helped develop the first mod for it, then known as "Bolted HOI." Making a statement like this, which is a personal attack, and, as well, in complete ignorance of information available to you about me right underneath my avatar, indicates a number of things - chief of which is contrarianism without investigation or even argument.

Good luck.

 

[DaveyDave]

This is where your suggestions crashes. It is very, highly, and extremely unlikely Paradox will open source their engine or code. Beyond that, NDAs reportedly come into play.

I also concur that the ai code for a large number of variable options involved in each move of HOI4 is far beyond anything required for a chess ai. Right now there are a lot of stacked and sequenced if/then statements and ratio RNG options.

If you believe this - which I summarize as belief that excellent AI, reliable, logical, human beating AI - isn't possible for a host of reasons, all I can say is that this simply isn't in accord with the literature on the subject. Big changes are coming, and the groundwork for incorporating that into gaming is *already being laid*.

Here's an article from 2015:

The last five years have seen a major revolution in artificial intelligence technology, via the use of so-called ‘deep learning’ techniques, which allow the efficient creation of much larger neural networks, which can reason abstractly about complex data. This kind of processing can be made much more efficient with the use of neuromorphic chips, which implement neural networks directly in hardware. Potentially, videogames could farm AI tasks out to cloud servers using dedicated neuromorphic chips. This would prevent game developers from having to choose between clever AI and good graphics, since the AI could be done cloud-side, taking the burden off of individual computers. This would also allow the AI algorithms in use to share insights and knowledge between users, making everybody’s experience smarter.


Right now, most videogame AI works using decision trees — simple flow charts of behaviors, structured by hand, which take a few variables into account and use them to select a course of action. This is very simple and computationally cheap, but not very robust – when you get to a situation that the flow chart creator didn’t plan for, you often get nonsensical behavior. In the future, it’ll be possible for developers to ‘train’ more sophisticated neural networks to respond to much more data in much subtler and more flexible ways.

Just this year, AI beat humans at DOTA2, using these emerging techniques. To put them in a more simple framework, think of it this way: You start with a basic move set (which is a series of actions taken simultaneously or in sequence to perform actions in the game; the movement of troops, the upgrade of units, creation and movement of air fleets, naval fleets, combat, retreats, and so on - the Clausewitz AI already does this, but not using a new AI paradigm, and always plays out the same way given the same inputs). This move-set is then put into action against another move set, one generated by humans or by AI, even by the existing Clausewitz engine. One move set will be more effective than the other. The results are evaluated and stored, and the more efficient move set (for that "country" in HOI4, fx) becomes the most *effective* given *certain criteria which the AI stores as triggering that move set.* This process is repeated 24 hours a day, for every "nation" generating in the space of one day perhaps millions of move sets for each nation, and hundreds or thousands of general criteria which trigger the set.

In sum, it is evolution -- self learning. The same techniques of evolutionary AI are already being used in other areas like market trading. The concept has already been proved as effective in a number of areas. In the next few years, it will expand exponentially. The first game publisher that makes available, optionally, to play against this kind of AI, is going to have an enormous market advantage.

Take care.

 

[Robosoldier1]

ya the big flaw with the Ai is the innate restriction based on the play and expectation by the Paradox development team instead of the varying player base. It really does limit the game and challenge so I hope that maybe your suggestion is a viable solution. All I know is that the parameters and general behaviors incorporated by Paradox for the nations is just becoming more and more apparent and it does get tiresome to see.

 

[Cavalry]

this is very interesting and it is a good business to start a start up if you can do this thing!

For now, just use Expert AI, it manually give AI the current meta templates to use, and more scripts for more situations.

 

[Chevaresqye]

Go and Chess are less complex than HoI by a ridiculously large factor. Same goes for Starcraft and Dota or whichever games someone already trained neural networks for.

Even if you ignore production and politics and just go for pure battlefield manouvering with given armies, greatly reducing the complexity, I'm quite sure it would not work out. Too many divisions, too many provinces, too many players, too many inputs. Too many combinations for a pattern to evolve.

Both Chess and Dota are more complex for AI than any PDX game. The production and politic is the easiest thing for AI to optimize since everything are extremely linear in PDX game. There are simpy two reason why PDX cannot have good AI. First is performance issue, it you make AI scan the whole map to make holistic decision, it may take an hours too finish a single game loop. Secondly, an optimize AI would be no fun to play against at all, as an AI gridlock is almost guaranteed.

In Chess and Dota, AI must compete with human in every step to gain any kind of advantage, and one small mistake by AI can be exploit by human player to snowball the game into victory. In PDX game, it is so clear cut between good and bad outcome, and a lot of complexity is superficial. Also, the game is much more forgiving, as even if one AI making a mistake, there still 199 other AI who can punish human mistake at anytime.

 

[Alex_brunius]

Both Chess and Dota are more complex for AI than any PDX game. The production and politic is the easiest thing for AI to optimize since everything are extremely linear in PDX game. There are simpy two reason why PDX cannot have good AI. First is performance issue, it you make AI scan the whole map to make holistic decision, it may take an hours too finish a single game loop. Secondly, an optimize AI would be no fun to play against at all, as an AI gridlock is almost guaranteed.

Maybe from an Economy or Law decision making process the AI in HoI4 could be relatively easy to improve ( although this is already done by several good AI mods through scripting alone, so no changes at all to the HoI4 exe is even needed here )...

But how does the AI evaluate or decide which of the 15000 provinces on the game board it should send one of it's 200+ pieces to? Or which of the 20+ stats that are important to focus on when designing divisions to fight an unknown enemy? Or plan a long term buildup and invasion strategy for 24*365 = 8760 "turns" in the future ( each "turn" being 1 hour in HoI4 from the perspective of the AI )?

Each of these operations are so much more complex than the decision making process needed by current chess or dota AIs that even breaking down the terms of success and problem to smaller pieces is a very complex process.

 

[Alspego]

An optimised AI would form a mega faction. If everyone is a member of the same faction, everyone is the winning team!