HOI4 AI status

[Joccend]

As awesome as that would be, it simply is not technically feasible for a game like HoI. As a comparison, consider the recent successes of googles deep neural network tech defeating champions at Go.
Go is significantly more complex than chess because of the much larger number of possible moves per turn. While chess AI gained success by calculating future possible board configurations and attempting to steer the game in a beneficial direction through this search tree, Go just has too many possible board configurations to do that (significantly more than the number of atoms in the known universe).
Google essentially used a supercomputer and a ridiculous ammount of training data (aside from letting it play against itself) to accomplish this feat.
While Go has 361 positions HoI 4 has over 13000 provinces. The number of possible states of each one of the 361 positions in Go is 3. The number of possible states of a province in HoI4 is hard to quantify with a discreet number. And apart from provinces there are multiple off map systems and partially off map systems.

On top of that Go is a turn based game where the AI can take time to process each move. HoI4 is pausable real time. Even if you consider each hour a turn, there are still expectations on the AI not taking several seconds, let alone minutes, to process what it is doing (which can be any number of things as part of the same tick.)

The math on this just explodes.

Frankly, as I write this I am amazed at the level we have managed to get to already with the HoI4 AI.

Trust me; when or if the time comes that this is a viable approach I will enjoy looking into it even if I have to use my free time to do it

As a software engineer, I understand completely. In university I had to code an AI for the game battleships that could also learn from past player's tendencies and moves. Comparing the two is like apples to planets, but I'm saying that just from that I can understand the sheer complexity of it.

 

[ChiliDip]

I just want to say that I think its remarkable the level of involvement and response from the Paradox team. I have played many strategy games for years and Paradox continues to demonstrate a community involvement level that is very gratifying. People who complain should realize most developers do not read, much less respond in a meaningful way to their community. That Paradox does is one of the main reasons I am happy to continue to give them my money.

 

[Axe99]

I just want to say that I think its remarkable the level of involvement and response from the Paradox team. I have played many strategy games for years and Paradox continues to demonstrate a community involvement level that is very gratifying. People who complain should realize most developers do not read, much less respond in a meaningful way to their community. That Paradox does is one of the main reasons I am happy to continue to give them my money.

Damn straight, a bit +1 here. They clearly believe in what they are doing, and in supporting their players. Cheers to Steelvolt and Wiz for their response in this thread and elsewhere .

 

[GermanPower]

As awesome as that would be, it simply is not technically feasible for a game like HoI. As a comparison, consider the recent successes of googles deep neural network tech defeating champions at Go.
Go is significantly more complex than chess because of the much larger number of possible moves per turn. While chess AI gained success by calculating future possible board configurations and attempting to steer the game in a beneficial direction through this search tree, Go just has too many possible board configurations to do that (significantly more than the number of atoms in the known universe).
Google essentially used a supercomputer and a ridiculous ammount of training data (aside from letting it play against itself) to accomplish this feat.
While Go has 361 positions HoI 4 has over 13000 provinces. The number of possible states of each one of the 361 positions in Go is 3. The number of possible states of a province in HoI4 is hard to quantify with a discreet number. And apart from provinces there are multiple off map systems and partially off map systems.

On top of that Go is a turn based game where the AI can take time to process each move. HoI4 is pausable real time. Even if you consider each hour a turn, there are still expectations on the AI not taking several seconds, let alone minutes, to process what it is doing (which can be any number of things as part of the same tick.)

The math on this just explodes.

Frankly, as I write this I am amazed at the level we have managed to get to already with the HoI4 AI.

Trust me; when or if the time comes that this is a viable approach I will enjoy looking into it even if I have to use my free time to do it

Informative. Thanks for the reply. That's certainly interesting. IT looks like it's a vast improvement over 3. So basically what you are saying is I need to build and make a extremely cheap quantum computer for my silly solution to work? I do think my other meaning of using players strategies would be wonderful to see. I should have explained that better.. I did below.

Whiz I think you missed my meaning. I said totally different for the fact I meant to mean I understand it wouldn't work at all. My part about learning from humans was the fact that it'd implement player strategies rather then simply preforming the basics of the game(Not the learning for Forza). A example would be tieing up units along a small front while sending mobile units at a weak point in a line to surround them and destroy them. That's what I mean...using player tactics instead of what feels like a broad front. I have no idea what the AI is like in HOI4 so I don't wanna make any judgement..but that certainly didn't exist in 3 or DH. The AI certainly looks like it's a huge improvement being made. Not at all do I mean doing something like Forza. Sorry for the confusion it wasn't worded well.

 

[Sun_Killer]

Informative. Thanks for the reply. That's certainly interesting. IT looks like it's a vast improvement over 3. So basically what you are saying is I need to build and make a extremely cheap quantum computer for my silly solution to work? I do think my other meaning of using players strategies would be wonderful to see. I should have explained that better.. I did below.

Whiz I think you missed my meaning. I said totally different for the fact I meant to mean I understand it wouldn't work at all. My part about learning from humans was the fact that it'd implement player strategies rather then simply preforming the basics of the game(Not the learning for Forza). A example would be tieing up units along a small front while sending mobile units at a weak point in a line to surround them and destroy them. That's what I mean...using player tactics instead of what feels like a broad front. I have no idea what the AI is like in HOI4 so I don't wanna make any judgement..but that certainly didn't exist in 3 or DH. The AI certainly looks like it's a huge improvement being made. Not at all do I mean doing something like Forza. Sorry for the confusion it wasn't worded well.

so what you want to say is, give the ai some standart stratgic moves. Like faints and encirclement plans?

 

[GermanPower]

so what you want to say is, give the ai some standart stratgic moves. Like faints and encirclement plans?

That's what I meant. Learn from the player was conflicting, haha. But that's what I meant. What the player does. Learn from that.

Secondly I did have Ai actually learning idea. What if you mined all the data from players during combat? So when similar situations eruppted you could select successful plans from prior combat in order to enact? Would something like that even been possible? I'm curious.

 

[Jwits]

I hope we can get a dev dairy and maybe a video with ai team to explain the ai developed in hoi4.

 

[ringhloth]

That's what I meant. Learn from the player was conflicting, haha. But that's what I meant. What the player does. Learn from that.

Secondly I did have Ai actually learning idea. What if you mined all the data from players during combat? So when similar situations eruppted you could select successful plans from prior combat in order to enact? Would something like that even been possible? I'm curious.

Getting data is easy. Using it is near impossible. The first week of data might take months to parse. Once you have it parsed, you'd have 98% terrible plans that failed and 2% okay ones. Once you've taken another handful of months weeding out those, you might have a handful of plans that would be useful as Germany or the SU (as those will see disproportionate playtime), and you'd be left with braindead AI for the rest of the world.

 

[EntropyAvatar]

My part about learning from humans was the fact that it'd implement player strategies rather then simply preforming the basics of the game(Not the learning for Forza). A example would be tieing up units along a small front while sending mobile units at a weak point in a line to surround them and destroy them. That's what I mean...using player tactics instead of what feels like a broad front.

I think the problem with this is that the AI just doesn't have "framework of thought" to understand concepts like "tieing up units along a small front". A human player can look at a replay and say "Oh, I see how Player A used those six infantry divisions to pin down the defenders while the 4 armored divisions moved to cut them off - maybe I can use that approach in a similar situation!" The AI would see "at time 11/9/41 player A's unit 726 attacked player B's unit 1259 in province 1823 from province 1817" and so on for 50 or 100 lines. It doesn't see the pattern, it lacks the proper concepts and so it can't generalize from the examples. If the AI is at a level where it can take the raw data from the map and see "pinning attack", and recognize a "similar situation" when it comes across it then you are 99.9% of the way to solving the problem anyway.

Plus, as is often remarked, if you can make an AI that can learn to generalize in a domain as complex as a HoI game, then what are you doing writing game AI? Sell the idea to Google for a bajillion dollars and go change the world.

 

[Sabotage13]

HoI4 AI is already better than HoI3 AI (well, the China AI performs better: that's actually because of a lot of things, mostly supply being a lot more important and harder to maintain), and there's 2 months left for improvements. I'd say we're in a good spot.

Yea, the Japan gameplay showcased looks like it captures pretty well the actual Japanese experience in China (initial overrunning of troops -> weak supply lines - > supply system strained to the limit -> retreat & regroup along major supply lines, leave the hinterland to the Chinese).

 

[Sabotage13]

My greatest AI concern is about naval invasion. The AI of HoI3 was particularly bad at this (I saw once UK landing troops in Netherlands, but without any support nor reinforcement so Germany pushed back this attack easily), and it was a big mess for the game as it means Japan and USA were basically stucked in "sitzkrieg". Why ? I don't know, but I think it's maybe due to the difficulty of the naval transport system. If this problem doesn't exist anymore in HoI4, you'll make me happy.

It does look like they massively simplified naval transportation and naval landing, so there is hope. The first problem (failed follow-up reserves after initial landing) may still persist, though.

 

[panzerzombie]

Hm, then I may come back to my old MP idea, substituting the AI for my opponents with a bunch of Chimpanzees...diplomacy will then heavily involve bananas.

 

[AapoAlas]

So basically what you are saying is I need to build and make a extremely cheap quantum computer for my silly solution to work?

Unfortunately a quantum computer would likely not help solve the problem. I cannot think of a single quantum algorithm that would make the process of teaching an AI like AlphaGo. Well, maybe QFT on the weight matrix... That might actually be a good idea... (Scratch that, it might actually help.)

@Wiz If you happen to find the time and interest, would you be willing to explain, at least in brief, the basic technical ideas behind the HoI4 strategic AI? How much of it is scripting? How much of it is using the Battle Plan system with piece-wise plans that the AI implements to its best judgement? Is there any procedural learning involved? (I presume not as aforementioned the challenges are quite hefty but you never know, you are a genius AI programmer after all.) What sort of "levels of thought" does the strategic AI run on? By that I imagine planning, counter-planning, counter-counter-planning, deceiving and such.

AI is sort of a hobby of mine, kind of.

 

[ruzen]

If I'm not mistaken. Killer Instinct has learing AI aswell. Learning frame traps& combos. Mimicing most player movement,etc..

Edit: Is there really no way for AI to learn small things. At least production order? or most common Teching or generic National Focus?.

 

[AapoAlas]

If I'm not mistaken. Killer Instinct has learing AI aswell. Learning frame traps& combos. Mimicing most player movement,etc..

Edit: Is there really no way for AI to learn small things. At least production order? or most common Teching or generic National Focus?.

Yes, learning such things as production, tech or national focus order / optimisation is quite doable, even simple at the outset. Problems come from over-training which can be curbed by either setting hard limits or retraining afterwards to reward more general ideas than what was optimised originally.

For instance if we train a production AI by showing it a bunch of data from player games, the AI will not attempt to optimise production order or numbers but rather will attempt to copy player actions. These are not the same thing. If we have reason to believe that a human player has a good idea of the optimisation we can however use that as a starting point, take the AI and then give it a new set of objectives to train for; for instance give it points per produced unit of equipment weighted with their combat values. Now we will actually get a production AI that tries to optimise production but it might end up doing something completely ridiculous like only producing infantry equipment if that is the most effective unit of equipment. So we need hard limits or guidelines OR we need to, from the get go, make the AI to optimise some better set of measurements.

Teching and national focuses should be relatively simple as long as you don't mind that the AI always does the same things; the average of what human players do. That is unless you plug that part of the AI to take into account information of the general situation, in which case it quickly becomes loads more complicated.

So yes, can be done but it is not an open-and-shut case of creating a brilliant AI. You need a brilliant designer to create a brilliant AI.

EDIT: Also notice that Killer Instinct is orders of magnitude simpler than any Grand Strategy Game. Despite the game probably having more depth than I could ever grasp it is still relatively simple to devise an AI to play the game and then have it optimise chance to win.

 

[lekim]

There are always 2 problems with AI, it is either:
- too good
- or too bad

If HoI AI will be like alpha go there will be no fun to play either. But it should not be like in HoI 3.

I think it is quite feasible with modern frameworks to make AI learn how to play on the local level, creating neural network for the front or part of it on the local level.
- Take 20-30 games of Daniel for the final version
- Record the winning wars (fighting in square of 20 provinces over a limited period of time) against lost wars
- Create a tactical AI, which does the extra clicking for the battle plan
- Choosing sub optimal variants to make it easier to play with

On strategic level it is harder to do and probable not necessary (but you have to script a lot of AI, which is what you are doing right now probably).
For me good scripted AI looks like:
- we have a goal - take over china (comes from scenario data or they are our "neighbough" and we are at war or other conditions)
- If we have no war and china's power is less then ours then start preparations, estimate the positive and negative conquences, probably compare to the other grad actions (like insated start a war with ussr)
- Get Intel
- Get troops
- Create battle plans
- Look for opportunities etc etc

So this way AI is dynamic and manageable and predictable to a certain degree (it is a game for fun not a real war - if you want a real war go for dota but remember bots are better there already).

 

[lekim]

Better AI is the one which makes players engage for longest, so this should be an optimization criteria for learning btw.

 

[AapoAlas]

I think it is quite feasible with modern frameworks to make AI learn how to play on the local level, creating neural network for the front or part of it on the local level.
- Take 20-30 games of Daniel for the final version
- Record the winning wars (fighting in square of 20 provinces over a limited period of time) against lost wars
- Create a tactical AI, which does the extra clicking for the battle plan
- Choosing sub optimal variants to make it easier to play with

This would essentially need to be a bunch of games played from a particular starting position on a limited map. One for even forces, one for superior numbers, one for inferior, one for technological advantage, one for disadvantage, one for superior numbers but technological disadvantage, one for inferior numbers but tech advantage. Maybe even one for superiority in numbers and tech. Then each of these scenarios would need to be played multiple times to give some semblance of usability to the data set instead of just overfitting to a small amount of cases. If all of this could be successfully pulled off there would still be the problem of having the AI recognise an actual situation and using the neural network in a situation it has never seen before, that is the actual game. Or if you trained the AI from Germany games (I'm sure Daniel loves those.) and then made it play Japan?

Of course there is also the problem of exploding numbers. Even with a limited map ... And what about teaching the AI attacking and defending rivers and mountains? Ugh.

I'm not sure we're talking about something feasible here. Chess, yes. Go, yes. Risk, yes. I can even imagine some simpler strategy games (Empires in Arms I have my own eyes on...) being feasible. HoI... I'm not so sure. Maybe.

Better AI is the one which makes players engage for longest, so this should be an optimization criteria for learning btw.

Christ I can only begin to imagine what sort of an undertaking that would be This did bring to mind something quite cool though, if you're up for some reading.

 

[GermanPower]

By too good? You mean a AI playing unfairly? Getting lots and lots of bonuses? haha

It's rather hard to beat a human at any game.

 

[Bap]

There are always 2 problems with AI, it is either:
- too good
- or too bad

Can you tell me of a game that has too good AI? I`d like to play it...

If HoI AI will be like alpha go there will be no fun to play either.

No fun? This is one of my dreams, perhaps one day it will come true!