Siege Rolls; Confirmation Bias my ARSE!!!

[Wizzington]

I have data that says otherwise. Just because you say so, doesn't make it so. Call me incredulous given the recent history of world leaders, bankers and various people in a position of authority throughout the world to not be honest. The only bias that I see that is confirmed is that rolling a "1" is more probable than a 1:14 chance and rolling a "14" is less than a 1:14 chance.
This is going no where ~ Peace out!

Truly, the oppression inherent to global politics finds its most poignant expression in the vicious lies of game developers regarding the siege dice rolls of AI armies.

One day we will all be free of the oppressive social order of the capitalist economic system and we can meet as equals, sharing truths long suppressed by the socio-economic elites and their cronies. On that blessed day I will reveal to you the secret signs and handshakes required to unlock the "golden dice" of EU4, allowing you to finally siege on equal terms with the filthy cheating illuminati AI.

Ooooooor you're just experiencing confirmation bias and is hella salty about it.

Who can say?

 

[Zelius]

You are right, but only (4) coin flips is not adequate. My data set is 257 on a random outcome of 14 possibilities. That means I used 18 times the possible outcomes. So an equivalent sample for a coin flip would be 36 or lets say 40 for round numbers. I just flipped a coin 40 times and got heads 24 and tails 16. To apply the same logic I did for the siege rolls, the mean is 60% heads. Is 60% indicative of the actual odds of an even sided coin? Let's check the confidence interval. The standard deviation is 7.84% and we need 1.96 standard dev's to be within the 95% confidence range. That means the 60% mean +/- 15.37%. Since we know the true probability is 50% exactly, and our confidence interval falls within this range, the data set is valid and the coin is fair. If I got a data set showing the coins confidence interval that did not include the 50% value, then we could question the validity of the coin being fair. I hope this makes sense. I think folks are getting very hot under the collar so I'm going to drop this now. It's almost like discussing politics or religion, gesh.

This is NOT HOW STATISTICS WORKS.

All you have calculated in the example case is that the mean is probably between ~0.45 and ~0.75. It does NOT prove that the actual mean is 0.50000

But then again, your results may be part of the 5% outside of the confidence intervals - certainly many real, published research papers were found to be. I suggest that, assuming you treated your data properly in accordance with "facts, logic, science and math", that you reproduce the experiment with a larger sample size, since it's evidently cheap and easy to do so.

 

[Wizzington]

(PS: I can say and it's the one where you're blaming your failures on AI cheating)

 

[tbhayward]

So Wiz is illuminati, confirmed.

 

[Wizzington]

So Wiz is illuminati, confirmed.

Man, if you didn't realize this ages ago you're not gonna survive the bayesian purges of 2018. I'm sorry.

 

[Republic of Mercury]

Man, if you didn't realize this ages ago you're not gonna survive the bayesian purges of 2018. I'm sorry.

Pfft, more like Byzantine purges. That's right, Blurple, I see you. Playing the long game. Taking the long odds, one might say.

 

[Philadelphus]

I just want to point out that I'm in the physical sciences (astrophysics), and I can say that generally you need at minimum a 5-sigma result for anyone to take it seriously enough to consider. 3-sigma events are ridiculously common when you're dealing with large data sets.

 

[jcranmer]

Every time this topic comes up PDX is quick to give the default reply that it's confirmation bias. I never accepted this as it appeared to me that "Disease Outbreaks" are much more common that "Wall Breaches" when I am sieging a hostile fort. Therefore, I took it upon myself to write down the actual siege rolls during a Brandenburg campaign. This is early on before cannons but all we are concerned with is the dice rolls, not what they do. In case you are not aware, a dice roll of "1" is a disease outbreak while a dice roll of 14 is a wall breach. Since the game is based on a 14 sided die, it would seem they both have equal odds of happening at a rate of 1:14. I wrote down all the siege rolls to acquire a sample size of 100 rolls. This should be large enough to illustrate the point. With 100 rolls, and 1:14 odds, we should see approximately 7 rolls of a "1" and 7 rolls of a "14". However, the actual data is much much worse. I got 18 rolls at "1" and 2 rolls at "14". So these supposedly equal odds are not so equal after all to the tune of a disease outbreak being 9 times more likely than a wall breach. I realize a sample of 100 rolls may not accurately reflect the 9 times more likely occurrence but the point of this post is to illustrate that a dice roll of "1" is not at all equal odds of happening as a dice roll of "14". The only rational explanation of this is that the random dice roll generator is skewed to give lower dice roll results for the player versus truly random dice. PDX, the evidence is indisputable, your confirmation bias excuse is complete and utter CRAP!
If the community would like, I can continue to gather data in this regard. However, I am convinced of the results and will be waiting for the 1.19 patch before seeking more data sets.

P(100 rolls of a 14-sided die yields at least 18 1s) ~= 0.00024, which is to say, if 10K people did this experiment and the true distribution were unbiased, we should expect to see 24 people having a result at least as lopsided as yours. Now, it's easy to say, "p < 0.05, we reject the null hypothesis!", but to actually say that is p-hacking. You're posting this message precisely because the results is so unfair not in your favor, so you're ignoring the results of all the runs which didn't seem so unfair. When you aggregate multiple experiments, the p-value that you look for in each experiment should be greatly reduced. Indeed, others have posted experimentation which found that the results were not so biased as in your case, which greatly suggests that your experimentation was merely statistical fluke.

It should be worth noting that I once got two heir-dies events within a very short time and at first said "WHAT BS, THIS SUCKS." Then I looked up the actual MTTH, computed how many game-years of time it would take for such an event to happen and then noticed that, given the amount of time I played, I had somewhere around a 10% chance of it happening.

I didn't trust the source RNG implementation when I came to EU4, but I did test it empirically, and it converged to the mean one would expect. IIRC there's a "testrng" console command remaining from that, though I'm unsure if output means anything without knowing context.

The source RNG generates a 31 bit positive random integer or alternatively a floating point number in the range [0, 1[. However, there's a possibility for the programmer to use generated numbers in an erroneous way due to lack of high level random variable shaping functions in Clausewitz. I've seen nothing of that in dice roll code though since that is utterly simple. e.g. die_roll = random31uint % 6 + 1; (lazy almost linear approximation of one die).

I think the bigger issue is the fact pure RNG is used which will sometimes be unfair by its very nature. It would perhaps be cool to try something that adjusts the random variable's expected value to cancel out mean unbalanced series after some number of outcomes. Not my concern any more though, but I once read about some version of Civilization taking a similar intervention approach in battle outcomes to prevent players from feeling unlucky or feeling that the game cheats even though it doesn't. (Perhaps they simply reduced the probability of the really bad events for humans, can't recall.)

Doing modulus operations to estimate the uniform distribution does introduce a bias of about N/2**31. For a 14-sided die, that comes out to 1 in 100 million or so, which is small enough to not matter, and I wouldn't accuse it of meaningful bias. If you're using modulus to do the weighted choice bias for the pulse triggers, that's going to be an error of maybe 1% in some of them, which would be enough for me to ask you to do the proper estimation there.

What PRNG are you using? LCPRNGs are known to be simple to implement, but they do have a particularly annoying property that their low-order bits have much smaller period than higher-order bits (with particularly disastrous consequences for modulo operations to estimate the uniform distribution). It's actually hard to find simple clear-cut advice for which PRNG to use for Monte-Carlo simulations, but the Mersenne Twister and xorshift generators do seem to be the best matches for nearly any scenario (except the need for CSPRNGs, but that's a different ball game).

 

[Cymsdale]

Now, I want to be clear that I think the OP has no basis to his claim. I just want to point out that there was an incident where some users claimed the RNG of a particular event in a game made by [FAMOUS_DEVELOPER] was not not 50/50 even though that's what the developer said it was. The developers told them they were wrong and it was confirmation bias. Pages and pages of threads were made about how people don't understand statistics. Lots of talk about pseudo random number generators are not real random (even thought hat really doesn't have anything to do about anything, people just like to bring up PRNGs in these type of discussions cause it makes them sound smart). The people claiming it were given a lot of shit from fanboys all over the forum. Lol you don't understand statistics and math and probability, lol blah blah blah.

It went on for a while until someone posted data of a very significant sample of rolls. Much more than the OP did in this particular case. [FAMOUS_DEVELOPER] was forced to look at it a little more and sheepishly had to conclude "Uh... yeah there was a bug. Sorry." The people making the claims were vindicated. The people who laughed at them for so long looked like idiots.

That's not what's happening here though. OP needs more data if he wants to win this. A lot more. He probably won't like the results though.

Just a lesson though, sometimes there really IS something wrong with the RNG. You gots to prove it though.

 

[durbal]

To be fair, it's not confirmation bias if someone records real results and analyzes them.

But I don't believe the die rolls are biased.

 

[Wizzington]

The dice rolls are 100% the same between AI and not AI. There is no AI cheating in regards to dice rolls and there has never been any AI cheating in regards to dice rolls.

End of story.

 

[Wizzington]

Pfft, more like Byzantine purges. That's right, Blurple, I see you. Playing the long game. Taking the long odds, one might say.

Well, everyone who likes Byzantium *is* going to be purged, but that's more of a side thing really.

 

[Republic of Mercury]

Well, everyone who likes Byzantium *is* going to be purged, but that's more of a side thing really.

I... I... I mean come on! +3 Tolerance of the true faith and + 3 missionary strength in addition to the patriarchal authority bonuses?! How I am supposed to stay away from that?

 

[Wizzington]

I... I... I mean come on! +3 Tolerance of the true faith and + 3 missionary strength in addition to the patriarchal authority bonuses?! How I am supposed to stay away from that?

Ever heard the term 'honeypot'?

No matter, you won't have to trouble yourself about it for long.

 

[Shock360]

Now, I want to be clear that I think the OP has no basis to his claim. I just want to point out that there was an incident where some users claimed the RNG of a particular event in a game made by [FAMOUS_DEVELOPER] was not not 50/50 even though that's what the developer said it was. The developers told them they were wrong and it was confirmation bias. Pages and pages of threads were made about how people don't understand statistics. Lots of talk about pseudo random number generators are not real random (even thought hat really doesn't have anything to do about anything, people just like to bring up PRNGs in these type of discussions cause it makes them sound smart). The people claiming it were given a lot of **** from fanboys all over the forum. Lol you don't understand statistics and math and probability, lol blah blah blah.

It went on for a while until someone posted data of a very significant sample of rolls. Much more than the OP did in this particular case. [FAMOUS_DEVELOPER] was forced to look at it a little more and sheepishly had to conclude "Uh... yeah there was a bug. Sorry." The people making the claims were vindicated. The people who laughed at them for so long looked like idiots.

That's not what's happening here though. OP needs more data if he wants to win this. A lot more. He probably won't like the results though.

Just a lesson though, sometimes there really IS something wrong with the RNG. You gots to prove it though.

I'm not sure what the point of this post was(unless your telling the OP to just get a large sample size- in which case my bad )

To be fair, it's not confirmation bias if someone records real results and analyzes them.

But I don't believe the die rolls are biased.

It is confirmation bias due to the thousands of siege rolls that have been ignored

 

[Republic of Mercury]

Ever heard the term 'honeypot'?

No matter, you won't have to trouble yourself about it for long.

Worth it.

 

[Cymsdale]

I'm not sure what the point of this post was

TL;DR: One time a developer said there was confirmation bias but they were wrong and looked dumb.

 

[Wizzington]

TL;DR: One time a developer said there was confirmation bias but they were wrong and looked dumb.

I mean, if you prove that the mersienne twister is biased in such a precisely specific way that somehow it always ends up giving favorable siege rolls to the AI in Europa Universalis 4, you're probably gonna make a whole lot of people look dumb.

I don't think it's very likely though.

 

[Cymsdale]

I mean, if you prove that the mersienne twister is biased in such a precisely specific way that somehow it always ends up giving favorable siege rolls to the AI in Europa Universalis 4, you're probably gonna make a whole lot of people look dumb.

I don't think it's very likely though.

No one thought [FAMOUS_DEVELOPER] could mess up something as simple as a 50% roll, but alas.

Is the OP even claiming that the roll is done differently for the AI versus the player, or simply that the numeric results do not follow a uniform distribution? Hard to tell, I'm not sure the OP could understand the distinction anyhow.

I don't think the OP has a case, he's done nothing significant to prove it.

 

[Lilje]

Did you get AI rolls sample too, if not could you try doing that. That is something that does matter too in determining if siege rolls are biased against the player.

Here you go, with larger sample size as well. Bonus is that the AI in this particular case is the Ottomans, which sometimes might "feel" like an unfair foe.