20 combat die rolls vs computer in iron man game.

[unmerged(785632)]

ME vs Computer

before modifier

1 - 7
2 - 4
2 - 3
3 - 5
6 - 9
8 - 9
2 - 1
2 - 2
1 - 3
4 - 5
3 - 1
2 - 2
1 - 1
1 - 7
1 - 8
2 - 9
4 - 4
9 - 9
5 - 4
6 - 7
7 - 6

 

[johnnyb]

The computer is against you!!

 

[asamy]

Not enough, you need a bigger sample

 

[grisamentum]

ME vs Computer

before modifier

1 - 7
2 - 4
2 - 3
3 - 5
6 - 9
8 - 9
2 - 1
2 - 2
1 - 3
4 - 5
3 - 1
2 - 2
1 - 1
1 - 7
1 - 8
2 - 9
4 - 4
9 - 9
5 - 4
6 - 7
7 - 6

correct me if i'm wrong but isn't this actually 21 die rolls

 

[208]

Please update your post after you collect the other 980 rolls needed to create a half-decent sample size.

 

[grisamentum]

Not enough, you need a bigger sample

not necessarily. if the skew were strong enough we could be sure it was not a "normal" dice from a relatively small sample. disproving the claim "the dice is fair" needs a large set if the skew is small, but smaller set if the skew is much bigger.

here the computer average is 5.048, and the player average is 3.429, so i think perhaps a little investigation is warranted...

 

[grisamentum]

Please update your post after you collect the other 980 rolls needed to create a half-decent sample size.

he's not trying to create a "representative sample" of dice roll. he is trying to test a theoretical claim like "the dice is fair."

 

[grisamentum]

The more important issue here is if he is fighting against a lucky nation, and do lucky nation get bonuses to otherwise "unmodified" combat rolls. Ie is the combat roll 1d10+1 for lucky nations, and 1d10 otherwise?

 

[Tom013]

Agreed that the sample size is too low. Even so, the median difference is -1 (i.e. 1 point against you). The mean is just slightly worse at -1.6. Only a couple or rolls are giving the sample set a bias at all...

 

[Tom013]

he's not trying to create a "representative sample" of dice roll. he is trying to test a theoretical claim like "the dice is fair."

You cannot test that theoretical claim without a sufficient sample size.

The more important issue here is if he is fighting against a lucky nation, and do lucky nation get bonuses to otherwise "unmodified" combat rolls. Ie is the combat roll 1d10+1 for lucky nations, and 1d10 otherwise?

According to the developers, lucky nations do not get bonuses in combat rolls. If they did get such a bonus, this sample set would appear to match with that for a player vs. a lucky AI (though the sample set is really too small to say this conclusively even so).

 

[unmerged(785632)]

could every one record 20 samples so we got enough sample size.... collect sample from 1 player only is not fair enough...

ME <= bad luck guy may be ..... lol

 

[zodium]

Loaded up R, punched in the values:

> player <- c(1,2,2,3,6,8,2,2,1,4,3,2,1,1,1,2,4,9,5,6,7)
> ai <- c(7,4,3,5,9,9,1,2,3,5,1,2,1,7,8,9,4,9,4,7,6)
> t.test(ai,player)

	Welch Two Sample t-test

data:  ai and player
[b]t = 1.9617, df = 39.24, p-value = 0.05692[/b]
alternative hypothesis: true difference in means is not equal to 0

95 percent confidence interval:
 [b]-0.05002486  3.28812009[/b]

sample estimates:
mean of x mean of y 
 5.047619  3.428571

I'm not going to calculate the power, since it's safe to say you need a bigger sample.

 

[grisamentum]

You cannot test that theoretical claim without a sufficient sample size.

of course you can. you can test any claim with any sample size. it's just that the significance of your result will quite low if you have an unclear result with a smaller sample size. there is no magic "sufficient sample size" and certainly not the 1000 that someone else demanded.

the stronger the result, the smaller the sample size necessary to confirm it. here we don't have a particularly strong result but there is certainly something that merits more investigation.

 

[Al. I. Cuza]

Isn't 2000 the ideal sample size or something? Didn't pay much attention in statistics class.

 

[zodium]

Isn't 2000 the ideal sample size or something? Didn't pay much attention in statistics class.

As the OP explained, it depends on the magnitude you expect the effect to have. A strong effect can be detected with n as low as 15, at least in Fisherian (what you didn't learn in stats class :V) theory.

of course you can. you can test any claim with any sample size. it's just that the significance of your result will quite low if you have an unclear result with a smaller sample size. there is no magic "sufficient sample size" and certainly not the 1000 that someone else demanded.

the stronger the result, the smaller the sample size necessary to confirm it. here we don't have a particularly strong result but there is certainly something that merits more investigation.

Er, no. If your inference relies on the Central Limit Theorem (if you're not a trained statistician, it does, and if you are, it probably still does), at least, you can forget about using anything smaller than n = 15.

 

[grisamentum]

Isn't 2000 the ideal sample size or something? Didn't pay much attention in statistics class.

2000 what?

 

[HansBaer]

Isn't 2000 the ideal sample size or something? Didn't pay much attention in statistics class.

Depends on the standard deviation and probabilities. With dices, 20 throws are utterly useless in any scenario, even if he just got 1s.
I bet i would need max. 30 battles to find one with similar outcomes on player's side.

 

[Tom013]

of course you can. you can test any claim with any sample size. it's just that the significance of your result will quite low if you have an unclear result with a smaller sample size. there is no magic "sufficient sample size" and certainly not the 1000 that someone else demanded.

the stronger the result, the smaller the sample size necessary to confirm it. here we don't have a particularly strong result but there is certainly something that merits more investigation.

Uh... okay. Sorry for not being precise enough in my language: You can draw no substantial conclusions about that theoretical claim from such a small sample size. As you put it, "the significance of the result will be quite low" ... and in this case, it is indeed quite low.

There are statistical measures to help you determine what sample size is "good enough." 20 is not it in this case. Take a look at Zodium's post.

Er, no. If your inference relies on the Central Limit Theorem (if you're not a trained statistician, it does, and if you are, it probably still does), at least, you can forget about using anything smaller than n = 15.

Does this mean my lifelong assumption that all things are constant biases needs to be revisited? My sample size of one had me quite convinced. Blast!

 

[grisamentum]

Uh... okay. Sorry for not being precise enough in my language: You can draw no substantial conclusions about that theoretical claim from such a small sample size. As you put it, "the significance of the result will be quite low" ... and in this case, it is indeed quite low.

There are statistical measures to help you determine what sample size is "good enough." 20 is not it in this case. Take a look at Zodium's post.

I did. The p-value is .05692, which means the result in this case is actually almost significant enough to reject the null hypothesis (the dice is fair). I also agreed he needs a bigger sample size, but the result is actually quite close to rejecting the null hypothesis already.

 

[zodium]

You won't need that many, though, so if you like, you could collect maybe 100 (shot from the hip). I'd test it myself if I wasn't very confident the AI doesn't cheat on dice rolls, even lucky AIs.

I did. The p-value is .05692, which means the result in this case is actually almost significant enough to reject the null hypothesis (the dice is fair). I also agreed he needs a bigger sample size, but the result is actually quite close to rejecting the null hypothesis already.

There is no such thing as "almost" significant, since significance is a completely arbitrary distinction. The p-value 0.57 means that there is a 5.7% chance of observing a difference as great or greater than the one we observed, assuming various assumptions are met. P-values do not provide information about the appropriateness of your sample size, the confidence interval, or better yet, power analysis, does that. Since the 95% CI is fairly large, he needs a larger sample size.

If you're tangentially interested in this stuff, a great video about dispelling p-value worship is Geoff Cummings' Dance of the p-Values.