Lucky Dice

Do you have lucky dice?  This is a slightly modified version of something I wrote a couple years ago that I think has a place here.

I have encountered more than one gamer over the years who believes that he has lucky dice. These dice are reported to roll the highest damage, the most critical hits, and generally turn the player's characters into unstoppable juggernauts. Of course, I did not notice any disproportionate numbers of good or bad rolls. I argued with these players that their perceptions were tainted by the availability heuristic and confirmation bias. When they can more easily recall rolling good numbers with certain dice, they label the dice "lucky". Then, whenever those dice roll well, they say, "See? This proves the dice are lucky!" Whenever the dice roll poorly, they say, "That was a fluke! These lucky dice usually roll well.", and fail to store the failures for future recall, feeding back into the availability heuristic. Well, there is a relatively easy way to determine if a die is lucky.

If you think you have a lucky die, record what it rolls EVERY TIME. If you believe it is lucky only in a certain situation (e.g.: "This is my saving-throw die!"), then record what it rolls every time you roll it in that situation. Do not leave out any rolls. There are no Mulligans. When you have a good amount of rolls (at least 30, but more is better), you can calculate the z-score for them, then compare that z-score to a threshold. This comparison is called a one-tailed test of proportion, and we use this specific type of comparison to test the hypothesis that your die rolls a higher (not just different) proportion of good rolls than a fair die. Though there are different ways to calculate the z-score based on how sure you want to be of the results, the following simple examples should be good enough for most of you.

The equation for z is:
z = (p - Po) / (sqrt((Po(1 - Po))/n))

The variables are:
p: the proportion of the rolls you recorded that were good (natural 20, 11 or higher, 8-10 on 1d10, etc...). p = ((# of good rolls) / (total # of rolls)).
Po: the proportion expected of fair dice (0.05 for a natural 20, 0.5 for 11 or higher on d20, 0.3 for 8-10 on d10). This should technically be written as pi subscript 0.
n: the total number of rolls you recorded

To make things easy, we will use an alpha of .05 in the examples (I explain further down). This gives us a threshold of 1.645. So, all you have to do is figure out z using the equation above (or one of the examples below), and if z > 1.645 you can claim that you may have a lucky die.

Example 1:
"My die rolls high."
Po = 0.5 (half of a fair die's rolls will be above the mean, and half below)
z = (p - 0.5) / (sqrt(0.25 / n))

Example 2:
"I roll 20s."
Po = 0.05
z = (p - 0.05) / (sqrt(0.0475 / n))

To elaborate on Example 1, let's say that I claim I have a d20 that usually rolls at least an 11, and I have countless stories of how it let my characters defeat monsters and overcome challenges. I want to prove to my naysayer friends that it really is lucky, so I write down every roll for a month. I rolled the die 100 times. 58 of the rolls were at least 11, and 42 were 10 or less. I get really excited about rubbing it in everyone's faces, but remember that there's more to do. After all, I only rolled the die 100 times, and even a fair die is likely to roll 58/100 good rolls eventually. Fair dice are only expected to roll exactly 50/50 as the number of rolls approaches infinity. So, I plug in the variables:
p = 0.58
n = 100
Po = 0.5
z = (0.58 - 0.5) / (sqrt(0.25 / 100)) = 1.600
1.600 < 1.645 (our threshold)
We should probably not be impressed that 58 of the rolls were above average. If the die had rolled 59 good rolls out of 100, z would have been 1.800.  What would that mean?

This is where I need to explain about alpha. In this case, alpha is the probability that you have a fair die even if z is greater than the threshold. Usually, an alpha of 0.05 lets us feel confident enough that the effect we have observed is based on the properties of what we are observing instead of a statistical fluke of the sample of observations.  It is easy to increase our confidence by using a smaller alpha, such as 0.01 (which would increase the z threshold), but using a smaller alpha increases the probability that we falsely think a lucky die is fair. There is a trade-off between the risks. Personally, I would demand a high degree of confidence, and would insist on an alpha of at most 0.01.  The best way to manage both risks of false results is to get a bigger sample of rolls. So, if you want to be more sure of whether your die is lucky or not, record 200 rolls, or 500. The bigger n gets, the better.


When alpha is 0.01, the z threshold is 2.328, so even 59% good rolls does not make me confident that the die is lucky.  For an alpha of 0.01, 62 out of 100 rolls would have to be above average.
To elaborate on Example 2, let's find out how many natural 20s have to be rolled with an alpha of 0.01 for someone to tell me that he rolls 20s.  
(p-0.05)/(sqrt(0.0475 / 100)) > 2.328
(p-0.05)/0.0218 > 2.328
p-0.05 > .0507
p > .1007
You would have to roll 11 20s out of 100 rolls.


Good luck.