Consider a tiny text classification example with a test set x of 10 documents. The first row of Fig. 4 .8 shows the results of two classifiers (A and B) on this test set, with each document labeled by one of the four possibilities: (A and B both right, both wrong, A right and B wrong, A wrong and B right); a slash through a letter ( B) means that that classifier got the answer wrong. On the first document both A and B get the correct class (AB), while on the second document A got it right but B got it wrong (A B). If we assume for simplicity that our metric is accuracy, A has an accuracy of .70 and B of .50, so δ (x) is .20. Now we create a large number b (perhaps 10 5 ) of virtual test sets x (i) , each of size n = 10. Fig. 4 .8 shows a couple examples. To create each virtual test set x (i) , we repeatedly (n = 10 times) select a cell from row x with replacement. For example, to create the first cell of the first virtual test set x (1) , if we happened to randomly select the second cell of the x row; we would copy the value A B into our new cell, and move on to create the second cell of x (1) , each time sampling (randomly choosing) from the original x with replacement. 1 2 3 4 5 6 7 8 9 10 A% B% δ () Now that we have the b test sets, providing a sampling distribution, we can do statistics on how often A has an accidental advantage. There are various ways to compute this advantage; here we follow the version laid out in Berg-Kirkpatrick et al. (2012) . Assuming H 0 (A isn't better than B), we would expect that δ (X), estimated over many test sets, would be zero; a much higher value would be surprising, since H 0 specifically assumes A isn't better than B. To measure exactly how surprising is our observed δ (x) we would in other circumstances compute the p-value by counting over many test sets how often δ (x (i) ) exceeds the expected zero value by δ (x) or more: