(We use the notation 1(x) to mean "1 if x is true, and 0 otherwise".) However, although it's generally true that the expected value of δ (X) over many test sets, (again assuming A isn't better than B) is 0, this isn't true for the bootstrapped test sets we created. That's because we didn't draw these samples from a distribution with 0 mean; we happened to create them from the original test set x, which happens to be biased (by .20) in favor of A. So to measure how surprising is our observed δ (x), we actually compute the p-value by counting over many test sets how often δ (x (i) ) exceeds the expected value of δ (x) by δ (x) or more: