effect size a bigger δ means that A seems to be way better than B; a small δ means A seems to be only a little better. Why don't we just check if δ (x) is positive? Suppose we do, and we find that the F 1 score of A is higher than B's by .04. Can we be certain that A is better? We cannot! That's because A might just be accidentally better than B on this particular x. We need something more: we want to know if A's superiority over B is likely to hold again if we checked another test set x , or under some other set of circumstances.