Non-parametric methods for computing statistical significance were used first in NLP in the MUC competition (Chinchor et al., 1993) , and even earlier in speech recognition (Gillick and Cox 1989, Bisani and Ney 2004) . Our description of the bootstrap draws on the description in Berg-Kirkpatrick et al. (2012) . Recent work has focused on issues including multiple test sets and multiple metrics (Søgaard et al. 2014 , Dror et al. 2017 .