A span is a contiguous sequence of one or more words selected from a training text, prior to subword tokenization. In span-based masking, a set of randomly selected spans from a training sequence are chosen. In the SpanBERT work that originated this technique (Joshi et al., 2020) , a span length is first chosen by sampling from a geometric distribution that is biased towards shorter spans an with upper bound of 10. Given this span length, a starting location consistent with the desired span length and the length of the input is sampled uniformly.