The final loss is the sum of the BERT MLM loss and the SBO loss. Fig. 11 .6 illustrates this with one of our earlier examples. Here the span selected is and thanks for which spans from position 3 to 5. The total loss associated with the masked token thanks is the sum of the cross-entropy loss generated from the prediction of thanks from the output y 4 , plus the cross-entropy loss from the prediction of thanks from the output vectors for y 2 , y 6 and the embedding for position 4 in the span. Figure 11 .6 Span-based language model training. In this example, a span of length 3 is selected for training and all of the words in the span are masked. The figure illustrates the loss computed for word thanks; the loss for the entire span is based on the loss for all three of the words in the span.