With a predicted probability distribution for each masked item, we can use crossentropy to compute the loss for each masked item-the negative log probability assigned to the actual masked word, as shown in Fig. 11 .5. The gradients that form the basis for the weight updates are based on the average loss over the sampled learning items from a single training sequence (or batch of sequences).