Linear Layer Figure 9 .21 Training a transformer as a language model. Note the key difference between this figure and the earlier RNN-based version shown in Fig. 9 .6. There the calculation of the outputs and the losses at each step was inherently serial given the recurrence in the calculation of the hidden states. With transformers, each training item can be processed in parallel since the output for each element in the sequence is computed separately. Once trained, we can compute the perplexity of the resulting model, or autoregressively generate novel text just as with RNN-based models.