Thus at each word position t of the input, the model takes as input the correct sequence of tokens w 1:t , and uses them to compute a probability distribution over possible next words so as to compute the model's loss for the next token w t+1 . Then we move to the next word, we ignore what the model predicted for the next word and instead use the correct sequence of tokens w 1:t+1 to estimate the probability of token w t+2 . This idea that we always give the model the correct history sequence to