We trained causal transformer language models in Chapter 9 by making them iteratively predict the next word in a text. But eliminating the causal mask makes the guess-the-next-word language modeling task trivial since the answer is now directly available from the context, so we're in need of a new training scheme. Fortunately, the traditional learning objective suggests an approach that can be used to train bidirectional encoders. Instead of trying to predict the next word, the model learns to perform a fill-in-the-blank task, technically called the cloze task (Taylor, 1953) . To cloze task see this, let's return to the motivating example from Chapter 3. Instead of predicting which words are likely to come next in this example: