Second, we'll introduce in this chapter the idea of pretraining and fine-tuning. We call pretraining the process of learning some sort of representation of meaning for words or sentences by processing very large amounts of text. We'll call these pretrained models pretrained language models, since they can take the form of the transformer language models we introduced in Chapter 9. We call fine-tuning the process of taking the representations from these pretrained models, and further training the model, often via an added neural net classifier, to perform some downstream task like named entity tagging or question answering or coreference. The intuition is that the pretraining phase learns a language model that instantiates a rich representations of word meaning, that thus enables the model to more easily learn ('be fine-tuned to') the requirements of a downstream language understanding task. The pretrain-finetune paradigm is an instance of what is called transfer learning in machine learning: the method of acquiring knowledge from one task or domain, and then applying it (transferring it) to solve a new task. Of course, adding grounding from vision or from real-world interaction into pretrained models can help build even more powerful models, but even text alone is remarkably useful, and we will limit our attention here to purely textual models. There are two common paradigms for pretrained language models. One is the causal or left-to-right transformer model we introduced in Chapter 9. In this chapter we'll introduce a second paradigm, called the bidirectional transformer encoder, and the method of masked language modeling, introduced with the BERT model (Devlin et al., 2019 ) that allows the model to see entire texts at a time, including both the right and left context.