The corpus used in training BERT and other early transformer-based language models consisted of an 800 million word corpus of book texts called BooksCorpus (Zhu et al., 2015 ) and a 2.5 Billion word corpus derived from the English Wikipedia, for a combined size of 3.3 Billion words. The BooksCorpus is no longer used (for intellectual property reasons), and in general, as we'll discuss later, state-of-the-art models employ corpora that are orders of magnitude larger than these early efforts. To train the original BERT models, pairs of sentences were selected from the training corpus according to the next sentence prediction 50/50 scheme. Pairs were sampled so that their combined length was less than the 512 token input. Tokens within these sentence pairs were then masked using the MLM approach with the combined loss from the MLM and NSP objectives used for a final loss. Approximately 40 passes (epochs) over the training data was required for the model to converge.