As shown in Fig. 11 .3, the full set of self-attention scores represented by QK T constitute an all-pairs comparison between the keys and queries for each element of the input. In the case of causal language models in Chapter 9, we masked the upper triangular portion of this matrix (in Fig. ??) to eliminate information about future words since this would make the language modeling training task trivial. With bidirectional encoders we simply skip the mask, allowing the model to contextualize each token using information from the entire input.