Bidirectional encoders overcome this limitation by allowing the self-attention mechanism to range over the entire input, as shown in Fig. 11 .2. The focus of bidirectional encoders is on computing contextualized representations of the tokens in an input sequence that are generally useful across a range of downstream applications. Therefore, bidirectional encoders use self-attention to map sequences of input embeddings (x 1 , ..., x n ) to sequences of output embeddings the same length (y 1 , ..., y n ), where the output vectors have been contextualized using information from the entire input sequence.