The encoder-decoder architecture can also be implemented using transformers (rather than RNN/LSTMs) as the component modules. At a high-level, the architecture, sketched in Fig. 10 .15, is quite similar to what we saw for RNNs. It consists of an encoder that takes the source language input words X = x 1 , ..., x T and maps them to an output representation H enc = h 1 , ..., h T ; usually via N = 6 stacked encoder blocks. The decoder, just like the encoder-decoder RNN, is essentially a conditional language model that attends to the encoder representation and generates the target words one by one, at each timestep conditioning on the source sentence and the previously generated target language words. Figure 10 .15 The encoder-decoder architecture using transformer components. The encoder uses the transformer blocks we saw in Chapter 9, while the decider uses a more powerful block with an extra encoder-decoder attention layer. The final output of the encoder H enc = h 1 , ..., h T is used to form the K and V inputs to the crossattention layer in each decoder block.