We only have to make one slight change to turn this language model with autoregressive generation into a translation model that can translate from a source text source in one language to a target text in a second: add an sentence separation marker at target the end of the source text, and then simply concatenate the target text. We briefly introduced this idea of a sentence separator token in Chapter 9 when we considered using a Transformer language model to do summarization, by training a conditional language model. If we call the source text x and the target text y, we are computing the probability p(y|x) as follows: Fig. 10.4 shows the setup for a simplified version of the encoder-decoder model (we'll see the full model, which requires attention, in the next section). Fig. 10.4 shows an English source text ("the green witch arrived"), a sentence separator token (<s>, and a Spanish target text ("llegó la bruja verde"). To translate a source text, we run it through the network performing forward inference to generate hidden states until we get to the end of the source. Then we begin autoregressive generation, asking for a word in the context of the hidden layer from the end of the source input as well as the end-of-sentence marker. Subsequent words are conditioned on the previous hidden state and the embedding for the last word generated. Let's formalize and generalize this model a bit in Fig. 10 .5. (To help keep things straight, we'll use the superscripts e and d where needed to distinguish the hidden states of the encoder and the decoder.) The elements of the network on the left process the input sequence x and comprise the encoder. While our simplified figure shows only a single network layer for the encoder, stacked architectures are the norm, where the output states from the top layer of the stack are taken as the final representation. A widely used encoder design makes use of stacked biLSTMs where the hidden states from top layers from the forward and backward passes are concatenated as described in Chapter 9 to provide the contextualized representations for each time step. The entire purpose of the encoder is to generate a contextualized representation of the input. This representation is embodied in the final hidden state of the encoder, h e n . This representation, also called c for context, is then passed to the decoder. The decoder network on the right takes this state and uses it to initialize the first