A simple but surprisingly effective approach to applying transformers to summarization is to append a summary to each full-length article in a corpus, with a unique marker separating the two. More formally, each article-summary pair (x 1 , ..., x m ), (y 1 , ..., y n ) in a training corpus is converted into a single training instance (x 1 , ..., x m , δ , y 1 , ...y n ) with an overall length of n + m + 1. These training instances are treated as long sentences and then used to train an autoregressive language model using teacher forcing, exactly as we did earlier.