• In simple Recurrent Neural Networks sequences are processed one element at a time, with the output of each neural unit at time t based both on the current input at t and the hidden layer from time t − 1. • RNNs can be trained with a straightforward extension of the backpropagation algorithm, known as backpropagation through time (BPTT). • Simple recurrent networks fail on long inputs because of problems like vanishing gradients; instead modern systems use more complex gated architectures such as LSTMs that explicitly decide what to remember and forget in their hidden and context layers. • Transformers are non-recurrent networks based on self-attention. A selfattention layer maps input sequences to output sequences of the same length, based on a set of attention heads that each model how the surrounding words are relevant for the processing of the current word. • A transformer block consists of a single attention layer followed by a feedforward layer with residual connections and layer normalizations following each. Transformer blocks can be stacked to make deeper and more powerful networks. • Common language-based applications for RNNs and transformers include: