x t h t y t Figure 9 .2 Simple recurrent neural network after Elman (1990). The hidden layer includes a recurrent connection as part of its input. That is, the activation value of the hidden layer depends on the current input as well as the activation value of the hidden layer from the previous time step. Fig. 9 .2 illustrates the structure of an RNN. As with ordinary feedforward networks, an input vector representing the current input, x t , is multiplied by a weight matrix and then passed through a non-linear activation function to compute the values for a layer of hidden units. This hidden layer is then used to calculate a corresponding output, y t . In a departure from our earlier window-based approach, sequences are processed by presenting one item at a time to the network. We'll use 9.2 • RECURRENT NEURAL NETWORKS 181 subscripts to represent time, thus x t will mean the input vector x at time t. The key difference from a feedforward network lies in the recurrent link shown in the figure with the dashed line. This link augments the input to the computation at the hidden layer with the value of the hidden layer from the preceding point in time.