[0 0 0 0 1 0 0 ... 0 0 0 0] 1 2 3 4 5 6 7 ... ... |V| The feedforward neural language model (sketched in Fig. 7 .13) has a moving window that can see N words into the past. We'll let N-3, so the 3 words w t−1 , w t−2 , and w t−3 are each represented as a one-hot vector. We then multiply these one-hot vectors by the embedding matrix E. The embedding weight matrix E has a column for each word, each a column vector of d dimensions, and hence has dimensionality d × |V |. Multiplying by a one-hot vector that has only one non-zero element x i = 1 simply selects out the relevant column vector for word i, resulting in the embedding for word i, as shown in Fig. 7 .12.