Since each output vector, y i , is computed independently, the processing of an entire sequence can be parallelized via matrix operations. The first step is to pack the input embeddings x i into a matrix X ∈ R N×d h . That is, each row of X is the embedding of one token of the input. We then multiply X by the key, query, and value weight matrices (all of dimensionality d × d) to produce matrices Q ∈ R N×d , K ∈ R N×d , and V ∈ R N×d , containing all the key, query, and value vectors in a single step.