This description of the self-attention process has been from the perspective of computing a single output at a single time step i. However, since each output, y i , is computed independently this entire process can be parallelized by taking advantage of efficient matrix multiplication routines by packing the input embeddings of the N tokens of the input sequence into a single matrix X ∈ R N×d . That is, each row of X is the embedding of one token of the input. We then multiply X by the key, query, and value matrices (all of dimensionality d × d) to produce matrices Q ∈ R N×d , K ∈ R N×d , and V ∈ R N×d , containing all the key, query, and value vectors: