i . These are used to project the inputs into separate key, value, and query embeddings separately for each head, with the rest of the self-attention computation remaining unchanged. In multi-head attention, instead of using the model dimension d that's used for the input and output from the model, the key and query embeddings have dimensionality d k , and the value embeddings are dimensionality d v (in the original transformer paper d k = d v = 64). Thus for each head i, we have weight layers W Q i ∈ R d×d k , W K i ∈ R d×d k , and