The inputs x and outputs y of transformers, as well as the intermediate vectors after the various layers, all have the same dimensionality 1 × d. For now let's assume the dimensionalities of the transform matrices are W Q ∈ R d×d , W K ∈ R d×d , and W V ∈ R d×d . Later we'll need separate dimensions for these matrices when we introduce multi-headed attention, so let's just make a note that we'll have a dimension d k for the key and query vectors, and a dimension d v for the value vectors, both of which for now we'll set to d. In the original transformer work (Vaswani et al., 2017) , d was 1024.