This contextualization is accomplished through the use of the same self-attention mechanism used in causal models. As with these models, the first step is to generate a set of key, query and value embeddings for each element of the input vector x through the use of learned weight matrices W Q , W K , and W V . These weights project each input vector x i into its specific role as a key, query, or value.