Contextual embeddings are thus vectors representing some aspect of the meaning of a token in context. For example, given a sequence of input tokens x 1 , ..., x n , we can use the output vector y i from the final layer of the model as a representation of the meaning of token x i in the context of sentence x 1 , ..., x n . Or instead of just using the vector y i from the final layer of the model, it's common to compute a representation for x i by averaging the output tokens y i from each of the last four layers of the model.