Another option, instead of using just the last token h n to represent the whole sequence, is to use some sort of pooling function of all the hidden states h i for each pooling word i in the sequence. For example, we can create a representation that pools all the n hidden states by taking their element-wise mean: