Machine translation systems generally use a fixed vocabulary, A common way to generate this vocabulary is with the BPE or wordpiece algorithms sketched in Chapwordpiece ter 2. Generally a shared vocabulary is used for the source and target languages, which makes it easy to copy tokens (like names) from source to target, so we build the wordpiece/BPE lexicon on a corpus that contains both source and target language data. Wordpieces use a special symbol at the beginning of each token; here's a resulting tokenization from the Google MT system (Wu et al., 2016):