Note Useful to have more insights about the tokenizer trained and used for GPT-2, which is a modified BPE as defined in this paper. Additionally, it's implemented in `tiktoken` at https://github.com/openai/tiktoken

Note Attention is explained after being introduced at https://arxiv.org/abs/1409.0473, this paper proposed an Encoder-Decoder architecture, the Transformer. The whole architecture is interesting, but we'll transition into Decoder-only architectures for GPT.

Note Is interesting even though not comparable to GPT-2, as it's build using the encoder blocks and it's not auto-regressive in nature, but adds context on both sides of a word to achieve better results. But nice to read before GPT-2 to understand the differences and why it's been a relevant architecture.

Note Introduces the concept of Decoder-only architectures, which is later on adopted by GPT.

openai-community/gpt2

Text Generation • Updated Feb 19 • 10.5M • • 2.46k

Note * GPT: "Improving Language Understanding by Generative Pre-Training" at https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf * GPT-2: "Language Models are Unsupervised Multitask Learners" at https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf