• A subword vocabulary consisting of 30,000 tokens generated using the Word-Piece algorithm (Schuster and Nakajima, 2012) , • Hidden layers of size of 768, • 12 layers of transformer blocks, with 12 multihead attention layers each.