---
library_name: transformers
tags:
- bert
- cramming
- NLU
license: apache-2.0
datasets:
- TucanoBR/GigaVerbo
language:
- pt
pipeline_tag: fill-mask
---

# crammed BERT Portuguese

<!-- Provide a quick summary of what the model is/does. -->

This is a BERT model trained for 24 hours on a single A6000 GPU. It follows the architecture described in "Cramming: Training a Language Model on a Single GPU in One Day".

To use this model, clone the code from my fork https://github.com/wilsonjr/cramming and `import cramming` before using the 🤗 transformers `AutoModel` (see below). 


## How to use


```python

import cramming # needed to load crammed model
from transformers import AutoModelForMaskedLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("wilsonmarciliojr/crammed-bert-portuguese")
model  = AutoModelForMaskedLM.from_pretrained("wilsonmarciliojr/crammed-bert-portuguese")

text = "Oi, eu sou um modelo <mask>."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

```

## Training Details

### Training Data & Config

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

- 30M entries from `TucanoBR/GigaVerbo`
- 107M sequences of length 128
- tokenizer: WordPiece
- vocab_size: 32768
- seq_length: 128
- include_cls_token_in_corpus: false
- include_sep_token_in_corpus: true

### Training Procedure


- **optim**:
  
  -  type: AdamW
  -  lr: 0.001
  -  betas:
    - 0.9
    - 0.98
  -  eps: 1.0e-12
  -  weight_decay: 0.01
  -  amsgrad: false
  -  fused: null
  - warmup_steps: 0
  - cooldown_steps: 0
  - steps: 900000
  - batch_size: 8192
  - gradient_clipping: 0.5
    
- **objective**:
  -  name: masked-lm
  -  mlm_probability: 0.25
  -  token_drop: 0.0


#### Training Hyperparameters

- num_transformer_layers: 16
- hidden_size: 768
-  intermed_size: 3072
-  hidden_dropout_prob: 0.1
-  norm: LayerNorm
-  norm_eps: 1.0e-12
-  norm_scheme: pre
-  nonlin: GELUglu
-  tie_weights: true
-  decoder_bias: false
-  sparse_prediction: 0.25
-  loss: cross-entropy

- **embedding**:  
  -  vocab_size: null
  -  pos_embedding: scaled-sinusoidal
  -  dropout_prob: 0.1
  -  pad_token_id: 0
  -  max_seq_length: 128
  -  embedding_dim: 768
  -  normalization: true
  -  stable_low_precision: false
    
- **attention**:
  -  type: self-attention
  -  causal_attention: false
  -  num_attention_heads: 12
  -  dropout_prob: 0.1
  -  skip_output_projection: false
  -  qkv_bias: false
  -  rotary_embedding: false
  -  seq_op_in_fp32: false
  -  sequence_op: torch-softmax
    
- **init**:

  -  type: normal
  -  std: 0.02
  
-  ffn_layer_frequency: 1
-  skip_head_transform: true
-  use_bias: false
 
-  **classification_head**:

  - pooler: avg
  -  include_ff_layer: true
  -  head_dim: 1024
  -  nonlin: Tanh
  -  classifier_dropout: 0.1

#### Speeds, Sizes, Times 

 - ~ 0.1674s per step (97886t/s)


## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

TBD