File size: 435 Bytes
779316b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
---
datasets:
- hlillemark/c4_t5_corrupted_seqlen256
language:
- en
metrics:
- perplexity
---
|Hyperparameter |Value |
|---------------------|---------|
|Steps | 150k|
|Max length | 256|
|LR | 1e-4|
|LR schedule | constant|
|Optimizer | AdamW|
|beta_1, beta_2 |0.9, 0.95|
|Final eval loss | 2.245|
|Final eval perplexity| 9.44|
|