metadata
library_name: transformers
tags:
- mergekit
- merge
niro is an improvement over the excellent WizardLM-Evol-V2-Unfiltered model, which at the time of writting is one of the best 1.8 billion parameters models. Keep in mind, nero is an un-trained merge, further improvements are to be released later.
benchmarks
zero-shot evaluations performed on current sota ~1.8b models.
| Parameters | Model | MMLU | ARC | HellaSwag | PIQA | Winogrande | Average |
|---|---|---|---|---|---|---|---|
| 1.7b | smollm | 27.65 | 46.26 | 65.74 | 76.06 | 60.93 | 55.33 |
| 1.8B | danube | 40.29 | 40.02 | 72.40 | 77.04 | 64.25 | 58.80 |
| 1.8B | wizard | 40.79 | 40.87 | 71.85 | 78.02 | 64.33 | 59.17 |
| 1.8B | niro | 41.75 | 40.96 | 72.07 | 77.97 | 65.51 | 59.65 |