LLäMmlein: Compact and Competitive German-Only Language Models from Scratch
Abstract
We create two German-only decoder models, LL\"aMmlein 120M and 1B, transparently from scratch and publish them, along with the training data, for the German NLP research community to use. The model training involved several key steps, including extensive data preprocessing, the creation of a custom German tokenizer, the training itself, as well as the evaluation of the final models on various benchmarks. Throughout the training process, multiple checkpoints were saved and analyzed using the SuperGLEBer benchmark to monitor the models' learning dynamics. Compared to state-of-the-art models on the SuperGLEBer benchmark, both LL\"aMmlein models performed competitively, consistently matching or surpassing models with similar parameter sizes. The results show that the models' quality scales with size as expected, but performance improvements on some tasks plateaued early, offering valuable insights into resource allocation for future model development.
Community
Very interesting new German LLM!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- EuroLLM: Multilingual Language Models for Europe (2024)
- SAG: Style-Aligned Article Generation via Model Collaboration (2024)
- Multilingual Pretraining Using a Large Corpus Machine-Translated from a Single Source Language (2024)
- Xmodel-1.5: An 1B-scale Multilingual LLM (2024)
- Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 11
Browse 11 models citing this paperDatasets citing this paper 0
No dataset linking this paper