Transformers
GGUF
Inference Endpoints
conversational
aashish1904 commited on
Commit
f0304d6
1 Parent(s): ecfdb16

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +98 -0
README.md ADDED
@@ -0,0 +1,98 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ ---
3
+
4
+ library_name: transformers
5
+ license: cc-by-4.0
6
+ datasets:
7
+ - uonlp/CulturaX
8
+
9
+ ---
10
+
11
+ [![QuantFactory Banner](https://lh7-rt.googleusercontent.com/docsz/AD_4nXeiuCm7c8lEwEJuRey9kiVZsRn2W-b4pWlu3-X534V3YmVuVc2ZL-NXg2RkzSOOS2JXGHutDuyyNAUtdJI65jGTo8jT9Y99tMi4H4MqL44Uc5QKG77B0d6-JfIkZHFaUA71-RtjyYZWVIhqsNZcx8-OMaA?key=xt3VSDoCbmTY7o-cwwOFwQ)](https://hf.co/QuantFactory)
12
+
13
+
14
+ # QuantFactory/lola_v1-GGUF
15
+ This is quantized version of [dice-research/lola_v1](https://huggingface.co/dice-research/lola_v1) created using llama.cpp
16
+
17
+ # Original Model Card
18
+
19
+
20
+ <!-- Provide a quick summary of what the model is/does. -->
21
+ # LOLA &mdash; An Open-Source Massively Multilingual Large Language Model
22
+
23
+ ## Abstract
24
+
25
+ LOLA is a massively multilingual large language model trained on more than 160 languages using a sparse Mixture-of-Experts Transformer architecture. Our architectural and implementation choices address the challenge of harnessing linguistic diversity while maintaining efficiency and avoiding the common pitfalls of multilinguality. Our analysis of the evaluation results shows competitive performance in natural language generation and understanding tasks. Additionally, we demonstrate how the learned expert-routing mechanism exploits implicit phylogenetic linguistic patterns to potentially alleviate the curse of multilinguality. We provide an in-depth look at the training process, an analysis of the datasets, and a balanced exploration of the model's strengths and limitations. As an open-source model, LOLA promotes reproducibility and serves as a robust foundation for future research. Our findings enable the development of compute-efficient multilingual models with strong, scalable performance across languages.
26
+
27
+ Paper: https://arxiv.org/abs/2409.11272
28
+
29
+ ## Model Description
30
+
31
+ - **Developed by:** DICE Research Group (https://dice-research.org/) @ Paderborn University (https://www.uni-paderborn.de/)
32
+ - **Model type:** GPT2 style (decoder-only) with alternating sparse Mixture-of-Experts layers
33
+ - **Number of Experts**: 16
34
+ - **Model Size**: 1.3 Billion (active*) / 7.4 Billion (total)
35
+ - **Language(s) (NLP):** 160+
36
+ - **License:** CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
37
+ - **Repository:** https://github.com/dice-group/LOLA
38
+
39
+ <sub>* The number of parameters a model utilizes per token (ref: [Fedus et al, 2022](https://arxiv.org/abs/2101.03961); [Du et al, 2022](https://arxiv.org/abs/2112.06905)). This distinction is crucial for understanding the efficiency and performance of MoE models.</sub>
40
+
41
+ ## How to Get Started with the Model
42
+
43
+ This pre-trained (causal language modeling) model can only be used for text-generation and requires further fine-tuning on downstream tasks.
44
+
45
+ ### How to use
46
+
47
+ You can use this model directly with a pipeline for text generation.
48
+
49
+ ```python
50
+ >>> from transformers import pipeline
51
+
52
+ >>> generator = pipeline('text-generation', model="dice-research/lola_v1", trust_remote_code=True)
53
+ >>> generator("The quick brown fox", max_length=13)
54
+ [{'generated_text': 'The quick brown fox jumps over the lazy dog.'}]
55
+ ```
56
+
57
+ To use the top-k sampling, please set `do_sample` to `True`.
58
+
59
+ **Note:** The tokenizer used in the model comes from mGPT (https://github.com/ai-forever/mgpt)
60
+
61
+
62
+ ## Training Details
63
+
64
+
65
+ ### Training Framework
66
+
67
+ - DeepSpeed Megatron (https://github.com/microsoft/Megatron-DeepSpeed)
68
+ - Architecture type: Transformers (Decoder-only) with Mixture-of-Experts (MoE)
69
+ - Number of Experts: 16
70
+ - Model Size: 1.3 Billion Dense / 7.4 Billion Sparse
71
+
72
+ ### Pretraining Dataset
73
+
74
+ - CulturaX (https://huggingface.co/datasets/uonlp/CulturaX)
75
+ - Total Tokens: 6.3 Trillion
76
+ - Total Languages: 167
77
+
78
+ ### LOLA v1 Training:
79
+
80
+ - Computing cluster: Noctua2 (https://pc2.uni-paderborn.de/hpc-services/available-systems/noctua2)
81
+ - Number of GPUs: 96x Nvidia A100 (40GB)
82
+ - Training steps: 296000
83
+ - Tokens consumed: 465 Billion
84
+ - Training time: ~19 days
85
+
86
+ ## Citation
87
+ If you use our work in your research, please make sure to cite it:
88
+ ```bibtex
89
+ @misc{srivastava2024lolaopensourcemassively,
90
+ title={LOLA -- An Open-Source Massively Multilingual Large Language Model},
91
+ author={Nikit Srivastava and Denis Kuchelev and Tatiana Moteu Ngoli and Kshitij Shetty and Michael Roeder and Diego Moussallem and Hamada Zahera and Axel-Cyrille Ngonga Ngomo},
92
+ year={2024},
93
+ eprint={2409.11272},
94
+ archivePrefix={arXiv},
95
+ primaryClass={cs.CL},
96
+ url={https://arxiv.org/abs/2409.11272},
97
+ }
98
+ ```