File size: 1,002 Bytes
76c0cfb 6038656 76c0cfb 6038656 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
---
language:
- fi
pipeline_tag: text-generation
---
Generative Pretrained Transformer with 8B parameteres for Finnish.
TurkuNLP Finnish GPT-3-models are a model family of pretrained monolingual GPT-style language models that are based on BLOOM-architecture.
Models are pure language models meaning that they are not [instruction finetuned](https://arxiv.org/abs/2203.02155) for dialogue or answering questions.
These models are intended to be used as foundational models that can be e.g. instruction finetuned to serve as modern chat-models.
**Parameters**
| Model | Layers | Dim | Heads | Params |
|--------|--------|------|-------|--------|
| Small | 12 | 768 | 12 | 186M |
| Medium | 24 | 1024 | 16 | 437M |
| Large | 24 | 1536 | 16 | 881M |
| XL | 24 | 2064 | 24 | 1.5B |
| ”2.7B” | 32 | 2560 | 32 | 2.8B |
| ”6.7B” | 32 | 4096 | 32 | 7.5B |
| "13B" | 40 | 5120 | 40 | 13.3B |
More documentation coming soon!
|