File size: 679 Bytes
0211520 5c146d3 0211520 5c146d3 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
---
license: cc-by-nc-4.0
language:
- pl
tags:
- llama
- qlora
---
This repo contains a qlora adapter for [Llama-2-7b](https://huggingface.co./meta-llama/Llama-2-7b-hf), trained on 1B tokens, only in Polish.
The training took 20 days on a single RTX 4090 with the following hyperparameters:
* context length: 4096
* batch_size: 128
* learning_rate: 0.0002, cosine with warmup
* lora_r: 64
* lora_alpha: 16
* lora_modules: all
* lora_dropout: 0.0
* weight_decay: 0.1
* max_grad_norm: 0.3
* double_quant, nf4
* optimizer: paged_adamw_32bit
This adapter allows the model to speak Polish more accurately than vanilla [Llama-2-7b](https://huggingface.co./meta-llama/Llama-2-7b-hf).
|