File size: 917 Bytes
86f35d8 60e947a 86f35d8 60e947a 0ca321a 60e947a 72d9c90 60e947a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
---
license: cc-by-nc-4.0
language:
- ja
---
BaseModel: RWKV-4-Pile-14B-Instruct-test5-20230329-ctx4096.pth
URL: https://huggingface.co./BlinkDL/rwkv-4-pile-14b
Training Code:
https://github.com/Blealtan/RWKV-LM-LoRA
Dataset:
https://github.com/shi3z/alpaca_ja
Training Prompt:
python train.py \
--load_model RWKV-4-Pile-14B-Instruct-test5-20230329-ctx4096.pth \
--proj_dir out \
--data_file "train.npy" \
--data_type "numpy" \
--vocab_size 50277 \
--ctx_len 1024 \
--epoch_save 5 \
--epoch_count 100 \
--n_layer 40 \
--n_embd 5120 \
--epoch_steps 1000 --epoch_begin 0 --micro_bsz 1 --pre_ffn 0 --head_qk 0 --lr_init 1e-5 --lr_final 1e-5 --warmup_steps 0 --beta1 0.9 --beta2 0.999 --adam_eps 1e-8 --accelerator gpu --devices 1 --precision bf16 --strategy deepspeed_stage_2 --grad_cp 0 \
--lora --lora_r 8 --lora_alpha 32 --lora_dropout 0.01
Training Environment:
A6000x1 + XEON 256GB RAM
|