File size: 908 Bytes
32dfe89 ea9357c 32dfe89 ea9357c b19f02f ea9357c cb9f38f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
---
license: apache-2.0
datasets:
- TigerResearch/pretrain_zh
language:
- zh
pipeline_tag: text-generation
---
based [gpt2](https://huggingface.co./gpt2)
trained with [LLaMA-Efficient-Tuning](https://github.com/hiyouga/LLaMA-Efficient-Tuning)
```sh
python src/train_bash.py \
--stage pt \
--model_name_or_path gpt2 \
--do_train \
--finetuning_type full \
--dataset tiger_pretrain_zh \
--template ziya \
--use_fast_tokenizer \
--preprocessing_num_workers 64 \
--per_device_train_batch_size 8 \
--gradient_accumulation_steps 16 \
--output_dir .cache/gpt2-tigerResearch_pretrain_zh \
--lr_scheduler_type cosine \
--logging_steps 10 \
--save_steps 1000 \
--eval_steps 500 \
--learning_rate 5e-5 \
--num_train_epochs 1.0 \
--val_size 3000 \
--evaluation_strategy steps \
--plot_loss \
--max_source_length 1024 \
--bf16
```
|