| license: apache-2.0 | |
| a LoRA fine-tuned model with Japanese dataset | |
| LoRA Experiment | |
| rwkv-7b-jp-280.pth is merged model with base | |
| Base Model | |
| https://huggingface.co./BlinkDL/rwkv-4-raven/tree/main | |
| RWKV-4-Raven-7B-v10-Eng89%25-Jpn10%25-Other1%25-20230420-ctx4096.pth | |
| Parameters: | |
| Lora Rank 1024 | |
| Lora Alpha 2048 | |
| ctx length 1024 | |
| Lora Size: almost 1.6GB | |
| Dataset | |
| https://huggingface.co./datasets/kunishou/hh-rlhf-49k-ja | |
| https://huggingface.co./datasets/kunishou/databricks-dolly-15k-ja | |