license: apache-2.0

a LoRA fine-tuned model with Japanese dataset

LoRA Experiment

rwkv-7b-jp-280.pth is merged model with base

Base Model

https://huggingface.co./BlinkDL/rwkv-4-raven/tree/main

RWKV-4-Raven-7B-v10-Eng89%25-Jpn10%25-Other1%25-20230420-ctx4096.pth

Parameters:
Lora Rank 1024
Lora Alpha 2048
ctx length 1024
Lora Size: almost 1.6GB


Dataset 

https://huggingface.co./datasets/kunishou/hh-rlhf-49k-ja

https://huggingface.co./datasets/kunishou/databricks-dolly-15k-ja