---
language:
- zh
- en
license: llama3
datasets:
- KaifengGGG/WenYanWen_English_Parallel
metrics:
- bleu
- chrf
- meteor
- bertscore
---

# Model Summary

Hanscripter is an instruction-tuned language model focused on translation classical Chinese (i.e WenYanwen 文言文) to English. 
Our [Github repo](https://github.com/Kaifeng-Gao/HanScripter).

- Base Model: Meta-Llama-3-8B-Instruct
- SFT Dataset: KaifengGGG/WenYanWen_English_Parallel
- Fine-tune Method: QLoRA

# Version

# Usage

# Fine-tuning Details
Below are detailed descriptions of the various parameters and technologies used.

## LoRA Parameters
- **lora_r**: 64
- **lora_alpha**: 16
- **lora_dropout**: 0.1  

## Quantization
The model uses Bitsandbytes for state-of-the-art model quantization, enhancing computational efficiency:
- **use_4bit**: `True` - Enables the use of 4-bit quantization.
- **bnb_4bit_compute_dtype**: "float16" - The datatype used for computation in quantized state.
- **bnb_4bit_quant_type**: "nf4" - Specifies the quantization type.
- **use_nested_quant**: `False` - Nested quantization is not used.

## Training Arguments
Settings for training the model are as follows:
- **num_train_epochs**: 10
- **fp16**: `False`
- **bf16**: `True` - Optimized for use with A100 GPUs, employing Brain Floating Point (bf16).
- **per_device_train_batch_size**: 2
- **per_device_eval_batch_size**: 2
- **gradient_accumulation_steps**: 4
- **gradient_checkpointing**: `True`
- **max_grad_norm**: 0.3
- **learning_rate**: 0.0002
- **weight_decay**: 0.001
- **optim**: "paged_adamw_32bit"
- **lr_scheduler_type**: "cosine"
- **max_steps**: -1
- **warmup_ratio**: 0.03
- **group_by_length**: `True`