---
language:
- vi
library_name: peft
pipeline_tag: question-answering
base_model: VietAI/gpt-neo-1.3B-vietnamese-news
---

## Information

- Model Pretrain: VietAI/gpt-neo-1.3B-vietnamese-news
## Training procedure

batch_size: int = 128

micro_batch_size: int = 2

num_epochs: int = 1

learning_rate: float = 3e-4

cutoff_len: int = 256

val_set_size: int = 0

### Lora hyperparams

lora_r: int = 6

lora_alpha: int = 12

lora_dropout: float = 0.05

lora_target_modules: str = "q_proj k_proj v_proj" 

### Framework versions

- transformer 4.32.1
- pytorch
- PEFT 0.5.0

## How to use

```python
BASE_MODEL = "VietAI/gpt-neo-1.3B-vietnamese-news"
PEFT_WEIGHTS = "NghiemAbe/QAvietnamese"

model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, PEFT_WEIGHTS, torch_dtype=torch.bfloat16)
```