metadata

library_name: peft
license: llama2
datasets:
  - izumi-lab/llm-japanese-dataset
language:
  - ja
pipeline_tag: text-generation

AIgroup-CVM-utokyohospital/Llama-2-70b-chat-4bit-japanese

This model is Llama-2-Chat 70B fine-tuned with a part of the following Japanese version of the alpaca dataset.

https://huggingface.co./datasets/izumi-lab/llm-japanese-dataset

10000 steps
batch_size = 4

Copyright Notice

This model is built on the copyright of Meta's LLaMA.

Users of this model must also agree to Meta's license below.

https://ai.meta.com/llama/

How to use

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3"
import torch
torch.cuda.empty_cache()
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, AutoConfig

# Load models
model_id = "meta-llama/Llama-2-70b-chat-hf"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)
config = AutoConfig.from_pretrained(model_id)
config.pretraining_tp = 1 #LLama-2-70bなら必要
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, 
                                             device_map="auto")

# Load weights
peft_name = "AIgroup-CVM-utokyohospital/Llama-2-70b-chat-4bit-japanese"
model_peft = PeftModel.from_pretrained(
    model, 
    peft_name, 
    device_map="auto"
)
model_peft.eval()

device = "cuda:0"

inputs = tokenizer(text, return_tensors="pt").to(device)

with torch.no_grad():
  outputs = model.generate(**inputs,
            temperature=0.0,
            repetition_penalty=1.00)
  print(tokenizer.decode(outputs[0], skip_special_tokens=True))
  outputs = model_peft.generate(**inputs,
            temperature=0.0,
            repetition_penalty=1.00)
  print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Sample Responses

Training procedure

The following bitsandbytes quantization config was used during training:

load_in_8bit: False
load_in_4bit: True
llm_int8_threshold: 6.0
llm_int8_skip_modules: None
llm_int8_enable_fp32_cpu_offload: False
llm_int8_has_fp16_weight: False
bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: True
bnb_4bit_compute_dtype: bfloat16

Framework versions

PEFT 0.4.0