LoftQ Initialization

| Paper | Code | PEFT Example |

LoftQ (LoRA-fine-tuning-aware Quantization) provides a quantized backbone Q and LoRA adapters A and B, given a full-precision pre-trained weight W.

This model, Meta-Llama-3-70B-4bit-64rank-1iter, is obtained from LLAMA-3-70B. The backbone is under LoftQ/Meta-Llama-3-70B-4bit-64rank-1iter and LoRA adapters are under the subfolder='loftq_init'.

Model Info

Backbone

  • Size: ~ 43 GiB
  • Loaded format: bitsandbytes nf4
  • Size loaded on GPU: ~43 GiB

LoRA adapters

  • rank: 64
  • lora_alpha: 16
  • target_modules: ["down_proj", "up_proj", "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj"]

Usage

Training. Here's an example of loading this model and preparing for the LoRA fine-tuning.

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

MODEL_ID = "LoftQ/Meta-Llama-3-70B-4bit-64rank-1iter"

base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, 
    torch_dtype=torch.bfloat16,  # you may change it with different models
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.bfloat16,  # bfloat16 is recommended
        bnb_4bit_use_double_quant=False,
        bnb_4bit_quant_type='nf4',
    ),
)
peft_model = PeftModel.from_pretrained(
    base_model,
    MODEL_ID,
    subfolder="loftq_init",
    is_trainable=True,
)

# Do training with peft_model ...

Inference. Here is an example code for inference after the model has been fine-tuned.

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

MODEL_ID = "LoftQ/Meta-Llama-3-70B-4bit-64rank-1iter"
ADAPTER_PATH = "you/adapter/path"

base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, 
    torch_dtype=torch.bfloat16,  # you may change it with different models
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.bfloat16,  # bfloat16 is recommended
        bnb_4bit_use_double_quant=False,
        bnb_4bit_quant_type='nf4',
    ),
)
peft_model = PeftModel.from_pretrained(
    base_model,
    ADAPTER_PATH,
)

# Do inference with peft_model ...

See the full code at our Github Repo

Citation

@article{li2023loftq,
  title={Loftq: Lora-fine-tuning-aware quantization for large language models},
  author={Li, Yixiao and Yu, Yifan and Liang, Chen and He, Pengcheng and Karampatziakis, Nikos and Chen, Weizhu and Zhao, Tuo},
  journal={arXiv preprint arXiv:2310.08659},
  year={2023}
}
Downloads last month
13
Safetensors
Model size
37.4B params
Tensor type
F32
BF16
U8
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.