Hugging Face's logo

reissbaker
/
llama-3.1-70b-abliterated-lora

PEFT
Safetensors
Model card Files Community
1
llama-3.1-70b-abliterated-lora / README.md
reissbaker's picture
reissbaker
Add README
97e01d4
preview code
|
raw
history blame contribute delete
1.36 kB
metadata
base_model: meta-llama/Meta-Llama-3.1-70B-Instruct
library_name: peft

Model Card for Model ID

This LoRA adapter was extracted from mlabonne/Meta-Llama-3.1-70B-Instruct-lorablated and uses meta-llama/Llama-3.1-70B-Instruct as a base.

Model Details

The model was extracted by running mlabonne/harmful_behaviors and the user prompts (but not assistant responses or system messages) from Guilherme34/uncensor through the original abliterated model to generate a dataset of prompt/completion pairs, and was trained for 2 epochs on a 8xA100s with Axolotl using FSDP. Since the original abliterated model isn't perfect at avoiding refusals, the dataset was cleaned to remove the few refusals generated prior to training.

Model Description

  • Developed by: @reissbaker
  • Funded by: Synthetic Lab
  • License: Apache 2.0
  • Finetuned from model: Llama 3.1 70B Instruct

How to Get Started with the Model

Run the model with one click on glhf.chat.

Training Hyperparameters

  • BF16 mixed-precision
  • 4e-4 LR
  • Linear LR schedule
  • Fused AdamW optimizer