File size: 3,408 Bytes
30f6c8c fbe6327 30f6c8c 7ea2327 30f6c8c fbe6327 30f6c8c fbe6327 30f6c8c fbe6327 30f6c8c fbe6327 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 |
---
language:
- en
license: apache-2.0
datasets:
- tatsu-lab/alpaca
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
- sft
base_model: unsloth/llama-3-8b-Instruct-bnb-4bit
---
## lxyuan/llama-3-8b-Instruct-lora-merged
**Model Description**: Finetuned the [Llama-3-8B-Instruct Model](https://huggingface.co./unsloth/llama-3-8b-Instruct-bnb-4bit) using [unsloth](https://github.com/unslothai/unsloth)
on [Alpaca Dataset](https://huggingface.co./datasets/tatsu-lab/alpaca) for 1000 steps.
- **Developed by:** lxyuan
- **License:** apache-2.0
- **Finetuned from model :** unsloth/llama-3-8b-Instruct-bnb-4bit
- **Finetuned from model :** tatsu-lab/alpaca
## Installation
```python
import torch
major_version, minor_version = torch.cuda.get_device_capability()
# Must install separately since Colab has torch 2.2.1, which breaks packages
!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
if major_version >= 8:
# Use this for new GPUs like Ampere, Hopper GPUs (RTX 30xx, RTX 40xx, A100, H100, L40)
!pip install --no-deps packaging ninja einops flash-attn xformers trl peft accelerate bitsandbytes
else:
# Use this for older GPUs (V100, Tesla T4, RTX 20xx)
!pip install --no-deps xformers trl peft accelerate bitsandbytes
```
## Inference example
```python
from transformers import pipeline
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "lxyuan/llama-3-8b-Instruct-lora-merged",
dtype = None, # auto detect
load_in_4bit = True, # default is True
)
FastLanguageModel.for_inference(model) # Enable native 2x faster inference
pipeline = pipeline("text-generation", model=model, tokenizer=tokenizer)
messages = [
{"role": "system", "content": "You are helpful AI bot that follows instruction to complete task."},
{"role": "user", "content": "Write me 10 sentences that end with 'apple"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
terminators = [
tokenizer.eos_token_id,
tokenizer.convert_tokens_to_ids("<|eot_id|>")
]
outputs = pipeline(
prompt,
max_new_tokens=512,
eos_token_id=terminators,
do_sample=True,
temperature=0.6,
top_p=0.9,
)
print(outputs[0]["generated_text"])
```
#### Inference Output
```markdown
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are helpful AI bot that follows instruction to complete task.<|eot_id|><|start_header_id|>user<|end_header_id|>
Write me 10 sentences that end with 'apple<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Here are 10 sentences that end with the word "apple":
1. The farmer grew a juicy red apple.
2. She ate a crunchy green apple.
3. The tree bore a ripe yellow apple.
4. He bit into a sweet Granny Smith apple.
5. The basket was filled with fresh apples.
6. The juice was squeezed from a ripe red apple.
7. She picked a perfect autumn apple.
8. The pie was filled with tender Granny Smith apple.
9. The farmer's market sold a variety of apples.
10. The snack was a crisp, juicy apple.
```
## Training procedure
- [Finetuning notebook](https://github.com/LxYuan0420/nlp/blob/main/notebooks/Lora_finetuning_Llama_3_8b_Instruct_with_Alpaca.ipynb)
- [Original Notebook from unsloth](https://colab.research.google.com/drive/135ced7oHytdxu3N2DNe1Z0kqjyYIkDXp?usp=sharing#scrollTo=MKX_XKs_BNZR)
|