license: apache-2.0
model-index:
- name: llama-2-7b-miniguanaco
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: AI2 Reasoning Challenge (25-Shot)
type: ai2_arc
config: ARC-Challenge
split: test
args:
num_few_shot: 25
metrics:
- type: acc_norm
value: 50
name: normalized accuracy
source:
url: >-
https://huggingface.co./spaces/HuggingFaceH4/open_llm_leaderboard?query=decruz07/llama-2-7b-miniguanaco
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: HellaSwag (10-Shot)
type: hellaswag
split: validation
args:
num_few_shot: 10
metrics:
- type: acc_norm
value: 76.96
name: normalized accuracy
source:
url: >-
https://huggingface.co./spaces/HuggingFaceH4/open_llm_leaderboard?query=decruz07/llama-2-7b-miniguanaco
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU (5-Shot)
type: cais/mmlu
config: all
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 48.05
name: accuracy
source:
url: >-
https://huggingface.co./spaces/HuggingFaceH4/open_llm_leaderboard?query=decruz07/llama-2-7b-miniguanaco
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: TruthfulQA (0-shot)
type: truthful_qa
config: multiple_choice
split: validation
args:
num_few_shot: 0
metrics:
- type: mc2
value: 42.84
source:
url: >-
https://huggingface.co./spaces/HuggingFaceH4/open_llm_leaderboard?query=decruz07/llama-2-7b-miniguanaco
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: Winogrande (5-shot)
type: winogrande
config: winogrande_xl
split: validation
args:
num_few_shot: 5
metrics:
- type: acc
value: 73.48
name: accuracy
source:
url: >-
https://huggingface.co./spaces/HuggingFaceH4/open_llm_leaderboard?query=decruz07/llama-2-7b-miniguanaco
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GSM8k (5-shot)
type: gsm8k
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 19.11
name: accuracy
source:
url: >-
https://huggingface.co./spaces/HuggingFaceH4/open_llm_leaderboard?query=decruz07/llama-2-7b-miniguanaco
name: Open LLM Leaderboard
llama-2-7b-miniguanaco
This is my first model, with LLama-2-7b model finetuned with miniguanaco datasets.
This is a simple finetune based off a Google Colab notebook. Finetune instructions were from Labonne's first tutorial.
To run it: import torch from transformers import AutoTokenizer, AutoModelForCausalLM import math
model_path = "decruz07/llama-2-7b-miniguanaco"
tokenizer = AutoTokenizer.from_pretrained(model_path, use_default_system_prompt=False) model = AutoModelForCausalLM.from_pretrained( model_path, torch_dtype=torch.float32, device_map='auto',local_files_only=False, load_in_4bit=True ) print(model) prompt = input("please input prompt:") while len(prompt) > 0: input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")
generation_output = model.generate( input_ids=input_ids, max_new_tokens=500,repetition_penalty=1.2 ) print(tokenizer.decode(generation_output[0])) prompt = input("please input prompt:")
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 51.74 |
| AI2 Reasoning Challenge (25-Shot) | 50.00 |
| HellaSwag (10-Shot) | 76.96 |
| MMLU (5-Shot) | 48.05 |
| TruthfulQA (0-shot) | 42.84 |
| Winogrande (5-shot) | 73.48 |
| GSM8k (5-shot) | 19.11 |