kr-manish/Llama-2-7b-chat-fine-tune-text-to-python

#dataset-used: codeparrot/xlcost-text-to-code

#github notebook: https://github.com/manishzed/LLM-Fine-tune/blob/main/Llama_2_7b_chat_fine_tune_text_to_python.ipynb

#code

#testing and loading model

import torch, gc
gc.collect()
torch.cuda.empty_cache()

import numpy as np
import pandas as pd
import os
from tqdm import tqdm
import bitsandbytes as bnb
import torch
import torch.nn as nn
import transformers
from datasets import Dataset
from peft import LoraConfig, PeftConfig
from trl import SFTTrainer
from transformers import (AutoModelForCausalLM,
                          AutoTokenizer,
                          BitsAndBytesConfig,
                          TrainingArguments,
                          pipeline,
                          logging)
from sklearn.metrics import (accuracy_score,
                             classification_report,
                             confusion_matrix)
from sklearn.model_selection import train_test_split

from datasets import load_dataset

#testing----1


# Ruta del modelo guardado en el dataset de Kaggle
from peft import LoraConfig, PeftModel

device_map = {"": 0}
PEFT_MODEL = "kr-manish/Llama-2-7b-chat-fine-tune-text-to-python"
#model_name = "NousResearch/Llama-2-7b-hf"

# Cargar la configuración del modelo
config = PeftConfig.from_pretrained(PEFT_MODEL)

# Cargar el modelo
model = AutoModelForCausalLM.from_pretrained(
    config.base_model_name_or_path,
    low_cpu_mem_usage=True,
    return_dict=True,
    #quantization_config=bnb_config,
    device_map="auto",
    #trust_remote_code=True,
    torch_dtype=torch.float16,
)

# Cargar el tokenizador
tokenizer=AutoTokenizer.from_pretrained(config.base_model_name_or_path)
tokenizer.pad_token = tokenizer.eos_token

# Cargar el modelo PEFT
load_model = PeftModel.from_pretrained(model, PEFT_MODEL)

input_text ="Program to convert Centimeters to Pixels | Function to convert centimeters to pixels ; Driver Code"
prompt_test = input_text
pipe_test = pipeline(task="text-generation",
                model=load_model,
                tokenizer=tokenizer,
                max_length =200,
                #max_new_tokens =25,
                )
#result_test = pipe_test(prompt_test)
#answer_test = result_test[0]['generated_text']
#answer_test
#or
result = pipe_test(f"<s>[INST] {input_text} [/INST]")
print(result[0]['generated_text'])

#Program to convert Centimeters to Pixels | Function to convert centimeters to pixels ; Driver Code [/code] def convertCentimetersToPixels ( cm ) : NEW_LINE INDENT pixels =

#code

results

This model is a fine-tuned version of NousResearch/Llama-2-7b-chat-hf on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.7746

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 1
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.7836	1.0	463	0.7746

Framework versions

• PEFT 0.10.0
• Transformers 4.38.2
• Pytorch 2.2.1+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2