--- library_name: transformers tags: - trl - sft license: apache-2.0 language: - en base_model: - meta-llama/Llama-3.1-8B-Instruct pipeline_tag: text-generation --- ----------------------------------------------------------------------------------------------------- **Remeber this model is for illustration and knowlwdge Purpose. I have only used online freely available materials in whole process.** ## Model Details This Model is Trained on Custum data related to Sales interactive conversations as Array of objects having Instruction and Response as Keys. -**Parameters:** ~8 Billion -**Quantization:** 4 Bit (Q-LORA) ### Model Description This is the model card of a 🤗 transformers model that has been pushed on the Hub. - **Trained by:** [vakodiya] [Viru Akodiya] - **Model type:** [Text-Generation] - **License:** [apache-2.0] - **Finetuned from model:** [meta-llama/Llama-3.1-8B-Instruct] ### Training Data Training Data is specifically generated by me to train to my use case. It consits of Just 500 examples, so to increase dataset size, duplicated the original data and makes it 1000. #### Training Hyperparameters - **Hardware Type:** [Kaggle's GPU T4X2] - **Time used:** [37 Minutes] - **Cloud Provider:** [Kaggle] ----------------------------------------------------------------------------------------------------------- ## INFERENCE (It will need GPU) ------------------------------------------------------------------------------------------------------------ # Install Dependencies ``` %%capture !pip install transformers accelerate bitsandbytes ``` ``` from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig, pipeline, AutoConfig import torch ``` --------------------------------------------------------------------------------------------------------- # Load model and Tokenizer ``` model_name = "vakodiya/Llama-3-8B-instruct-4bit-salesbot" config = AutoConfig.from_pretrained(model_name) bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, ) tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained( model_name, quantization_config=bnb_config, device_map="auto", torch_dtype=torch.bfloat16, ) # Model evaluation mode model.eval() ``` ------------------------------------------------------------------------------------------------------- # Creating Inference Point ``` def Trained_Llama3_1_inference(prompt): model.eval() conversation=[ {"role": "user", "content": prompt}, ] input_ids = tokenizer.apply_chat_template(conversation, add_generation_prompt=True, return_tensors="pt", padding=True, truncation=True, return_attention_mask=True) if input_ids.shape[1] > 8192: input_ids = input_ids[:, -8192:] return "Input tokens more than 8k" inputs = input_ids.to(model.device) attention_mask = torch.ones_like(inputs, dtype=torch.long) final_prompt=tokenizer.decode(inputs[0]) outputs = model.generate(inputs, max_new_tokens=256, temperature=0.4,attention_mask=attention_mask,pad_token_id=tokenizer.pad_token_id) response = tokenizer.decode(outputs[0]) final_response= response.replace(final_prompt,"").replace('<|eot_id|>',"") # Exclude prompt from response return final_response ``` ------------------------------------------------------------------------------------------------------------------------------- # Invoking Inference ``` Trained_Llama3_1_inference("What are qualities of good Sales-person ?") ``` ----------End of Inferece -------------------- ---------------------------------------------------------------------------------------------------------------------------------- ---------- Start of Training ----------------- #### Training (on Kaggle Notebook) This training is done on Kaggle Notebook enabling GPU(Required in quantized training/ inference). # Install Dependencies ``` %%capture !pip install -U transformers[torch] datasets !pip install -q bitsandbytes trl peft accelerate !pip install flash-attn --no-build-isolation !pip install huggingface_hub ``` ------------------------------------------------------------------------------------------------------------------------------------------ # Import Modules ``` from transformers import BitsAndBytesConfig, AutoTokenizer, AutoModelForCausalLM, TrainingArguments from trl import SFTTrainer from peft import LoraConfig from huggingface_hub import notebook_login import torch from huggingface_hub import login from datasets import Dataset from kaggle_secrets import UserSecretsClient import os ``` ------------------------------------------------------------------------------------------------------------------------------------------ # Remember to generate a Token with edit access on HuggingFace and add it as secret in Kaggle Notebook ``` hf_token = UserSecretsClient().get_secret("HF_TOKEN_LLAMA3") login(token = hf_token) os.environ["CUDA_VISIBLE_DEVICES"] = "0" # Use only GPU 0 ``` ------------------------------------------------------------------------------------------------------------------------------------------- # Remember to Customize your own data with at least 1000 examples. ``` Data_examples = [{"instruction":"Who has taken oath as Prime minister of India in 2024", "response":" Shri Narendra Modi has taken oath as Prime minister of india on 9th June 2024. He is now become prime minister having 3 consecutive terms."}, ...................................................................................,] ``` ------------------------------------------------------------------------------------------------------------------------------------------ # Process data to stringify only the `text` field ``` processed_data = [] for example in Data_examples : processed_data.append({'text':f"{example['instruction']} \n {example['response']}"}) # Create a Dataset from the list of dictionaries dataset = Dataset.from_list(processed_data) # Split into train and test Data sets dataset = dataset.train_test_split(test_size=0.01) # Access train and test splits train_dataset = dataset['train'] test_dataset = dataset['test'] ``` --------------------------------------------------------------------------------------------------------------------------------------- # Firstly add model to Kaggle notebook navigating to Add Input and Add LLama3.1 8 B in out Notebook ``` model_path="/kaggle/input/llama-3.1/transformers/8b-instruct/2" # Change it according to your model path in Notebook trained_model_name = "Llama-3-8B-instruct-4bit-finetuned" output_dir = 'kaggle/working/' + trained_model_name ``` ---------------------------------------------------------------------------------------------------------------------------------------- ## For 4 bit quantization (Q-LoRA) set Configs ``` quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16,) peft_config = LoraConfig( r=16, lora_alpha=16, lora_dropout=0.1, bias="none", task_type="CAUSAL_LM", target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], ) ``` ----------------------------------------------------------------------------------------------------------------------------------------- # Load the Model and Tokenizer and set pad token ``` tokenizer = AutoTokenizer.from_pretrained(model_path) model = AutoModelForCausalLM.from_pretrained( model_path, quantization_config=quantization_config, device_map="auto") # Use eos_token as pad_token tokenizer.pad_token = tokenizer.eos_token ``` ----------------------------------------------------------------------------------------------------------------------------------------- # Set Training configurations ``` training_args = TrainingArguments( fp16=False, # specify bf16=True instead when training on GPUs that support bf16 else fp16 bf16=True, do_eval=True, eval_strategy="epoch", gradient_accumulation_steps=4, gradient_checkpointing=True, gradient_checkpointing_kwargs={"use_reentrant": False}, learning_rate=2.0e-05, log_level="info", logging_steps=5, logging_strategy="steps", lr_scheduler_type="cosine", max_steps=-1, num_train_epochs=1, # Number of times training will go through with same dataset. output_dir=output_dir, overwrite_output_dir=True, per_device_eval_batch_size=8, # You can reduce if out-of memory errors occurs per_device_train_batch_size=8, # You can reduce if out-of memory errors occurs report_to="none", # for skipping wandb logging save_strategy="no", save_total_limit=None, ) ``` -------------------------------------------------------------------------------------------------------------------------------------------- # Set-up Trainer (Supervised-fine-tuning) ``` trainer = SFTTrainer( model=model, # Use above quantized model args=training_args, train_dataset=train_dataset, # If Training Fails Try to reduce Dataset Size eval_dataset=test_dataset, dataset_text_field="text", tokenizer=tokenizer, packing=False, # Setting it True will Reduce dataset size as it will exclude similar examples occuring repetitive peft_config=peft_config, max_seq_length=1024, ) ``` ------------------------------------------------------------------------------------------------------------------------------------------------- # Note: It may take long Time to train model (several minutes to Hours) depending on your dataset size ``` # To clear out cache for unsuccessful run torch.cuda.empty_cache() train_result = trainer.train() ``` ------------------------------------------------------------------------------------------------------------------------------------------------------ # Save model in Notebook (in output_directory) ``` trainer.save_model() ``` ------------------------------------------------------------------------------------------------------------------------------------------------------- # Merge LoRA with the base model and save the merged model ``` merged_model = trainer.model.merge_and_unload() merged_model.save_pretrained("merged_model",safe_serialization=True) tokenizer.save_pretrained("merged_model") ``` --------------------------------------------------------------------------------------------------------------------------------------------------------- # push merged model to the HuggingFace-hub (You must have logged in already) ``` merged_model.push_to_hub("username/model_name") tokenizer.push_to_hub("username/model_name") ``` ------------------- End of Training and uploading trained model on our huggingface Space ----------------------------------