license: apache-2.0
datasets:
- iamtarun/python_code_instructions_18k_alpaca
language:
- en
library_name: peft
pipeline_tag: text2text-generation
tags:
- code
Here's a brief description of my project.
Table of Contents
Introduction
colab_code_generator_FT_code_gen_UT, an instruction-following large language model trained on the Google Colab Pro with T4 GPU and fine-tuned on 'Salesforce/codegen-350M-mono' that is licensed for commercial use. Code Generator_UT is trained on ~19k instructions/response fine-tuning records from 'iamtarun/python_code_instructions_18k_alpaca'.
Getting Started
Loading the fine-tuned Code Generator
from peft import AutoPeftModelForCausalLM>
test_model_UT = AutoPeftModelForCausalLM.from_pretrained("01GangaPutraBheeshma/colab_code_generator_FT_code_gen_UT")
test_tokenizer_UT = AutoTokenizer.from_pretrained("01GangaPutraBheeshma/colab_code_generator_FT_code_gen_UT")
Documentation
This model was fine-tuned using LoRA because I wanted the model's weights to be efficient in solving other types of Python problems(Ones that were not included in the training data). Setting lora_alpha to 16 suggests that I chose a relatively strong regularization. The specific value of this hyperparameter often requires experimentation and tuning to find the optimal balance between preventing overfitting and allowing the model to capture important patterns in the data.
The lora_dropout rate is 0.1, which dropped out 10% of the neurons randomly during training. This helps to prevent overfitting by introducing a level of randomness and redundancy in the network. 'r' in LoRa represents a rank which helps to decide the level of representation of the model in terms of number of dimensions or features. This proved to be advantageous for tasks like fine-tuning, where reducing the complexity of the model while preserving information is paramount.