deci7b-ft-lora-sql-v2adapters

README.md

@@ -20,7 +20,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [Deci/DeciLM-7B](https://huggingface.co/Deci/DeciLM-7B) on the generator dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.8011
+- Loss: 0.3593
 
 ## Model description
 
@@ -39,26 +39,36 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 0.0002
-- train_batch_size: 4
+- learning_rate: 0.0003
+- train_batch_size: 8
 - eval_batch_size: 8
 - seed: 1399
-- gradient_accumulation_steps: 8
+- gradient_accumulation_steps: 4
 - total_train_batch_size: 32
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: constant
-- lr_scheduler_warmup_steps: 10
-- training_steps: 100
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 100
+- training_steps: 500
 
 ### Training results
 
 | Training Loss | Epoch | Step | Validation Loss |
 |:-------------:|:-----:|:----:|:---------------:|
-| 0.9088        | 0.87  | 5    | 0.7531          |
-| 0.6954        | 1.74  | 10   | 0.6887          |
-| 0.5878        | 2.61  | 15   | 0.6839          |
-| 0.4975        | 3.48  | 20   | 0.7174          |
-| 0.4174        | 4.35  | 25   | 0.8011          |
+| 0.7841        | 0.06  | 20   | 0.5204          |
+| 0.4837        | 0.11  | 40   | 0.4376          |
+| 0.4325        | 0.17  | 60   | 0.4122          |
+| 0.4112        | 0.23  | 80   | 0.4029          |
+| 0.4028        | 0.28  | 100  | 0.3925          |
+| 0.3958        | 0.34  | 120  | 0.3855          |
+| 0.3895        | 0.4   | 140  | 0.3816          |
+| 0.3818        | 0.45  | 160  | 0.3784          |
+| 0.3753        | 0.51  | 180  | 0.3756          |
+| 0.3722        | 0.56  | 200  | 0.3734          |
+| 0.3687        | 0.62  | 220  | 0.3702          |
+| 0.3678        | 0.68  | 240  | 0.3665          |
+| 0.3636        | 0.73  | 260  | 0.3627          |
+| 0.3582        | 0.79  | 280  | 0.3580          |
+| 0.3594        | 0.85  | 300  | 0.3593          |
 
 
 ### Framework versions

adapter_config.json

@@ -10,22 +10,22 @@
   "layers_to_transform": null,
   "loftq_config": {},
   "lora_alpha": 32,
-  "lora_dropout": 0.1,
+  "lora_dropout": 0.05,
   "megatron_config": null,
   "megatron_core": "megatron.core",
   "modules_to_save": null,
   "peft_type": "LORA",
-  "r": 8,
+  "r": 16,
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "o_proj",
     "q_proj",
-    "down_proj",
-    "up_proj",
+    "o_proj",
     "v_proj",
+    "k_proj",
     "gate_proj",
-    "k_proj"
+    "up_proj",
+    "down_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c893f96a909179fe3e5917354d665e787f5533e2f909477abc8d522cee862fac
+oid sha256:1344f6bb27d1611316b75829667428168abae25859c5eac7150e22e0e3ea85c5
 size 4920