nelkh/pgd_lora_8bits_mistral_1_r4_e4_

README.md

@@ -16,7 +16,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [mistralai/Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.3707
+- Loss: 0.4218
 
 ## Model description
 
@@ -35,30 +35,24 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 1e-05
+- learning_rate: 0.0003
 - train_batch_size: 3
 - eval_batch_size: 3
 - seed: 42
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: linear
 - lr_scheduler_warmup_steps: 20
-- num_epochs: 10
+- num_epochs: 4
 - mixed_precision_training: Native AMP
 
 ### Training results
 
 | Training Loss | Epoch | Step | Validation Loss |
 |:-------------:|:-----:|:----:|:---------------:|
-| 2.2656        | 1.0   | 90   | 0.8020          |
-| 0.5399        | 2.0   | 180  | 0.4374          |
-| 0.4287        | 3.0   | 270  | 0.4115          |
-| 0.4052        | 4.0   | 360  | 0.3926          |
-| 0.3858        | 5.0   | 450  | 0.3802          |
-| 0.3756        | 6.0   | 540  | 0.3757          |
-| 0.3699        | 7.0   | 630  | 0.3717          |
-| 0.366         | 8.0   | 720  | 0.3712          |
-| 0.3626        | 9.0   | 810  | 0.3702          |
-| 0.3607        | 10.0  | 900  | 0.3707          |
+| 1.2146        | 1.0   | 90   | 0.4449          |
+| 0.4336        | 2.0   | 180  | 0.4277          |
+| 0.4216        | 3.0   | 270  | 0.4245          |
+| 0.4146        | 4.0   | 360  | 0.4218          |
 
 
 ### Framework versions

adapter_config.json

@@ -10,20 +10,18 @@
   "layers_pattern": null,
   "layers_to_transform": null,
   "loftq_config": {},
-  "lora_alpha": 32,
-  "lora_dropout": 0,
+  "lora_alpha": 16,
+  "lora_dropout": 0.1,
   "megatron_config": null,
   "megatron_core": "megatron.core",
   "modules_to_save": null,
   "peft_type": "LORA",
-  "r": 2,
+  "r": 4,
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
     "k_proj",
-    "q_proj",
-    "v_proj",
-    "o_proj"
+    "q_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f9991c1446cd255278b21f28a6e80fefe47faa0f5e7bb118c3058a89151601a0
-size 6849416
+oid sha256:6c6fdabb38a768a4d35be92a860b3ee964d242b4c66f10368dd9f8c7d574cad6
+size 6832592

runs/Jul24_10-10-27_19c3b9d08bc5/events.out.tfevents.1721815834.19c3b9d08bc5.30992.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:431e5e3050658fe18a52b94982bd80f1b58eb711fd4d671e51375c9f9db03291
+size 7640

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c77869c9678b3b7dfffd551ac7f37345217ba31b3d01986a4694956da1e7e210
+oid sha256:521736074d9c7f8dd468dfb97c0e1926b26a0af499487f5d0b2a017d360c089d
 size 5112