Model save

README.md

@@ -0,0 +1,77 @@
+---
+license: apache-2.0
+library_name: peft
+tags:
+- trl
+- dpo
+- llama-factory
+- generated_from_trainer
+base_model: mistralai/Mistral-7B-Instruct-v0.3
+model-index:
+- name: Mistral-7B-Instruct-v0.3-ORPO-SALT
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# Mistral-7B-Instruct-v0.3-ORPO-SALT
+
+This model is a fine-tuned version of [mistralai/Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.8434
+- Rewards/chosen: -0.0777
+- Rewards/rejected: -0.0988
+- Rewards/accuracies: 0.5691
+- Rewards/margins: 0.0210
+- Logps/rejected: -0.9877
+- Logps/chosen: -0.7773
+- Logits/rejected: -3.1073
+- Logits/chosen: -3.0834
+- Sft Loss: 0.7773
+- Odds Ratio Loss: 0.6614
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-06
+- train_batch_size: 2
+- eval_batch_size: 2
+- seed: 42
+- gradient_accumulation_steps: 8
+- total_train_batch_size: 16
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 0.1
+- num_epochs: 3.0
+
+### Training results
+
+| Training Loss | Epoch  | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | Sft Loss | Odds Ratio Loss |
+|:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|:--------:|:---------------:|
+| 0.8803        | 0.8082 | 500  | 0.8619          | -0.0796        | -0.0983          | 0.5655             | 0.0187          | -0.9834        | -0.7962      | -3.0746         | -3.0520       | 0.7962   | 0.6572          |
+| 0.7341        | 1.6165 | 1000 | 0.8450          | -0.0779        | -0.0980          | 0.5673             | 0.0201          | -0.9804        | -0.7795      | -3.1194         | -3.0960       | 0.7795   | 0.6550          |
+| 0.7691        | 2.4247 | 1500 | 0.8434          | -0.0777        | -0.0988          | 0.5691             | 0.0210          | -0.9877        | -0.7773      | -3.1073         | -3.0834       | 0.7773   | 0.6614          |
+
+
+### Framework versions
+
+- PEFT 0.10.0
+- Transformers 4.40.1
+- Pytorch 2.3.0
+- Datasets 2.19.0
+- Tokenizers 0.19.1

trainer_log.jsonl

@@ -151,3 +151,39 @@
 {"current_steps": 1490, "total_steps": 1854, "loss": 0.8511, "accuracy": 0.5687500238418579, "learning_rate": 4.607082849092523e-07, "epoch": 2.40856738735098, "percentage": 80.37, "elapsed_time": "7:34:24", "remaining_time": "1:51:00"}
 {"current_steps": 1500, "total_steps": 1854, "loss": 0.7691, "accuracy": 0.5625, "learning_rate": 4.3649635614901405e-07, "epoch": 2.4247322691452817, "percentage": 80.91, "elapsed_time": "7:37:24", "remaining_time": "1:47:56"}
 {"current_steps": 1500, "total_steps": 1854, "eval_loss": 0.8434417247772217, "epoch": 2.4247322691452817, "percentage": 80.91, "elapsed_time": "7:43:36", "remaining_time": "1:49:24"}
+{"current_steps": 1510, "total_steps": 1854, "loss": 0.7959, "accuracy": 0.6000000238418579, "learning_rate": 4.128769732701973e-07, "epoch": 2.4408971509395836, "percentage": 81.45, "elapsed_time": "7:46:37", "remaining_time": "1:46:18"}
+{"current_steps": 1520, "total_steps": 1854, "loss": 0.7949, "accuracy": 0.543749988079071, "learning_rate": 3.8985691870233046e-07, "epoch": 2.4570620327338855, "percentage": 81.98, "elapsed_time": "7:49:32", "remaining_time": "1:43:10"}
+{"current_steps": 1530, "total_steps": 1854, "loss": 0.7993, "accuracy": 0.5687500238418579, "learning_rate": 3.6744280277467904e-07, "epoch": 2.4732269145281873, "percentage": 82.52, "elapsed_time": "7:52:26", "remaining_time": "1:40:02"}
+{"current_steps": 1540, "total_steps": 1854, "loss": 0.7295, "accuracy": 0.6625000238418579, "learning_rate": 3.456410618180503e-07, "epoch": 2.489391796322489, "percentage": 83.06, "elapsed_time": "7:55:28", "remaining_time": "1:36:56"}
+{"current_steps": 1550, "total_steps": 1854, "loss": 0.7689, "accuracy": 0.606249988079071, "learning_rate": 3.244579563165753e-07, "epoch": 2.5055566781167915, "percentage": 83.6, "elapsed_time": "7:58:22", "remaining_time": "1:33:49"}
+{"current_steps": 1560, "total_steps": 1854, "loss": 0.7985, "accuracy": 0.625, "learning_rate": 3.038995691099697e-07, "epoch": 2.521721559911093, "percentage": 84.14, "elapsed_time": "8:01:14", "remaining_time": "1:30:41"}
+{"current_steps": 1570, "total_steps": 1854, "loss": 0.9096, "accuracy": 0.5687500238418579, "learning_rate": 2.839718036468192e-07, "epoch": 2.5378864417053952, "percentage": 84.68, "elapsed_time": "8:04:12", "remaining_time": "1:27:35"}
+{"current_steps": 1580, "total_steps": 1854, "loss": 0.8158, "accuracy": 0.5687500238418579, "learning_rate": 2.646803822893723e-07, "epoch": 2.5540513234996967, "percentage": 85.22, "elapsed_time": "8:07:11", "remaining_time": "1:24:29"}
+{"current_steps": 1590, "total_steps": 1854, "loss": 0.8268, "accuracy": 0.5625, "learning_rate": 2.460308446703341e-07, "epoch": 2.570216205293999, "percentage": 85.76, "elapsed_time": "8:10:08", "remaining_time": "1:21:22"}
+{"current_steps": 1600, "total_steps": 1854, "loss": 0.7238, "accuracy": 0.668749988079071, "learning_rate": 2.2802854610213143e-07, "epoch": 2.5863810870883004, "percentage": 86.3, "elapsed_time": "8:13:04", "remaining_time": "1:18:16"}
+{"current_steps": 1610, "total_steps": 1854, "loss": 0.8356, "accuracy": 0.612500011920929, "learning_rate": 2.106786560391072e-07, "epoch": 2.6025459688826027, "percentage": 86.84, "elapsed_time": "8:15:58", "remaining_time": "1:15:10"}
+{"current_steps": 1620, "total_steps": 1854, "loss": 0.7951, "accuracy": 0.581250011920929, "learning_rate": 1.9398615659308255e-07, "epoch": 2.6187108506769046, "percentage": 87.38, "elapsed_time": "8:18:59", "remaining_time": "1:12:04"}
+{"current_steps": 1630, "total_steps": 1854, "loss": 0.8102, "accuracy": 0.606249988079071, "learning_rate": 1.7795584110272184e-07, "epoch": 2.6348757324712064, "percentage": 87.92, "elapsed_time": "8:22:08", "remaining_time": "1:09:00"}
+{"current_steps": 1640, "total_steps": 1854, "loss": 0.7825, "accuracy": 0.5687500238418579, "learning_rate": 1.6259231275709636e-07, "epoch": 2.6510406142655083, "percentage": 88.46, "elapsed_time": "8:25:11", "remaining_time": "1:05:55"}
+{"current_steps": 1650, "total_steps": 1854, "loss": 0.7818, "accuracy": 0.6000000238418579, "learning_rate": 1.478999832738548e-07, "epoch": 2.66720549605981, "percentage": 89.0, "elapsed_time": "8:28:06", "remaining_time": "1:02:49"}
+{"current_steps": 1660, "total_steps": 1854, "loss": 0.7852, "accuracy": 0.581250011920929, "learning_rate": 1.338830716323769e-07, "epoch": 2.683370377854112, "percentage": 89.54, "elapsed_time": "8:31:10", "remaining_time": "0:59:44"}
+{"current_steps": 1670, "total_steps": 1854, "loss": 0.7441, "accuracy": 0.6187499761581421, "learning_rate": 1.205456028622723e-07, "epoch": 2.699535259648414, "percentage": 90.08, "elapsed_time": "8:33:59", "remaining_time": "0:56:37"}
+{"current_steps": 1680, "total_steps": 1854, "loss": 0.7691, "accuracy": 0.625, "learning_rate": 1.0789140688756805e-07, "epoch": 2.7157001414427158, "percentage": 90.61, "elapsed_time": "8:36:56", "remaining_time": "0:53:32"}
+{"current_steps": 1690, "total_steps": 1854, "loss": 0.8064, "accuracy": 0.550000011920929, "learning_rate": 9.592411742693098e-08, "epoch": 2.7318650232370176, "percentage": 91.15, "elapsed_time": "8:39:45", "remaining_time": "0:50:26"}
+{"current_steps": 1700, "total_steps": 1854, "loss": 0.7608, "accuracy": 0.65625, "learning_rate": 8.464717095022168e-08, "epoch": 2.7480299050313195, "percentage": 91.69, "elapsed_time": "8:42:41", "remaining_time": "0:47:20"}
+{"current_steps": 1710, "total_steps": 1854, "loss": 0.8177, "accuracy": 0.518750011920929, "learning_rate": 7.406380569169841e-08, "epoch": 2.7641947868256214, "percentage": 92.23, "elapsed_time": "8:45:39", "remaining_time": "0:44:15"}
+{"current_steps": 1720, "total_steps": 1854, "loss": 0.7931, "accuracy": 0.550000011920929, "learning_rate": 6.417706072013808e-08, "epoch": 2.7803596686199232, "percentage": 92.77, "elapsed_time": "8:48:40", "remaining_time": "0:41:11"}
+{"current_steps": 1730, "total_steps": 1854, "loss": 0.8244, "accuracy": 0.606249988079071, "learning_rate": 5.498977506615294e-08, "epoch": 2.796524550414225, "percentage": 93.31, "elapsed_time": "8:51:44", "remaining_time": "0:38:06"}
+{"current_steps": 1740, "total_steps": 1854, "loss": 0.8438, "accuracy": 0.643750011920929, "learning_rate": 4.6504586906947756e-08, "epoch": 2.812689432208527, "percentage": 93.85, "elapsed_time": "8:54:43", "remaining_time": "0:35:01"}
+{"current_steps": 1750, "total_steps": 1854, "loss": 0.8997, "accuracy": 0.5375000238418579, "learning_rate": 3.8723932808754914e-08, "epoch": 2.828854314002829, "percentage": 94.39, "elapsed_time": "8:57:46", "remaining_time": "0:31:57"}
+{"current_steps": 1760, "total_steps": 1854, "loss": 0.7929, "accuracy": 0.612500011920929, "learning_rate": 3.1650047027158014e-08, "epoch": 2.8450191957971307, "percentage": 94.93, "elapsed_time": "9:00:50", "remaining_time": "0:28:53"}
+{"current_steps": 1770, "total_steps": 1854, "loss": 0.7445, "accuracy": 0.606249988079071, "learning_rate": 2.5284960865517848e-08, "epoch": 2.8611840775914326, "percentage": 95.47, "elapsed_time": "9:03:54", "remaining_time": "0:25:48"}
+{"current_steps": 1780, "total_steps": 1854, "loss": 0.7761, "accuracy": 0.637499988079071, "learning_rate": 1.9630502091670388e-08, "epoch": 2.8773489593857344, "percentage": 96.01, "elapsed_time": "9:06:47", "remaining_time": "0:22:43"}
+{"current_steps": 1790, "total_steps": 1854, "loss": 0.7146, "accuracy": 0.6000000238418579, "learning_rate": 1.4688294413074677e-08, "epoch": 2.8935138411800363, "percentage": 96.55, "elapsed_time": "9:09:39", "remaining_time": "0:19:39"}
+{"current_steps": 1800, "total_steps": 1854, "loss": 0.7989, "accuracy": 0.550000011920929, "learning_rate": 1.0459757010556626e-08, "epoch": 2.909678722974338, "percentage": 97.09, "elapsed_time": "9:12:34", "remaining_time": "0:16:34"}
+{"current_steps": 1810, "total_steps": 1854, "loss": 0.7943, "accuracy": 0.574999988079071, "learning_rate": 6.94610413078306e-09, "epoch": 2.92584360476864, "percentage": 97.63, "elapsed_time": "9:15:23", "remaining_time": "0:13:30"}
+{"current_steps": 1820, "total_steps": 1854, "loss": 0.7354, "accuracy": 0.65625, "learning_rate": 4.14834473758563e-09, "epoch": 2.942008486562942, "percentage": 98.17, "elapsed_time": "9:18:23", "remaining_time": "0:10:25"}
+{"current_steps": 1830, "total_steps": 1854, "loss": 0.753, "accuracy": 0.643750011920929, "learning_rate": 2.067282222230349e-09, "epoch": 2.9581733683572438, "percentage": 98.71, "elapsed_time": "9:21:32", "remaining_time": "0:07:21"}
+{"current_steps": 1840, "total_steps": 1854, "loss": 0.7369, "accuracy": 0.637499988079071, "learning_rate": 7.035141727212979e-10, "epoch": 2.9743382501515456, "percentage": 99.24, "elapsed_time": "9:24:19", "remaining_time": "0:04:17"}
+{"current_steps": 1850, "total_steps": 1854, "loss": 0.8952, "accuracy": 0.612500011920929, "learning_rate": 5.743220219761592e-11, "epoch": 2.9905031319458475, "percentage": 99.78, "elapsed_time": "9:27:21", "remaining_time": "0:01:13"}
+{"current_steps": 1854, "total_steps": 1854, "epoch": 2.9969690846635686, "percentage": 100.0, "elapsed_time": "9:28:31", "remaining_time": "0:00:00"}