willtensora
/

fd1980a0-7e71-4e52-addb-318dca5991d5

@@ -1,11 +1,12 @@
 ---
 library_name: transformers
-base_model: katuni4ka/tiny-random-qwen1.5-moe
 tags:
 - axolotl
 - generated_from_trainer
 model-index:
-- name: e61e89f0-854a-4922-8d25-dae435e91af0
   results: []
 ---
@@ -17,20 +18,20 @@ should probably proofread and complete it, then remove this comment. -->
 axolotl version: `0.4.1`
 ```yaml
-base_model: katuni4ka/tiny-random-qwen1.5-moe
 batch_size: 32
 bf16: true
 chat_template: tokenizer_default_fallback_alpaca
 datasets:
 - data_files:
-  - 95544452e61c7393_train_data.json
   ds_type: json
   format: custom
-  path: /workspace/input_data/95544452e61c7393_train_data.json
   type:
-    field_input: input
-    field_instruction: instruction
-    field_output: output
     format: '{instruction} {input}'
     no_input_format: '{instruction}'
     system_format: '{system}'
@@ -40,7 +41,7 @@ flash_attention: true
 gpu_memory_limit: 80GiB
 gradient_checkpointing: true
 group_by_length: true
-hub_model_id: willtensora/e61e89f0-854a-4922-8d25-dae435e91af0
 hub_strategy: checkpoint
 learning_rate: 0.0002
 logging_steps: 10
@@ -56,13 +57,13 @@ sample_packing: false
 save_steps: 40
 save_total_limit: 1
 sequence_len: 2048
-tokenizer_type: Qwen2TokenizerFast
 train_on_inputs: false
 trust_remote_code: true
 val_set_size: 0.1
 wandb_entity: ''
 wandb_mode: online
-wandb_name: katuni4ka/tiny-random-qwen1.5-moe-/workspace/input_data/95544452e61c7393_train_data.json
 wandb_project: Gradients-On-Demand
 wandb_run: your_name
 wandb_runid: default
@@ -73,11 +74,11 @@ xformers_attention: true
 </details><br>
-# e61e89f0-854a-4922-8d25-dae435e91af0
-This model is a fine-tuned version of [katuni4ka/tiny-random-qwen1.5-moe](https://huggingface.co/katuni4ka/tiny-random-qwen1.5-moe) on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 11.6281
 ## Model description
@@ -106,16 +107,21 @@ The following hyperparameters were used during training:
 - total_eval_batch_size: 32
 - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: cosine
-- lr_scheduler_warmup_steps: 2
-- training_steps: 40
 ### Training results
 | Training Loss | Epoch  | Step | Validation Loss |
 |:-------------:|:------:|:----:|:---------------:|
-| No log        | 0.0031 | 1    | 11.9223         |
-| 11.7325       | 0.0629 | 20   | 11.6783         |
-| 11.6304       | 0.1258 | 40   | 11.6281         |
 ### Framework versions

 ---
 library_name: transformers
+license: mit
+base_model: fxmarty/tiny-random-GemmaForCausalLM
 tags:
 - axolotl
 - generated_from_trainer
 model-index:
+- name: fd1980a0-7e71-4e52-addb-318dca5991d5
   results: []
 ---
 axolotl version: `0.4.1`
 ```yaml
+base_model: fxmarty/tiny-random-GemmaForCausalLM
 batch_size: 32
 bf16: true
 chat_template: tokenizer_default_fallback_alpaca
 datasets:
 - data_files:
+  - b7c2a4a781c93416_train_data.json
   ds_type: json
   format: custom
+  path: /workspace/input_data/b7c2a4a781c93416_train_data.json
   type:
+    field_input: context
+    field_instruction: question
+    field_output: answer
     format: '{instruction} {input}'
     no_input_format: '{instruction}'
     system_format: '{system}'
 gpu_memory_limit: 80GiB
 gradient_checkpointing: true
 group_by_length: true
+hub_model_id: willtensora/fd1980a0-7e71-4e52-addb-318dca5991d5
 hub_strategy: checkpoint
 learning_rate: 0.0002
 logging_steps: 10
 save_steps: 40
 save_total_limit: 1
 sequence_len: 2048
+tokenizer_type: GemmaTokenizerFast
 train_on_inputs: false
 trust_remote_code: true
 val_set_size: 0.1
 wandb_entity: ''
 wandb_mode: online
+wandb_name: fxmarty/tiny-random-GemmaForCausalLM-/workspace/input_data/b7c2a4a781c93416_train_data.json
 wandb_project: Gradients-On-Demand
 wandb_run: your_name
 wandb_runid: default
 </details><br>
+# fd1980a0-7e71-4e52-addb-318dca5991d5
+This model is a fine-tuned version of [fxmarty/tiny-random-GemmaForCausalLM](https://huggingface.co/fxmarty/tiny-random-GemmaForCausalLM) on the None dataset.
 It achieves the following results on the evaluation set:
+- Loss: 11.7971
 ## Model description
 - total_eval_batch_size: 32
 - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 7
+- training_steps: 156
 ### Training results
 | Training Loss | Epoch  | Step | Validation Loss |
 |:-------------:|:------:|:----:|:---------------:|
+| No log        | 0.0008 | 1    | 12.4537         |
+| 12.4357       | 0.0161 | 20   | 12.4267         |
+| 12.392        | 0.0322 | 40   | 12.3762         |
+| 12.3026       | 0.0483 | 60   | 12.2651         |
+| 12.1177       | 0.0645 | 80   | 12.0658         |
+| 11.9286       | 0.0806 | 100  | 11.8860         |
+| 11.8324       | 0.0967 | 120  | 11.8100         |
+| 11.798        | 0.1128 | 140  | 11.7971         |
 ### Framework versions

generation_config.json CHANGED Viewed

@@ -1,7 +1,8 @@
 {
   "_from_model_config": true,
-  "bos_token_id": 151643,
   "do_sample": true,
-  "eos_token_id": 151643,
   "transformers_version": "4.46.0"
 }

 {
   "_from_model_config": true,
+  "bos_token_id": 2,
   "do_sample": true,
+  "eos_token_id": 1,
+  "pad_token_id": 0,
   "transformers_version": "4.46.0"
 }

pytorch_model.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:244c30fce0d5c4892e3b25d25e50c952fa49cb08493bb32684f850179545a7e3
-size 19817334

 version https://git-lfs.github.com/spec/v1
+oid sha256:4bc76ae4e72c9fc13bfe9567ae655234c8d3f2fcf4460d169dedaebd1865dcc9
+size 16392015