--- library_name: peft license: other base_model: mistralai/Mistral-Small-Instruct-2409 tags: - generated_from_trainer datasets: - Fizzarolli/inkmix-v2 model-index: - name: ckpts results: [] --- [Built with Axolotl](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config axolotl version: `0.6.0` ```yaml base_model: mistralai/Mistral-Small-Instruct-2409 load_in_8bit: true load_in_4bit: false plugins: - axolotl.integrations.liger.LigerPlugin liger_rope: true liger_rms_norm: true liger_glu_activation: true liger_fused_linear_cross_entropy: true #unsloth_lora_mlp: true #unsloth_lora_qkv: true #unsloth_lora_o: true strict: false adapter: lora lora_r: 16 lora_alpha: 32 lora_dropout: 0.25 lora_target_modules: - q_proj - v_proj - k_proj - o_proj lora_target_linear: true peft_layers_to_transform: loraplus_lr_ratio: 16 chat_template: jinja chat_template_jinja: "{%- if messages[0][\"role\"] == \"fake\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" or message[\"role\"] == \"system\"%}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS][\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST]\" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST]\" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif (message.tool_calls is defined and message.tool_calls is not none) %}\n {{- \"[TOOL_CALLS][\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- message[\"content\"] + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS]{\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n" datasets: - path: Fizzarolli/inkmix-v2 type: chat_template chat_template: tokenizer_default split: train field_messages: conversations message_field_role: from message_field_content: value dataset_prepared_path: last_run_prepared #val_set_size: 0.02 output_dir: ./ckpts sequence_len: 8192 sample_packing: true pad_to_sequence_len: true #wandb_project: teleut-7b-rp #wandb_entity: #wandb_watch: #wandb_name: #wandb_log_model: checkpoint # mlflow configuration if you're using it mlflow_tracking_uri: https://public-tracking.mlflow-e00zzfjq11ky6jcgtv.backbone-e00bgn6e63256prmhq.msp.eu-north1.nebius.cloud mlflow_experiment_name: ms-12b-rp-inkmixv2 mlflow_run_name: v1 hf_mlflow_log_artifacts: true gradient_accumulation_steps: 1 micro_batch_size: 8 num_epochs: 2 optimizer: paged_adamw_8bit lr_scheduler: cosine learning_rate: 6e-5 train_on_inputs: false group_by_length: false bf16: auto fp16: tf32: false gradient_checkpointing: unsloth gradient_checkpointing_kwargs: use_reentrant: false early_stopping_patience: resume_from_checkpoint: logging_steps: 1 xformers_attention: flash_attention: true #deepspeed: deepspeed_configs/zero3_bf16.json warmup_steps: 25 #evals_per_epoch: 4 eval_table_size: saves_per_epoch: 10 debug: weight_decay: 0.01 ```

# ckpts This model is a fine-tuned version of [mistralai/Mistral-Small-Instruct-2409](https://huggingface.co./mistralai/Mistral-Small-Instruct-2409) on the Fizzarolli/inkmix-v2 dataset. ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 6e-05 - train_batch_size: 8 - eval_batch_size: 8 - seed: 42 - optimizer: Use paged_adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 25 - num_epochs: 2 ### Training results ### Framework versions - PEFT 0.14.0 - Transformers 4.47.1 - Pytorch 2.5.1+cu124 - Datasets 3.1.0 - Tokenizers 0.21.0