Edens-Gate/sd-prompter-1.5b-exp

See axolotl config

axolotl version: 0.4.1

base_model: MangyMango/testing1
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

trust_remote_code: true

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: civit-slop-combined.jsonl
    type: alpaca
    conversation: mpt-30b-instruct

chat_template: alpaca

dataset_prepared_path:
val_set_size: 0.05
output_dir: ./outputs/sd-prompter
sequence_len: 2048
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

adapter:
lora_model_dir:
lora_r:
lora_alpha:
lora_dropout:
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project: Mango-SDprompt-qwen
wandb_entity:
wandb_watch:
wandb_name: qwen1.5b-2
wandb_log_model:

gradient_accumulation_steps: 64
micro_batch_size: 2
num_epochs: 3
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 0.00002

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_ratio: 0.05
evals_per_epoch: 4
saves_per_epoch: 1
debug:
#deepspeed: deepspeed_configs/zero2.json
#deepspeed: /training/axolotl/axolotl/deepspeed_configs/zero2.json
weight_decay: 0.0
#fsdp:
#fsdp_config:
#  fsdp_limit_all_gathers: true
#  fsdp_sync_module_states: true
#  fsdp_offload_params: true
#  fsdp_use_orig_params: false
#  fsdp_cpu_ram_efficient_loading: true
#  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
#  fsdp_transformer_layer_cls_to_wrap: Qwen2DecoderLayer
#  fsdp_state_dict_type: FULL_STATE_DICT
special_tokens:

outputs/sd-prompter

This model is a fine-tuned version of MangyMango/testing1 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 3.4889

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 64
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 2
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
3.4783	0.0793	1	4.2312
3.3803	0.2379	3	3.8651
3.0646	0.4758	6	3.6872
2.8913	0.7138	9	3.6106
2.9159	0.9517	12	3.5590
2.819	1.1660	15	3.5307
2.8095	1.4040	18	3.5109
2.8054	1.6419	21	3.4995
2.9067	1.8798	24	3.4933
2.8035	2.0954	27	3.4903
2.7619	2.3333	30	3.4890
2.8226	2.5713	33	3.4891
2.7211	2.8092	36	3.4889

Framework versions

• Transformers 4.44.0.dev0
• Pytorch 2.1.2+cu118
• Datasets 2.19.1
• Tokenizers 0.19.1