Edens-Gate/sd-gen-1.5b

See axolotl config

axolotl version: 0.4.1

base_model: Qwen/Qwen2-1.5B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

trust_remote_code: true

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: civit-slop-combined.jsonl
    type: alpaca

chat_template: chatml

dataset_prepared_path:
val_set_size: 0.05
output_dir: ./outputs/out
sequence_len: 2048
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true

adapter:
lora_model_dir:
lora_r:
lora_alpha:
lora_dropout:
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project: Mango-SDprompt-qwen
wandb_entity:
wandb_watch:
wandb_name: qwen1.5b-2
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 1
num_epochs: 4
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_ratio: 0.2
evals_per_epoch: 4
saves_per_epoch: 1
debug:
#deepspeed: deepspeed_configs/zero2.json
#deepspeed: /training/axolotl/axolotl/deepspeed_configs/zero2.json
weight_decay: 0.0
#fsdp:
#fsdp_config:
#  fsdp_limit_all_gathers: true
#  fsdp_sync_module_states: true
#  fsdp_offload_params: true
#  fsdp_use_orig_params: false
#  fsdp_cpu_ram_efficient_loading: true
#  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
#  fsdp_transformer_layer_cls_to_wrap: Qwen2DecoderLayer
#  fsdp_state_dict_type: FULL_STATE_DICT
special_tokens:

outputs/out

This model is a fine-tuned version of Qwen/Qwen2-1.5B on the None dataset. It achieves the following results on the evaluation set:

• Loss: 4.9785

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 312
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss
3.4279	0.0024	1	4.1321
2.6008	0.2516	106	3.4527
2.7346	0.5033	212	3.6512
3.0676	0.7549	318	3.8178
3.2084	1.0065	424	3.7616
2.6999	1.2368	530	3.8072
2.3664	1.4884	636	3.7223
3.0722	1.7401	742	3.6350
2.7612	1.9917	848	3.5670
1.2179	2.2226	954	4.0217
1.5539	2.4742	1060	3.9957
1.255	2.7258	1166	4.0404
0.9165	2.9774	1272	4.0294
0.1785	3.2089	1378	4.9794
0.1931	3.4605	1484	4.9763
0.3729	3.7122	1590	4.9785

Framework versions

• Transformers 4.44.0.dev0
• Pytorch 2.1.2+cu118
• Datasets 2.19.1
• Tokenizers 0.19.1