collapse_gemma-2-27b_hs2_replace_iter5_sftsd1

This model is a fine-tuned version of google/gemma-2-27b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.3937
• Num Input Tokens Seen: 2703900

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 4
• eval_batch_size: 16
• seed: 1
• gradient_accumulation_steps: 32
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.1282	0
4.0912	0.1071	5	1.1202	291172
3.8259	0.2142	10	1.2093	586952
3.656	0.3213	15	1.2610	881004
3.4522	0.4284	20	1.3078	1174016
3.2654	0.5355	25	1.3624	1467268
3.2203	0.6426	30	1.3656	1762408
3.0818	0.7497	35	1.3812	2055808
3.0687	0.8568	40	1.3846	2347528
3.0439	0.9639	45	1.3870	2645048

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1