collapse_gemma-2-27b_hs2_replace_iter5_sftsd2

This model is a fine-tuned version of google/gemma-2-27b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.3885
• Num Input Tokens Seen: 2751792

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 4
• eval_batch_size: 16
• seed: 2
• gradient_accumulation_steps: 32
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.1282	0
4.321	0.1060	5	1.1089	288828
4.0789	0.2119	10	1.2018	581504
3.8543	0.3179	15	1.2446	873364
3.6951	0.4238	20	1.3045	1165140
3.4936	0.5298	25	1.3632	1461128
3.4498	0.6358	30	1.3670	1750764
3.3639	0.7417	35	1.3663	2044840
3.2972	0.8477	40	1.3672	2339024
3.2375	0.9536	45	1.3869	2633860

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1