collapse_gemma-2-27b_hs2_replace_iter2_sftsd2

This model is a fine-tuned version of google/gemma-2-27b on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 1.2781
  • Num Input Tokens Seen: 3603012

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 8e-06
  • train_batch_size: 4
  • eval_batch_size: 16
  • seed: 2
  • gradient_accumulation_steps: 32
  • total_train_batch_size: 128
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant_with_warmup
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss Input Tokens Seen
No log 0 0 1.1282 0
2.7288 0.0560 5 1.0428 210528
2.2631 0.1121 10 1.0614 409500
1.7035 0.1681 15 1.0857 615964
1.0224 0.2242 20 1.1604 820452
0.7612 0.2802 25 1.1895 1018848
0.5972 0.3363 30 1.1930 1215560
0.6178 0.3923 35 1.1725 1409568
0.4143 0.4483 40 1.1575 1608480
0.5734 0.5044 45 1.1651 1812632
0.4687 0.5604 50 1.1621 2009960
0.6309 0.6165 55 1.1799 2223316
0.5393 0.6725 60 1.1957 2428084
0.271 0.7285 65 1.2141 2637248
0.4383 0.7846 70 1.2308 2837024
0.2703 0.8406 75 1.2080 3032684
0.3999 0.8967 80 1.2440 3240344
0.3003 0.9527 85 1.2357 3443968

Framework versions

  • Transformers 4.44.0
  • Pytorch 2.4.0+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1
Downloads last month
11
Safetensors
Model size
27.2B params
Tensor type
BF16
·
Inference API
Unable to determine this model's library. Check the docs .

Model tree for RylanSchaeffer/collapse_gemma-2-27b_hs2_replace_iter2_sftsd2

Base model

google/gemma-2-27b
Finetuned
(33)
this model