Hugging Face's logo

GaetanMichelet
/
Llama-31-8B_task-1_180-samples_config-4_full

PEFT
TensorBoard
Safetensors
llama
alignment-handbook
trl
sft
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files Metrics Community
Llama-31-8B_task-1_180-samples_config-4_full / README.md
GaetanMichelet's picture
GaetanMichelet
End of training
90df86c verified
preview code
|
raw
history blame contribute delete
4.58 kB
metadata
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
datasets:
  - GaetanMichelet/chat-60_ft_task-1
  - GaetanMichelet/chat-120_ft_task-1
  - GaetanMichelet/chat-180_ft_task-1
library_name: peft
license: llama3.1
tags:
  - alignment-handbook
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: Llama-31-8B_task-1_180-samples_config-4_full
    results: []

Llama-31-8B_task-1_180-samples_config-4_full

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-1, the GaetanMichelet/chat-120_ft_task-1 and the GaetanMichelet/chat-180_ft_task-1 datasets. It achieves the following results on the evaluation set:

  • Loss: 0.9048

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 1e-05
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • distributed_type: multi-GPU
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 150

Training results

Training Loss Epoch Step Validation Loss
2.4745 0.9412 8 2.4335
2.4286 2.0 17 2.4114
2.419 2.9412 25 2.3814
2.3475 4.0 34 2.3262
2.3147 4.9412 42 2.2541
2.2214 6.0 51 2.1716
2.1097 6.9412 59 2.0745
1.9617 8.0 68 1.9479
1.908 8.9412 76 1.8375
1.7669 10.0 85 1.6953
1.6325 10.9412 93 1.5461
1.3201 12.0 102 1.3739
1.2477 12.9412 110 1.2331
1.163 14.0 119 1.1330
1.0579 14.9412 127 1.0861
1.0655 16.0 136 1.0611
0.9976 16.9412 144 1.0455
1.0285 18.0 153 1.0318
0.998 18.9412 161 1.0205
1.0038 20.0 170 1.0102
0.9907 20.9412 178 1.0020
0.9673 22.0 187 0.9929
0.95 22.9412 195 0.9870
0.9467 24.0 204 0.9801
0.9423 24.9412 212 0.9737
0.937 26.0 221 0.9675
0.9035 26.9412 229 0.9626
0.9074 28.0 238 0.9582
0.8944 28.9412 246 0.9534
0.8785 30.0 255 0.9493
0.8797 30.9412 263 0.9451
0.8764 32.0 272 0.9422
0.8903 32.9412 280 0.9389
0.8835 34.0 289 0.9377
0.8452 34.9412 297 0.9332
0.8777 36.0 306 0.9272
0.8101 36.9412 314 0.9257
0.8526 38.0 323 0.9229
0.8228 38.9412 331 0.9197
0.8066 40.0 340 0.9176
0.7701 40.9412 348 0.9199
0.8132 42.0 357 0.9162
0.7804 42.9412 365 0.9104
0.7508 44.0 374 0.9083
0.7192 44.9412 382 0.9052
0.7633 46.0 391 0.9048
0.7534 46.9412 399 0.9052
0.666 48.0 408 0.9151
0.7298 48.9412 416 0.9143
0.6815 50.0 425 0.9157
0.6845 50.9412 433 0.9170
0.6524 52.0 442 0.9216
0.6397 52.9412 450 0.9228

Framework versions

  • PEFT 0.12.0
  • Transformers 4.44.0
  • Pytorch 2.1.2+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1