Hugging Face's logo

GaetanMichelet
/
Llama-31-8B_task-3_180-samples_config-4

PEFT
TensorBoard
Safetensors
llama
alignment-handbook
trl
sft
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files Metrics Community
Llama-31-8B_task-3_180-samples_config-4 / README.md
GaetanMichelet's picture
GaetanMichelet
Model save
c8bc8c4 verified
preview code
|
raw
history blame
3.31 kB
metadata
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
library_name: peft
license: llama3.1
tags:
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: Llama-31-8B_task-3_180-samples_config-4
    results: []

Llama-31-8B_task-3_180-samples_config-4

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 0.5094

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 1e-05
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • distributed_type: multi-GPU
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 150

Training results

Training Loss Epoch Step Validation Loss
2.5495 0.9412 8 2.5077
2.3733 2.0 17 2.4775
2.46 2.9412 25 2.4218
2.4585 4.0 34 2.3278
2.2624 4.9412 42 2.1919
2.0553 6.0 51 1.9704
1.7403 6.9412 59 1.7066
1.3756 8.0 68 1.3617
1.11 8.9412 76 1.0613
0.7161 10.0 85 0.7772
0.7609 10.9412 93 0.6787
0.4358 12.0 102 0.6182
0.4774 12.9412 110 0.5912
0.5569 14.0 119 0.5746
0.427 14.9412 127 0.5487
0.4672 16.0 136 0.5339
0.3495 16.9412 144 0.5525
0.4731 18.0 153 0.5323
0.3913 18.9412 161 0.5243
0.5624 20.0 170 0.5253
0.4684 20.9412 178 0.5222
0.3029 22.0 187 0.5100
0.3522 22.9412 195 0.5085
0.3855 24.0 204 0.4971
0.317 24.9412 212 0.5049
0.338 26.0 221 0.5016
0.391 26.9412 229 0.4942
0.3964 28.0 238 0.5010
0.2951 28.9412 246 0.5098
0.4021 30.0 255 0.5068
0.4021 30.9412 263 0.5070
0.3456 32.0 272 0.5025
0.4431 32.9412 280 0.5050
0.4131 34.0 289 0.5094

Framework versions

  • PEFT 0.12.0
  • Transformers 4.44.0
  • Pytorch 2.1.2+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1