Hugging Face's logo

GaetanMichelet
/
Llama-31-8B_task-2_120-samples_config-4

PEFT
TensorBoard
Safetensors
llama
alignment-handbook
trl
sft
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files Metrics Community
Llama-31-8B_task-2_120-samples_config-4 / README.md
GaetanMichelet's picture
GaetanMichelet
End of training
834f8f8 verified
preview code
|
raw
history blame contribute delete
3.86 kB
metadata
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
datasets:
  - GaetanMichelet/chat-60_ft_task-2
  - GaetanMichelet/chat-120_ft_task-2
library_name: peft
license: llama3.1
tags:
  - alignment-handbook
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: Llama-31-8B_task-2_120-samples_config-4
    results: []

Llama-31-8B_task-2_120-samples_config-4

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-2 and the GaetanMichelet/chat-120_ft_task-2 datasets. It achieves the following results on the evaluation set:

  • Loss: 0.7144

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 1e-05
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • distributed_type: multi-GPU
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 150

Training results

Training Loss Epoch Step Validation Loss
1.1544 0.9091 5 1.1237
1.132 2.0 11 1.1193
1.0689 2.9091 16 1.1136
1.0956 4.0 22 1.1011
1.1157 4.9091 27 1.0871
1.0778 6.0 33 1.0639
1.0458 6.9091 38 1.0393
0.9854 8.0 44 1.0027
0.9996 8.9091 49 0.9696
0.8991 10.0 55 0.9317
0.8897 10.9091 60 0.9052
0.8711 12.0 66 0.8788
0.8809 12.9091 71 0.8588
0.7972 14.0 77 0.8368
0.8156 14.9091 82 0.8208
0.7815 16.0 88 0.8057
0.7492 16.9091 93 0.7956
0.7587 18.0 99 0.7855
0.7483 18.9091 104 0.7780
0.7296 20.0 110 0.7695
0.7441 20.9091 115 0.7629
0.7176 22.0 121 0.7561
0.7033 22.9091 126 0.7508
0.6906 24.0 132 0.7443
0.6954 24.9091 137 0.7396
0.6578 26.0 143 0.7344
0.6495 26.9091 148 0.7310
0.6391 28.0 154 0.7269
0.6442 28.9091 159 0.7237
0.6268 30.0 165 0.7199
0.6536 30.9091 170 0.7183
0.6092 32.0 176 0.7163
0.621 32.9091 181 0.7149
0.5823 34.0 187 0.7144
0.5651 34.9091 192 0.7156
0.5951 36.0 198 0.7164
0.5637 36.9091 203 0.7195
0.5669 38.0 209 0.7219
0.5613 38.9091 214 0.7278
0.5156 40.0 220 0.7309
0.5044 40.9091 225 0.7395

Framework versions

  • PEFT 0.12.0
  • Transformers 4.44.0
  • Pytorch 2.1.2+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1