Hugging Face's logo

itimal
/
falcon7binstruct_fine_tunning_PPF_english_conversations_2

PEFT
TensorBoard
Safetensors
trl
sft
Generated from Trainer
Model card Files Metrics Community
falcon7binstruct_fine_tunning_PPF_english_conversations_2 / README.md
itimal's picture
itimal
End of training
51a2bca verified
preview code
|
raw
history blame
3.1 kB
metadata
base_model: vilsonrodrigues/falcon-7b-instruct-sharded
library_name: peft
license: apache-2.0
tags:
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: falcon7binstruct_fine_tunning_PPF_english_conversations_2
    results: []

Visualize in Weights & Biases

falcon7binstruct_fine_tunning_PPF_english_conversations_2

This model is a fine-tuned version of vilsonrodrigues/falcon-7b-instruct-sharded on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 0.3807

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.03
  • training_steps: 250
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
1.4574 0.1325 10 0.8588
0.7214 0.2649 20 0.6192
0.6047 0.3974 30 0.5418
0.5232 0.5298 40 0.4877
0.4663 0.6623 50 0.4642
0.4592 0.7947 60 0.4437
0.4663 0.9272 70 0.4345
0.4386 1.0596 80 0.4169
0.34 1.1921 90 0.4198
0.3441 1.3245 100 0.4103
0.3459 1.4570 110 0.3999
0.3322 1.5894 120 0.3963
0.3287 1.7219 130 0.3871
0.3212 1.8543 140 0.3783
0.2906 1.9868 150 0.3788
0.2526 2.1192 160 0.3839
0.2417 2.2517 170 0.3903
0.2451 2.3841 180 0.3880
0.2492 2.5166 190 0.3896
0.2404 2.6490 200 0.3854
0.2348 2.7815 210 0.3822
0.2375 2.9139 220 0.3810
0.2264 3.0464 230 0.3806
0.2141 3.1788 240 0.3806
0.2069 3.3113 250 0.3807

Framework versions

  • PEFT 0.11.2.dev0
  • Transformers 4.42.4
  • Pytorch 2.3.1+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1