llama3-2M-MedEV / README.md
Angelectronic's picture
Update README.md
d653cd2 verified
metadata
license: apache-2.0
library_name: peft
tags:
  - trl
  - sft
  - unsloth
  - generated_from_trainer
base_model: unsloth/llama-3-8b-Instruct-bnb-4bit
model-index:
  - name: llama3-2M-MedEV
    results: []

llama3-2M-MedEV

This model is a fine-tuned version of unsloth/llama-3-8b-Instruct-bnb-4bit on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 1.4249
  • Bleu: 47.7973

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 32
  • eval_batch_size: 16
  • seed: 3407
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 128
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 5
  • num_epochs: 3

Training results

Training Loss Epoch Step Validation Loss
1.534 0.1200 320 1.4902
1.3171 0.2399 640 1.4705
1.29 0.3599 960 1.4644
1.2699 0.4798 1280 1.4287
1.2567 0.5998 1600 1.4576
1.2448 0.7197 1920 1.4196
1.2353 0.8397 2240 1.4249
1.2274 0.9596 2560 1.4172
1.1635 1.0796 2880 1.4180
1.1337 1.1995 3200 1.4219
1.1346 1.3195 3520 1.3954
1.131 1.4394 3840 1.3714
1.1325 1.5594 4160 1.3923
1.1269 1.6793 4480 1.4118
1.1221 1.7993 4800 1.4251
1.1226 1.9192 5120 1.3970
1.0898 2.0392 5440 1.4198
1.0372 2.1591 5760 1.4310
1.0325 2.2791 6080 1.4209
1.0334 2.3990 6400 1.4205
1.0328 2.5190 6720 1.4306
1.0303 2.6389 7040 1.4222
1.0283 2.7589 7360 1.4266
1.0273 2.8788 7680 1.4251
1.0295 2.9988 8000 1.4249

Framework versions

  • PEFT 0.10.0
  • Transformers 4.40.2
  • Pytorch 2.3.0
  • Datasets 2.19.1
  • Tokenizers 0.19.1