Edit model card

llama-7b_oasst1_l0.0002_16-4-4-4-4

This model is a fine-tuned version of huggyllama/llama-7b on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 1.6192

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 0
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • lr_scheduler_warmup_ratio: 0.03
  • training_steps: 1875

Training results

Training Loss Epoch Step Validation Loss
1.5015 0.0018 1 1.6490
1.5011 0.3392 187 1.3137
1.1301 0.6783 374 1.3026
1.2775 1.0175 561 1.3072
1.1684 1.3566 748 1.3332
1.0755 1.6958 935 1.3290
0.631 2.0349 1122 1.4114
0.8068 2.3741 1309 1.4261
0.7406 2.7132 1496 1.4971
0.4579 3.0524 1683 1.5951
0.4512 3.3915 1870 1.5517

Framework versions

  • PEFT 0.12.1.dev0
  • Transformers 4.45.0.dev0
  • Pytorch 2.3.0+cu121
  • Datasets 2.19.0
  • Tokenizers 0.19.1
Downloads last month
2
Inference API
Unable to determine this model’s pipeline type. Check the docs .

Model tree for alexander-hm/llama-7b_oasst1_l0.0002_16-4-4-4-4

Adapter
(87)
this model