Hugging Face's logo

TERRYYYZHANG
/
shawgpt-ft

PEFT
TensorBoard
Safetensors
Generated from Trainer
Model card Files Metrics Community
shawgpt-ft / README.md
TERRYYYZHANG's picture
TERRYYYZHANG
TERRYYYZHANG/test-Llama-3.1-8B-Instruct-GPTQ-ytbcomment
f753964 verified
preview code
|
raw
history blame contribute delete
2.01 kB
metadata
base_model: hugging-quants/Meta-Llama-3.1-8B-Instruct-GPTQ-INT4
library_name: peft
license: llama3.1
tags:
  - generated_from_trainer
model-index:
  - name: shawgpt-ft
    results: []

shawgpt-ft

This model is a fine-tuned version of hugging-quants/Meta-Llama-3.1-8B-Instruct-GPTQ-INT4 on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 2.0354

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 2
  • num_epochs: 10
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
3.8384 0.9231 3 3.2747
3.666 1.8462 6 3.1189
3.4422 2.7692 9 2.9378
2.3978 4.0 13 2.6971
2.9756 4.9231 16 2.5165
2.7288 5.8462 19 2.3571
2.5339 6.7692 22 2.2220
1.796 8.0 26 2.0908
2.2918 8.9231 29 2.0411
1.6171 9.2308 30 2.0354

Framework versions

  • PEFT 0.12.0
  • Transformers 4.42.4
  • Pytorch 2.3.1+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1