README.md

metadata

license: cc-by-nc-4.0
base_model: KT-AI/midm-bitext-S-7B-inst-v1
tags:
  - generated_from_trainer
model-index:
  - name: lora-midm-7b-nsmc-review-understanding
    results: []
datasets:
  - nsmc

lora-midm-7b-nsmc-review-understanding

This model is a fine-tuned version of KT-AI/midm-bitext-S-7B-inst-v1 on an unknown dataset.

Model description

nsmc data 기반 미세튜닝 모델

Intended uses & limitations

More information needed

Training and evaluation data

training data로 nsmc train data 앞쪽 2000개, evaluation data로 nsmc test data 앞쪽 1000개를 사용했습니다.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 2
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• training_steps: 200
• mixed_precision_training: Native AMP

Training results

총 200step 돌렸습니다. 50step마다 check한 결과는 아래와 같습니다.
50 step training loss: 1.6881
100 step training loss: 1.1443
150 step training loss: 1.0563
200 step training loss: 1.0446

실험 내용 및 분류 결과

미세튜닝한 모델에 nsmc test data 1000개를 입력으로 주어 긍정 또는 부정 단어를 생성하도록 했습니다.
단어 생성 결과는 '긍정' 444개, '부정' 532개, ' , ' 4개, '정' 20개 입니다.
정확도는 정답수 / 1000 * 100으로 계산했으며, 결과는 87.80% 입니다.

Framework versions

• Transformers 4.35.2
• Pytorch 2.1.0+cu118
• Datasets 2.15.0
• Tokenizers 0.15.0