metadata

license: mit
tags:
  - generated_from_trainer
metrics:
  - f1
  - accuracy
model-index:
  - name: Kemenkeu-Sentiment-Classifier
    results:
      - task:
          name: Text Classification
          type: text-classification
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.66
          - name: F1
            type: f1
            value: 0.6368
language:
  - id
pipeline_tag: text-classification
widget:
  - text: sudah beli makan buat sahur?
    example_title: contoh tidak relevan
  - text: Mengawal APBN, Indonesia Maju
    example_title: contoh kalimat

Kemenkeu-Sentiment-Classifier

This model is a fine-tuned version of indobenchmark/indobert-base-p1 on the MoF-DAC Mini Challenge#1 dataset. It achieves the following results on the evaluation set:

Accuracy: 0.66
F1: 0.6368

Leaderboard score:

Public score: 0.63733
Private score: 0.65733

Model description & limitations

This model can be used to classify text with four possible outputs [netral, tdk-relevan, negatif, and positif]
only for specific cases related to the Ministry Of Finance Indonesia

How to use

You can use this model directly with a pipeline

pretrained_name = "hanifnoerr/Kemenkeu-Sentiment-Classifier"
class_model = pipeline(tokenizer=pretrained_name, model=pretrained_name)

test_data = "Mengawal APBN, Indonesia Maju"
class_model(test_data)

Training and evaluation data

The following hyperparameters were used during training:

learning_rate: 1e-05
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1
1.0131	1.0	500	0.8590	0.644	0.5964
0.7133	2.0	1000	0.8639	0.63	0.5924
0.5261	3.0	1500	0.9002	0.66	0.6368

Framework versions

Transformers 4.27.4
Pytorch 2.0.0+cu118
Datasets 2.11.0
Tokenizers 0.13.3