autoevaluator's picture
Add verifyToken field to verify evaluation results are produced by Hugging Face's automatic model evaluator
ca15c10
|
raw
history blame
7.26 kB
metadata
language: id
license: mit
tags:
  - indonesian-roberta-base-indonli
datasets:
  - indonli
widget:
  - text: Andi tersenyum karena mendapat hasil baik. </s></s> Andi sedih.
model-index:
  - name: w11wo/indonesian-roberta-base-indonli
    results:
      - task:
          type: natural-language-inference
          name: Natural Language Inference
        dataset:
          name: indonli
          type: indonli
          config: indonli
          split: test_expert
        metrics:
          - type: accuracy
            value: 0.6072386058981233
            name: Accuracy
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzY4NDkwNTdlYjI4MzY3ZTk3NmZjYjA1MjE2YWQ5MjJjMGM3NTc1NWVjODQzNTc1ZTYyZWVmYmY5NTI3NWY1ZSIsInZlcnNpb24iOjF9.Aeo_Id90j2JtyApv3LvJHkQtHz-9wO4SNvTdb8O_pp0KFQGfWXnkgX2t2hafIUxSKmZbETIte-FaPbZ9AGZSDA
          - type: precision
            value: 0.6304330508019023
            name: Precision Macro
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2I3Y2RkNjU0NzlkYmJiNWYyZjZhNzIzZGE5ODU4NzYxYmQ0NTYxYzZkM2JiNTQwZTdkMmYxOTRmMDlmOGFkMiIsInZlcnNpb24iOjF9.iEt7Mq6a3TubFQfdC3OAxAiZDXp0bPGhN9JPzSfKl89_dxnKzDp0IrVzkt1HNLHR_S22Q75Tevqh3_G8Pp05Dg
          - type: precision
            value: 0.6072386058981233
            name: Precision Micro
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYmI4ZmEwOTY0NTViNTM1ZjcwY2E2ODRmNGJiMTg2ZDJmZTgyNGUxM2UwNjZjYzVmYTcxZGY4OGY3OTI4MzcyMSIsInZlcnNpb24iOjF9.Jn1OPD1ZxkblCqKT1CfeUYOt5Xb6CL6C2ZENLmvfYNzh-p0oHcIBgapfbCHc89oMSR-FhjQk_ME8f8A3eyy6CA
          - type: precision
            value: 0.6320495884503851
            name: Precision Weighted
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYWU1NmI5MTk3ODY4Y2M5ZjYyNzMwZjRlZTAzNjFmNmUxNzgxYjVhODNjZDAwOTQyNDBlZDJkNDYyNzRlYzBmOSIsInZlcnNpb24iOjF9.ItCi8SouqOtM3P7c0KN5ifRpGOr1090aqo4zX4aVSlVOTq0iQj9_c3z0B_UAzFcr0qW7ObnvuiD8D5d-9EzkBg
          - type: recall
            value: 0.6127303344145852
            name: Recall Macro
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNTZmMGEzMGZjYTI2MzA0OGQ1Nzk5ZDRmNDNhMTRhOGMyM2I4Zjc2NzMzNmM2NjQ0YzVjZDY1Mzk1ZWI5Zjg2NiIsInZlcnNpb24iOjF9.fWCCNatB50nCptCbXopRjwxbWic6BvWIG6frUo_iXJVFXsi3Q_ik91_70fLgZc9NfhIpewpNoe4ETn0Gmps4Cw
          - type: recall
            value: 0.6072386058981233
            name: Recall Micro
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNGMwOWVmOWU4YzM2ZjMyY2NjNjljMDdjZTZmMjU5MjRiNDU0NmVhYWEzZmQ5MzUzMmRhMjdmZjhkNDU4ZTM2ZCIsInZlcnNpb24iOjF9.Sy2c29OhxT-x4UBSr9G7rfwtyqzYOX4KNRe2blonfOdKrqSfSEORY01Y67WweDiKdRvbECzI-DemJUXVtx-QCg
          - type: recall
            value: 0.6072386058981233
            name: Recall Weighted
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiODgwODAwMDNkZWJlMzYyYmQwYzNmMGZiMzhjZjVjNGYyMTg3OWVjNjZmMzFhMDczNGEyMDAyODkwZTZhZWM4NSIsInZlcnNpb24iOjF9.8IxdbjDQHzcNW71RAMtKHzlviweLTQvYVQ4JlrqoZsV-8gyzxpyYOmDjUm3n6uQNfRLRpyvsT-E8ysLHPyMqDg
          - type: f1
            value: 0.6010566054103847
            name: F1 Macro
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZmMwMjg2YjZmYzMzYzlmNTk3OGIyMDc5NGUxNTFlNmNkNmU3MzU2ZWMyZjY0OGFhYmY5YTUyNDkxNjJiODIwNSIsInZlcnNpb24iOjF9.r1ylajrOC-Qu4QNdNnXzisjGlczTF_9tYpNEr8LYdTtdmJtRjNtNmElINneuaWX7XGN9TExdzmg7OWTwutjsAg
          - type: f1
            value: 0.6072386058981233
            name: F1 Micro
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDdhMGU1NDNjMzgxN2NjNWYzNTk5OWY5OTZhZWRjOGZkNjI4ZDA1YjI5ZWYxOWNmNDc4NmVhNjllMjUyMTFkMSIsInZlcnNpb24iOjF9.5G1km-a2_ssO_b3WTD8Ools29e6h8X8rjpClFN5Q_I4ADbPxKI2QbCfd5vl89CMHclignQ1_H6vqYbdTL9usDQ
          - type: f1
            value: 0.5995456855334425
            name: F1 Weighted
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMjYyNTZjOTUzNmMxOTY3MzQzNjMxZGNhYTY3NTQ4Mjg3NWRlMjc2NmY1NjMxOWY0NTFiODlhZjA3ZTEzNGQ3MSIsInZlcnNpb24iOjF9.3iTI9IieFa3WJFr7ovDvO24IPScGB7WQk3Pw_Qxh32zKx5QyOwmZf_p2bgbEG6hBeCkR0KaMDvIiZXnbW6DBDQ
          - type: loss
            value: 1.157181739807129
            name: loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZmIxNTQyYWRkMjgxMTZkN2JhZTg5NDFiMDRlZGEzOGE5ZDIwYTE5NTU4YmU0NDUxOTE1MDQwMzFlMjQ5MDQ2YSIsInZlcnNpb24iOjF9.M-U6Dp-I-DEXZ3qSGLxYrCdQjgXi6DotHDgz1acjWnHIZWKPApy-n2194FZik1Tpv2AcJVe45tDRLxNSW3zVBg

Indonesian RoBERTa Base IndoNLI

Indonesian RoBERTa Base IndoNLI is a natural language inference (NLI) model based on the RoBERTa model. The model was originally the pre-trained Indonesian RoBERTa Base model, which is then fine-tuned on IndoNLI's dataset consisting of Indonesian Wikipedia, news, and Web articles [1].

After training, the model achieved an evaluation/dev accuracy of 77.06%. On the benchmark test_lay subset, the model achieved an accuracy of 74.24% and on the benchmark test_expert subset, the model achieved an accuracy of 61.66%.

Hugging Face's Trainer class from the Transformers library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.

Model

Model #params Arch. Training/Validation data (text)
indonesian-roberta-base-indonli 124M RoBERTa Base IndoNLI

Evaluation Results

The model was trained for 5 epochs, with a batch size of 16, a learning rate of 2e-5, a weight decay of 0.1, and a warmup ratio of 0.2, with linear annealing to 0. The best model was loaded at the end.

Epoch Training Loss Validation Loss Accuracy
1 0.989200 0.691663 0.731452
2 0.673000 0.621913 0.766045
3 0.449900 0.662543 0.770596
4 0.293600 0.777059 0.768320
5 0.194200 0.948068 0.764224

How to Use

As NLI Classifier

from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-indonli"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Andi tersenyum karena mendapat hasil baik. </s></s> Andi sedih.")

Disclaimer

Do consider the biases which come from both the pre-trained RoBERTa model and the IndoNLI dataset that may be carried over into the results of this model.

References

[1] Mahendra, R., Aji, A. F., Louvan, S., Rahman, F., & Vania, C. (2021, November). IndoNLI: A Natural Language Inference Dataset for Indonesian. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

Author

Indonesian RoBERTa Base IndoNLI was trained and evaluated by Wilson Wongso. All computation and development are done on Google Colaboratory using their free GPU access.