metadata

license: mit
base_model: roberta-large
tags:
  - generated_from_trainer
datasets:
  - launch/open_question_type
metrics:
  - f1
model-index:
  - name: roberta-large-question-classifier
    results:
      - task:
          name: Text Classification
          type: text-classification
        dataset:
          name: launch/open_question_type
          type: launch/open_question_type
          config: default
          split: validation
          args: default
        metrics:
          - name: F1 (macro avg.)
            type: f1
            value: 0.8123190611646329
      - task:
          name: Text Classification
          type: text-classification
        dataset:
          name: launch/open_question_type
          type: launch/open_question_type
          config: default
          split: test
          args: default
        metrics:
          - name: F1 (macro avg.)
            type: f1
            value: 0.8
widget:
  - text: >-
      When two bacteria exchange genetic information, what is the process
      called?
language:
  - en
arxiv: 2107.00152

roberta-large-question-classifier

This model classifies questions according to the question-type ontology defined in following paper: Controllable Open-ended Question Generation with A New Question Type Ontology (Cao & Wang, ACL-IJCNLP 2021). It is a fine-tuned roberta-large on the open_question_type dataset. It achieves the following results on the test set:

              precision    recall  f1-score   support
       cause       0.91      0.93      0.92        91
  comparison       0.62      0.83      0.71        30
     concept       0.85      0.65      0.74        54
 consequence       0.80      0.73      0.76        11
 disjunction       0.80      0.78      0.79        36
     example       0.83      0.85      0.84       139
      extent       0.82      0.94      0.87        48
  judgmental       0.68      0.56      0.62        94
  procedural       0.86      0.88      0.87        85
verification       0.79      0.86      0.83        72
    accuracy                           0.81       660
   macro avg       0.80      0.80      0.80       660
weighted avg       0.81      0.81      0.81       660

Training procedure

Script: https://gist.github.com/jantrienes/329479bdad6b2a239cfcea83b9159a8a

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 16
eval_batch_size: 512
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_ratio: 0.1
num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss	F1
1.9467	1.0	233	1.3099	0.4050
0.6381	2.0	466	0.5586	0.7785
0.628	3.0	699	0.6419	0.7831
0.4487	4.0	932	0.5770	0.8094
0.3319	5.0	1165	0.7713	0.7953
0.2095	6.0	1398	0.8799	0.8018
0.1355	7.0	1631	1.0646	0.7961
0.0956	8.0	1864	1.2175	0.7999
0.0687	9.0	2097	1.3647	0.7892
0.0371	10.0	2330	1.3809	0.7987
0.0303	11.0	2563	1.3591	0.8123
0.0263	12.0	2796	1.5317	0.8100
0.0144	13.0	3029	1.5726	0.7959
0.0436	14.0	3262	1.6160	0.7988
0.0048	15.0	3495	1.6826	0.7957
0.0001	16.0	3728	1.6913	0.7957
0.0001	17.0	3961	1.7076	0.7995
0.0034	18.0	4194	1.8018	0.7960
0.0228	19.0	4427	1.7457	0.7916
0.0083	20.0	4660	1.9279	0.7869
0.0001	21.0	4893	1.8367	0.7915
0.0003	22.0	5126	1.8620	0.7842
0.0002	23.0	5359	1.9192	0.7828
0.0	24.0	5592	1.9081	0.7927
0.0003	25.0	5825	1.9822	0.7813
0.0059	26.0	6058	1.8737	0.7954
0.0	27.0	6291	1.8793	0.7929
0.0	28.0	6524	1.8905	0.7940
0.0	29.0	6757	1.8971	0.7940
0.0002	30.0	6990	1.9002	0.7954

Framework versions

Transformers 4.33.2
Pytorch 2.1.0+cu118
Datasets 2.14.5
Tokenizers 0.13.3