metadata

base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-sep_tok_full_labels
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: sep_tok_full_labels
          split: train[80%:100%]
          args: sep_tok_full_labels
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8887729338495203

longformer-sep_tok_full_labels

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

Loss: 0.2742
B-claim: {'precision': 0.5965665236051502, 'recall': 0.5129151291512916, 'f1-score': 0.5515873015873016, 'support': 271.0}
B-majorclaim: {'precision': 0.8135593220338984, 'recall': 0.6906474820143885, 'f1-score': 0.7470817120622568, 'support': 139.0}
B-premise: {'precision': 0.8395953757225434, 'recall': 0.9178515007898894, 'f1-score': 0.8769811320754718, 'support': 633.0}
I-claim: {'precision': 0.6239658393381372, 'recall': 0.5843539115221195, 'f1-score': 0.6035105833763552, 'support': 4001.0}
I-majorclaim: {'precision': 0.8184480234260615, 'recall': 0.8330849478390462, 'f1-score': 0.8257016248153619, 'support': 2013.0}
I-premise: {'precision': 0.8878504672897196, 'recall': 0.9050811573747354, 'f1-score': 0.8963830159007513, 'support': 11336.0}
O: {'precision': 1.0, 'recall': 0.9998231966053748, 'f1-score': 0.9999115904871364, 'support': 11312.0}
Accuracy: 0.8888
Macro avg: {'precision': 0.7971407930593586, 'recall': 0.7776796178995493, 'f1-score': 0.7858795657578049, 'support': 29705.0}
Weighted avg: {'precision': 0.8862788836235276, 'recall': 0.8887729338495203, 'f1-score': 0.8873130640630906, 'support': 29705.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	B-claim	B-majorclaim	B-premise	I-claim	I-majorclaim	I-premise	O	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.3939	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 271.0}	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 139.0}	{'precision': 0.6519871106337272, 'recall': 0.9589257503949447, 'f1-score': 0.7762148337595907, 'support': 633.0}	{'precision': 0.4607276436586195, 'recall': 0.33866533366658336, 'f1-score': 0.390377412849323, 'support': 4001.0}	{'precision': 0.7161716171617162, 'recall': 0.53899652260308, 'f1-score': 0.6150793650793651, 'support': 2013.0}	{'precision': 0.8191702652683529, 'recall': 0.9371030345800988, 'f1-score': 0.8741770901909152, 'support': 11336.0}	{'precision': 0.9918942731277534, 'recall': 0.9952263083451203, 'f1-score': 0.9935574971317624, 'support': 11312.0}	0.8392	{'precision': 0.5199929871214527, 'recall': 0.538416707084261, 'f1-score': 0.5213437427158508, 'support': 29705.0}	{'precision': 0.8148175308318133, 'recall': 0.839185322336307, 'f1-score': 0.8227635981297234, 'support': 29705.0}
No log	2.0	82	0.3194	{'precision': 0.3546099290780142, 'recall': 0.18450184501845018, 'f1-score': 0.24271844660194175, 'support': 271.0}	{'precision': 0.7, 'recall': 0.050359712230215826, 'f1-score': 0.09395973154362416, 'support': 139.0}	{'precision': 0.7255813953488373, 'recall': 0.985781990521327, 'f1-score': 0.8359008707300737, 'support': 633.0}	{'precision': 0.5617191404297851, 'recall': 0.2809297675581105, 'f1-score': 0.37454181939353554, 'support': 4001.0}	{'precision': 0.7387470997679815, 'recall': 0.7908594138102335, 'f1-score': 0.7639155470249521, 'support': 2013.0}	{'precision': 0.8175242974459429, 'recall': 0.9572159491884262, 'f1-score': 0.8818724856759722, 'support': 11336.0}	{'precision': 0.9999112294718153, 'recall': 0.9957567185289957, 'f1-score': 0.9978296496434425, 'support': 11312.0}	0.8588	{'precision': 0.6997275845060538, 'recall': 0.6064864852651084, 'f1-score': 0.5986769358019346, 'support': 29705.0}	{'precision': 0.8404537879266404, 'recall': 0.8588453122369971, 'f1-score': 0.8392062502215126, 'support': 29705.0}
No log	3.0	123	0.2681	{'precision': 0.5185185185185185, 'recall': 0.46494464944649444, 'f1-score': 0.490272373540856, 'support': 271.0}	{'precision': 0.7804878048780488, 'recall': 0.460431654676259, 'f1-score': 0.579185520361991, 'support': 139.0}	{'precision': 0.8223776223776224, 'recall': 0.9289099526066351, 'f1-score': 0.8724035608308606, 'support': 633.0}	{'precision': 0.6154261057173679, 'recall': 0.5703574106473381, 'f1-score': 0.5920352834349463, 'support': 4001.0}	{'precision': 0.772628843655262, 'recall': 0.8862394436164928, 'f1-score': 0.8255437297547431, 'support': 2013.0}	{'precision': 0.8900924702774108, 'recall': 0.8915843330980946, 'f1-score': 0.8908377770922393, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9983203677510608, 'f1-score': 0.9991594779915948, 'support': 11312.0}	0.8835	{'precision': 0.7713616236320329, 'recall': 0.7429696874060535, 'f1-score': 0.7499196747153186, 'support': 29705.0}	{'precision': 0.8816455584137204, 'recall': 0.8834876283453964, 'f1-score': 0.8819115909019992, 'support': 29705.0}
No log	4.0	164	0.2651	{'precision': 0.5869565217391305, 'recall': 0.4981549815498155, 'f1-score': 0.5389221556886228, 'support': 271.0}	{'precision': 0.8034188034188035, 'recall': 0.6762589928057554, 'f1-score': 0.7343750000000001, 'support': 139.0}	{'precision': 0.8359712230215828, 'recall': 0.9178515007898894, 'f1-score': 0.8750000000000001, 'support': 633.0}	{'precision': 0.6168500134156157, 'recall': 0.5746063484128968, 'f1-score': 0.5949792960662525, 'support': 4001.0}	{'precision': 0.8074074074074075, 'recall': 0.812220566318927, 'f1-score': 0.8098068350668647, 'support': 2013.0}	{'precision': 0.8854211569962928, 'recall': 0.9059633027522935, 'f1-score': 0.8955744495312841, 'support': 11336.0}	{'precision': 0.9999115983026874, 'recall': 0.9999115983026874, 'f1-score': 0.9999115983026874, 'support': 11312.0}	0.8862	{'precision': 0.7908481034716457, 'recall': 0.769281041561752, 'f1-score': 0.7783670478079587, 'support': 29705.0}	{'precision': 0.8833991740695555, 'recall': 0.8862144420131292, 'f1-score': 0.884561060819018, 'support': 29705.0}
No log	5.0	205	0.2742	{'precision': 0.5965665236051502, 'recall': 0.5129151291512916, 'f1-score': 0.5515873015873016, 'support': 271.0}	{'precision': 0.8135593220338984, 'recall': 0.6906474820143885, 'f1-score': 0.7470817120622568, 'support': 139.0}	{'precision': 0.8395953757225434, 'recall': 0.9178515007898894, 'f1-score': 0.8769811320754718, 'support': 633.0}	{'precision': 0.6239658393381372, 'recall': 0.5843539115221195, 'f1-score': 0.6035105833763552, 'support': 4001.0}	{'precision': 0.8184480234260615, 'recall': 0.8330849478390462, 'f1-score': 0.8257016248153619, 'support': 2013.0}	{'precision': 0.8878504672897196, 'recall': 0.9050811573747354, 'f1-score': 0.8963830159007513, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9998231966053748, 'f1-score': 0.9999115904871364, 'support': 11312.0}	0.8888	{'precision': 0.7971407930593586, 'recall': 0.7776796178995493, 'f1-score': 0.7858795657578049, 'support': 29705.0}	{'precision': 0.8862788836235276, 'recall': 0.8887729338495203, 'f1-score': 0.8873130640630906, 'support': 29705.0}

Framework versions

Transformers 4.37.2
Pytorch 2.2.0+cu121
Datasets 2.17.0
Tokenizers 0.15.2