metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-sep_tok_full_labels
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: sep_tok_full_labels
          split: train[80%:100%]
          args: sep_tok_full_labels
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8938225887897661

longformer-sep_tok_full_labels

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

Loss: 0.2832
B-claim: {'precision': 0.65, 'recall': 0.5756457564575646, 'f1-score': 0.6105675146771037, 'support': 271.0}
B-majorclaim: {'precision': 0.8206896551724138, 'recall': 0.8561151079136691, 'f1-score': 0.8380281690140844, 'support': 139.0}
B-premise: {'precision': 0.8632218844984803, 'recall': 0.8973143759873617, 'f1-score': 0.8799380325329202, 'support': 633.0}
I-claim: {'precision': 0.6347290007706139, 'recall': 0.6175956010997251, 'f1-score': 0.6260450975424373, 'support': 4001.0}
I-majorclaim: {'precision': 0.8701298701298701, 'recall': 0.8653750620963736, 'f1-score': 0.8677459526774596, 'support': 2013.0}
I-premise: {'precision': 0.8889858614068773, 'recall': 0.8985532815808045, 'f1-score': 0.893743967710801, 'support': 11336.0}
O: {'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0}
Accuracy: 0.8938
Macro avg: {'precision': 0.8182508959968936, 'recall': 0.81576199714908, 'f1-score': 0.8165623020032957, 'support': 29705.0}
Weighted avg: {'precision': 0.8926884767049156, 'recall': 0.8938225887897661, 'f1-score': 0.8932001664235063, 'support': 29705.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 7

Training results

Training Loss	Epoch	Step	Validation Loss	B-claim	B-majorclaim	B-premise	I-claim	I-majorclaim	I-premise	O	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.3941	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 271.0}	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 139.0}	{'precision': 0.6378035902851109, 'recall': 0.9541864139020537, 'f1-score': 0.7645569620253164, 'support': 633.0}	{'precision': 0.44665194996320823, 'recall': 0.303424143964009, 'f1-score': 0.3613632981098378, 'support': 4001.0}	{'precision': 0.7471544715447155, 'recall': 0.4565325384997516, 'f1-score': 0.5667591736046871, 'support': 2013.0}	{'precision': 0.8023273050696709, 'recall': 0.9549223712067749, 'f1-score': 0.871999355566296, 'support': 11336.0}	{'precision': 0.9959356776815692, 'recall': 0.9964639321074965, 'f1-score': 0.9961997348652232, 'support': 11312.0}	0.8360	{'precision': 0.5185532849348964, 'recall': 0.5236470570971551, 'f1-score': 0.5086969320244801, 'support': 29705.0}	{'precision': 0.8098304255807603, 'recall': 0.8360208719070863, 'f1-score': 0.8155078749254555, 'support': 29705.0}
No log	2.0	82	0.3007	{'precision': 0.3785310734463277, 'recall': 0.24723247232472326, 'f1-score': 0.29910714285714285, 'support': 271.0}	{'precision': 0.8823529411764706, 'recall': 0.1079136690647482, 'f1-score': 0.1923076923076923, 'support': 139.0}	{'precision': 0.7395957193816884, 'recall': 0.9826224328593997, 'f1-score': 0.8439620081411127, 'support': 633.0}	{'precision': 0.566340160284951, 'recall': 0.3179205198700325, 'f1-score': 0.40723547302705304, 'support': 4001.0}	{'precision': 0.7801980198019802, 'recall': 0.7829110779930452, 'f1-score': 0.7815521943962312, 'support': 2013.0}	{'precision': 0.8230264162467552, 'recall': 0.9509527170077628, 'f1-score': 0.8823770156339527, 'support': 11336.0}	{'precision': 0.9998231027772864, 'recall': 0.9992927864214993, 'f1-score': 0.9995578742594394, 'support': 11312.0}	0.8630	{'precision': 0.7385524904450655, 'recall': 0.6269779536487444, 'f1-score': 0.6294427715175177, 'support': 29705.0}	{'precision': 0.8473214966654586, 'recall': 0.863019693654267, 'f1-score': 0.8468020526459685, 'support': 29705.0}
No log	3.0	123	0.2656	{'precision': 0.5555555555555556, 'recall': 0.44280442804428044, 'f1-score': 0.4928131416837782, 'support': 271.0}	{'precision': 0.8367346938775511, 'recall': 0.5899280575539568, 'f1-score': 0.6919831223628692, 'support': 139.0}	{'precision': 0.8120713305898491, 'recall': 0.9352290679304898, 'f1-score': 0.8693098384728339, 'support': 633.0}	{'precision': 0.6189608021877848, 'recall': 0.50912271932017, 'f1-score': 0.5586944596818432, 'support': 4001.0}	{'precision': 0.8277078085642318, 'recall': 0.8161947342275211, 'f1-score': 0.821910955477739, 'support': 2013.0}	{'precision': 0.8641668736031786, 'recall': 0.9209597741707833, 'f1-score': 0.8916599051970789, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9993811881188119, 'f1-score': 0.9996904982977407, 'support': 11312.0}	0.8826	{'precision': 0.7878852949111643, 'recall': 0.7448028527665732, 'f1-score': 0.7608659887391263, 'support': 29705.0}	{'precision': 0.8763419120561073, 'recall': 0.8826460191886888, 'f1-score': 0.8781757290617714, 'support': 29705.0}
No log	4.0	164	0.2673	{'precision': 0.6179245283018868, 'recall': 0.4833948339483395, 'f1-score': 0.5424430641821946, 'support': 271.0}	{'precision': 0.7857142857142857, 'recall': 0.7913669064748201, 'f1-score': 0.7885304659498209, 'support': 139.0}	{'precision': 0.8393632416787264, 'recall': 0.9162717219589257, 'f1-score': 0.8761329305135951, 'support': 633.0}	{'precision': 0.6405669929125886, 'recall': 0.47438140464883777, 'f1-score': 0.5450890292935094, 'support': 4001.0}	{'precision': 0.8220742150333016, 'recall': 0.8584202682563339, 'f1-score': 0.839854191980559, 'support': 2013.0}	{'precision': 0.8581196581196581, 'recall': 0.9299576570218772, 'f1-score': 0.8925955717370136, 'support': 11336.0}	{'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}	0.8854	{'precision': 0.7948232745372067, 'recall': 0.7791132560441619, 'f1-score': 0.7835207505223847, 'support': 29705.0}	{'precision': 0.8774744030572791, 'recall': 0.8854064972226898, 'f1-score': 0.8790840278083316, 'support': 29705.0}
No log	5.0	205	0.2698	{'precision': 0.631578947368421, 'recall': 0.6642066420664207, 'f1-score': 0.6474820143884892, 'support': 271.0}	{'precision': 0.8013245033112583, 'recall': 0.8705035971223022, 'f1-score': 0.8344827586206897, 'support': 139.0}	{'precision': 0.8976897689768977, 'recall': 0.8593996840442338, 'f1-score': 0.8781275221953189, 'support': 633.0}	{'precision': 0.6005734450816056, 'recall': 0.680579855036241, 'f1-score': 0.6380785002929115, 'support': 4001.0}	{'precision': 0.8148654810761514, 'recall': 0.8877297565822156, 'f1-score': 0.849738468854018, 'support': 2013.0}	{'precision': 0.9169016732468509, 'recall': 0.8604446012702893, 'f1-score': 0.8877764630927459, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9987623762376238, 'f1-score': 0.9993808049535603, 'support': 11312.0}	0.8890	{'precision': 0.8089905455801693, 'recall': 0.8316609303370467, 'f1-score': 0.819295218913962, 'support': 29705.0}	{'precision': 0.8954719932524351, 'recall': 0.8889749200471301, 'f1-score': 0.8914196505512982, 'support': 29705.0}
No log	6.0	246	0.2851	{'precision': 0.61328125, 'recall': 0.5793357933579336, 'f1-score': 0.5958254269449716, 'support': 271.0}	{'precision': 0.8717948717948718, 'recall': 0.7338129496402878, 'f1-score': 0.796875, 'support': 139.0}	{'precision': 0.8522388059701492, 'recall': 0.9020537124802528, 'f1-score': 0.8764389869531849, 'support': 633.0}	{'precision': 0.6405070118662352, 'recall': 0.5936015996001, 'f1-score': 0.6161629264496044, 'support': 4001.0}	{'precision': 0.8946784922394678, 'recall': 0.8017883755588674, 'f1-score': 0.845690332722033, 'support': 2013.0}	{'precision': 0.8785780629907962, 'recall': 0.9178722653493295, 'f1-score': 0.8977954182665343, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.999557991513437, 'f1-score': 0.9997789469030461, 'support': 11312.0}	0.8931	{'precision': 0.8215826421230743, 'recall': 0.789717526785744, 'f1-score': 0.8040810054627677, 'support': 29705.0}	{'precision': 0.890828586148026, 'recall': 0.8931493014644, 'f1-score': 0.8914853059005039, 'support': 29705.0}
No log	7.0	287	0.2832	{'precision': 0.65, 'recall': 0.5756457564575646, 'f1-score': 0.6105675146771037, 'support': 271.0}	{'precision': 0.8206896551724138, 'recall': 0.8561151079136691, 'f1-score': 0.8380281690140844, 'support': 139.0}	{'precision': 0.8632218844984803, 'recall': 0.8973143759873617, 'f1-score': 0.8799380325329202, 'support': 633.0}	{'precision': 0.6347290007706139, 'recall': 0.6175956010997251, 'f1-score': 0.6260450975424373, 'support': 4001.0}	{'precision': 0.8701298701298701, 'recall': 0.8653750620963736, 'f1-score': 0.8677459526774596, 'support': 2013.0}	{'precision': 0.8889858614068773, 'recall': 0.8985532815808045, 'f1-score': 0.893743967710801, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0}	0.8938	{'precision': 0.8182508959968936, 'recall': 0.81576199714908, 'f1-score': 0.8165623020032957, 'support': 29705.0}	{'precision': 0.8926884767049156, 'recall': 0.8938225887897661, 'f1-score': 0.8932001664235063, 'support': 29705.0}

Framework versions

Transformers 4.37.2
Pytorch 2.2.0+cu121
Datasets 2.17.0
Tokenizers 0.15.2