metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-sep_tok_full_labels
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: sep_tok_full_labels
          split: train[80%:100%]
          args: sep_tok_full_labels
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8970543679515233

longformer-sep_tok_full_labels

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

Loss: 0.3490
B-claim: {'precision': 0.6679841897233202, 'recall': 0.6236162361623616, 'f1-score': 0.6450381679389313, 'support': 271.0}
B-majorclaim: {'precision': 0.851063829787234, 'recall': 0.8633093525179856, 'f1-score': 0.8571428571428571, 'support': 139.0}
B-premise: {'precision': 0.8736517719568567, 'recall': 0.8957345971563981, 'f1-score': 0.8845553822152886, 'support': 633.0}
I-claim: {'precision': 0.6537938536850694, 'recall': 0.6008497875531117, 'f1-score': 0.6262047408179214, 'support': 4001.0}
I-majorclaim: {'precision': 0.9007751937984496, 'recall': 0.8658718330849479, 'f1-score': 0.8829787234042552, 'support': 2013.0}
I-premise: {'precision': 0.8802487011327825, 'recall': 0.911697247706422, 'f1-score': 0.8956970143432854, 'support': 11336.0}
O: {'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0}
Accuracy: 0.8971
Macro avg: {'precision': 0.8325025057262447, 'recall': 0.8229734070127556, 'f1-score': 0.8273548951044003, 'support': 29705.0}
Weighted avg: {'precision': 0.8945272549791482, 'recall': 0.8970543679515233, 'f1-score': 0.8955015738591867, 'support': 29705.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss	B-claim	B-majorclaim	B-premise	I-claim	I-majorclaim	I-premise	O	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.4114	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 271.0}	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 139.0}	{'precision': 0.6441908713692946, 'recall': 0.981042654028436, 'f1-score': 0.7777082028804008, 'support': 633.0}	{'precision': 0.45472716370177896, 'recall': 0.5686078480379905, 'f1-score': 0.5053309640159929, 'support': 4001.0}	{'precision': 0.7067901234567902, 'recall': 0.4550422255340288, 'f1-score': 0.5536415835599879, 'support': 2013.0}	{'precision': 0.8708348262271587, 'recall': 0.857621736062103, 'f1-score': 0.8641777777777776, 'support': 11336.0}	{'precision': 0.9955665898208902, 'recall': 0.9925742574257426, 'f1-score': 0.9940681717574147, 'support': 11312.0}	0.8336	{'precision': 0.5245870820822732, 'recall': 0.5506983887269001, 'f1-score': 0.5278466714273676, 'support': 29705.0}	{'precision': 0.834322086390543, 'recall': 0.8335970375357684, 'f1-score': 0.830493769902922, 'support': 29705.0}
No log	2.0	82	0.2953	{'precision': 0.4044943820224719, 'recall': 0.2656826568265683, 'f1-score': 0.32071269487750553, 'support': 271.0}	{'precision': 0.875, 'recall': 0.2014388489208633, 'f1-score': 0.327485380116959, 'support': 139.0}	{'precision': 0.7609756097560976, 'recall': 0.985781990521327, 'f1-score': 0.8589125946317963, 'support': 633.0}	{'precision': 0.5836245631552671, 'recall': 0.29217695576105973, 'f1-score': 0.38940706195869423, 'support': 4001.0}	{'precision': 0.7398819561551433, 'recall': 0.8718330849478391, 'f1-score': 0.8004561003420751, 'support': 2013.0}	{'precision': 0.8254749634643489, 'recall': 0.9467184191954834, 'f1-score': 0.8819492953116653, 'support': 11336.0}	{'precision': 0.9999114965926188, 'recall': 0.9987623762376238, 'f1-score': 0.9993366060766884, 'support': 11312.0}	0.8644	{'precision': 0.741337567306564, 'recall': 0.6517706189158234, 'f1-score': 0.6540371047593406, 'support': 29705.0}	{'precision': 0.8485436064203262, 'recall': 0.8644335970375358, 'f1-score': 0.8465825270165918, 'support': 29705.0}
No log	3.0	123	0.2542	{'precision': 0.5887445887445888, 'recall': 0.5018450184501845, 'f1-score': 0.5418326693227091, 'support': 271.0}	{'precision': 0.8333333333333334, 'recall': 0.6474820143884892, 'f1-score': 0.7287449392712549, 'support': 139.0}	{'precision': 0.8323863636363636, 'recall': 0.9257503949447078, 'f1-score': 0.8765893792071803, 'support': 633.0}	{'precision': 0.6355166572557877, 'recall': 0.5626093476630842, 'f1-score': 0.5968447567280923, 'support': 4001.0}	{'precision': 0.8084507042253521, 'recall': 0.8554396423248882, 'f1-score': 0.831281679942071, 'support': 2013.0}	{'precision': 0.8828767123287671, 'recall': 0.9096683133380381, 'f1-score': 0.8960722975321516, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9998231966053748, 'f1-score': 0.9999115904871364, 'support': 11312.0}	0.8890	{'precision': 0.7973297656463132, 'recall': 0.7718025611021095, 'f1-score': 0.7816110446415138, 'support': 29705.0}	{'precision': 0.8851266624914956, 'recall': 0.8889749200471301, 'f1-score': 0.886491656178783, 'support': 29705.0}
No log	4.0	164	0.2729	{'precision': 0.6325757575757576, 'recall': 0.6162361623616236, 'f1-score': 0.6242990654205607, 'support': 271.0}	{'precision': 0.75, 'recall': 0.8848920863309353, 'f1-score': 0.8118811881188119, 'support': 139.0}	{'precision': 0.8959349593495934, 'recall': 0.8704581358609794, 'f1-score': 0.8830128205128205, 'support': 633.0}	{'precision': 0.6295979469632165, 'recall': 0.5518620344913772, 'f1-score': 0.5881726158763986, 'support': 4001.0}	{'precision': 0.7551271534044298, 'recall': 0.914555389965226, 'f1-score': 0.8272298359919119, 'support': 2013.0}	{'precision': 0.8926786497150373, 'recall': 0.8981122088920254, 'f1-score': 0.8953871861395717, 'support': 11336.0}	{'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}	0.8882	{'precision': 0.7937020667154335, 'recall': 0.8194451454145952, 'f1-score': 0.8042832445800107, 'support': 29705.0}	{'precision': 0.8858206489932313, 'recall': 0.8881669752566907, 'f1-score': 0.886099481074347, 'support': 29705.0}
No log	5.0	205	0.2636	{'precision': 0.6716417910447762, 'recall': 0.6642066420664207, 'f1-score': 0.6679035250463822, 'support': 271.0}	{'precision': 0.8116883116883117, 'recall': 0.8992805755395683, 'f1-score': 0.8532423208191127, 'support': 139.0}	{'precision': 0.8971061093247589, 'recall': 0.8815165876777251, 'f1-score': 0.8892430278884461, 'support': 633.0}	{'precision': 0.6369410569105691, 'recall': 0.6265933516620845, 'f1-score': 0.6317248330603503, 'support': 4001.0}	{'precision': 0.836166194523135, 'recall': 0.8797814207650273, 'f1-score': 0.8574195110142822, 'support': 2013.0}	{'precision': 0.8971772409521281, 'recall': 0.8944071983062809, 'f1-score': 0.8957900781905729, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9994695898161244, 'f1-score': 0.9997347245556637, 'support': 11312.0}	0.8950	{'precision': 0.82153152920624, 'recall': 0.8350364808333187, 'f1-score': 0.827865431510687, 'support': 29705.0}	{'precision': 0.8946881901904717, 'recall': 0.8950008416091567, 'f1-score': 0.8947882658752071, 'support': 29705.0}
No log	6.0	246	0.2841	{'precision': 0.6382252559726962, 'recall': 0.6900369003690037, 'f1-score': 0.6631205673758864, 'support': 271.0}	{'precision': 0.8671875, 'recall': 0.7985611510791367, 'f1-score': 0.8314606741573035, 'support': 139.0}	{'precision': 0.8903225806451613, 'recall': 0.8720379146919431, 'f1-score': 0.8810853950518756, 'support': 633.0}	{'precision': 0.6216798277099784, 'recall': 0.6493376655836041, 'f1-score': 0.6352078239608802, 'support': 4001.0}	{'precision': 0.9020251778872469, 'recall': 0.8186785891703925, 'f1-score': 0.8583333333333334, 'support': 2013.0}	{'precision': 0.891806167400881, 'recall': 0.8929075511644319, 'f1-score': 0.8923565194393016, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9996463932107497, 'f1-score': 0.9998231653404068, 'support': 11312.0}	0.8930	{'precision': 0.8301780728022805, 'recall': 0.8173151664670374, 'f1-score': 0.823055354094141, 'support': 29705.0}	{'precision': 0.8948562426469627, 'recall': 0.8929809796330584, 'f1-score': 0.8937232929438362, 'support': 29705.0}
No log	7.0	287	0.3034	{'precision': 0.6413043478260869, 'recall': 0.6531365313653137, 'f1-score': 0.6471663619744058, 'support': 271.0}	{'precision': 0.9051724137931034, 'recall': 0.7553956834532374, 'f1-score': 0.8235294117647058, 'support': 139.0}	{'precision': 0.8725038402457758, 'recall': 0.8973143759873617, 'f1-score': 0.8847352024922119, 'support': 633.0}	{'precision': 0.6278366111951589, 'recall': 0.6223444138965258, 'f1-score': 0.6250784486004769, 'support': 4001.0}	{'precision': 0.9178082191780822, 'recall': 0.7655240933929458, 'f1-score': 0.8347778981581798, 'support': 2013.0}	{'precision': 0.8827297574308165, 'recall': 0.911697247706422, 'f1-score': 0.8969796910258636, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0}	0.8929	{'precision': 0.8353364556670034, 'recall': 0.8007353058156956, 'f1-score': 0.816019199126301, 'support': 29705.0}	{'precision': 0.8931174594003277, 'recall': 0.89294731526679, 'f1-score': 0.8924389300080743, 'support': 29705.0}
No log	8.0	328	0.3248	{'precision': 0.6555555555555556, 'recall': 0.6531365313653137, 'f1-score': 0.6543438077634011, 'support': 271.0}	{'precision': 0.8450704225352113, 'recall': 0.8633093525179856, 'f1-score': 0.8540925266903915, 'support': 139.0}	{'precision': 0.884310618066561, 'recall': 0.8815165876777251, 'f1-score': 0.8829113924050632, 'support': 633.0}	{'precision': 0.6295746785361028, 'recall': 0.6363409147713072, 'f1-score': 0.6329397141081416, 'support': 4001.0}	{'precision': 0.8876518218623481, 'recall': 0.8713363139592648, 'f1-score': 0.8794184006016544, 'support': 2013.0}	{'precision': 0.8911930815390046, 'recall': 0.890878616796048, 'f1-score': 0.8910358214222692, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9998231966053748, 'f1-score': 0.9999115904871364, 'support': 11312.0}	0.8943	{'precision': 0.8276223111563975, 'recall': 0.8280487876704313, 'f1-score': 0.8278076076397225, 'support': 29705.0}	{'precision': 0.894638112913048, 'recall': 0.894260225551254, 'f1-score': 0.8944410693249414, 'support': 29705.0}
No log	9.0	369	0.3427	{'precision': 0.6554307116104869, 'recall': 0.6457564575645757, 'f1-score': 0.650557620817844, 'support': 271.0}	{'precision': 0.896, 'recall': 0.8057553956834532, 'f1-score': 0.8484848484848485, 'support': 139.0}	{'precision': 0.8725038402457758, 'recall': 0.8973143759873617, 'f1-score': 0.8847352024922119, 'support': 633.0}	{'precision': 0.6360742705570291, 'recall': 0.5993501624593851, 'f1-score': 0.6171663878522712, 'support': 4001.0}	{'precision': 0.907469342251951, 'recall': 0.8087431693989071, 'f1-score': 0.8552666141318623, 'support': 2013.0}	{'precision': 0.8780322307039864, 'recall': 0.913196894848271, 'f1-score': 0.8952693937559457, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9996463932107497, 'f1-score': 0.9998231653404068, 'support': 11312.0}	0.8935	{'precision': 0.8350729136241756, 'recall': 0.809966121307529, 'f1-score': 0.8216147475536272, 'support': 29705.0}	{'precision': 0.8918196587589381, 'recall': 0.893485945127083, 'f1-score': 0.8922398854856441, 'support': 29705.0}
No log	10.0	410	0.3490	{'precision': 0.6679841897233202, 'recall': 0.6236162361623616, 'f1-score': 0.6450381679389313, 'support': 271.0}	{'precision': 0.851063829787234, 'recall': 0.8633093525179856, 'f1-score': 0.8571428571428571, 'support': 139.0}	{'precision': 0.8736517719568567, 'recall': 0.8957345971563981, 'f1-score': 0.8845553822152886, 'support': 633.0}	{'precision': 0.6537938536850694, 'recall': 0.6008497875531117, 'f1-score': 0.6262047408179214, 'support': 4001.0}	{'precision': 0.9007751937984496, 'recall': 0.8658718330849479, 'f1-score': 0.8829787234042552, 'support': 2013.0}	{'precision': 0.8802487011327825, 'recall': 0.911697247706422, 'f1-score': 0.8956970143432854, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0}	0.8971	{'precision': 0.8325025057262447, 'recall': 0.8229734070127556, 'f1-score': 0.8273548951044003, 'support': 29705.0}	{'precision': 0.8945272549791482, 'recall': 0.8970543679515233, 'f1-score': 0.8955015738591867, 'support': 29705.0}

Framework versions

Transformers 4.37.2
Pytorch 2.2.0+cu121
Datasets 2.17.0
Tokenizers 0.15.2