metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-sep_tok_full_labels
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: sep_tok_full_labels
          split: train[80%:100%]
          args: sep_tok_full_labels
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8903551590641307

longformer-sep_tok_full_labels

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

Loss: 0.3852
B-claim: {'precision': 0.6535433070866141, 'recall': 0.6125461254612546, 'f1-score': 0.6323809523809524, 'support': 271.0}
B-majorclaim: {'precision': 0.8652482269503546, 'recall': 0.8776978417266187, 'f1-score': 0.8714285714285713, 'support': 139.0}
B-premise: {'precision': 0.864406779661017, 'recall': 0.8862559241706162, 'f1-score': 0.875195007800312, 'support': 633.0}
I-claim: {'precision': 0.6240760295670539, 'recall': 0.590852286928268, 'f1-score': 0.6070098857362948, 'support': 4001.0}
I-majorclaim: {'precision': 0.8983761131482452, 'recall': 0.8519622454048683, 'f1-score': 0.874553799082101, 'support': 2013.0}
I-premise: {'precision': 0.8760940449631028, 'recall': 0.9006704304869443, 'f1-score': 0.8882122662026969, 'support': 11336.0}
O: {'precision': 1.0, 'recall': 0.9998231966053748, 'f1-score': 0.9999115904871364, 'support': 11312.0}
Accuracy: 0.8904
Macro avg: {'precision': 0.8259635001966268, 'recall': 0.8171154358262779, 'f1-score': 0.8212417247311521, 'support': 29705.0}
Weighted avg: {'precision': 0.8885140762517735, 'recall': 0.8903551590641307, 'f1-score': 0.8892576926816647, 'support': 29705.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 12

Training results

Training Loss	Epoch	Step	Validation Loss	B-claim	B-majorclaim	B-premise	I-claim	I-majorclaim	I-premise	O	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.4269	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 271.0}	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 139.0}	{'precision': 0.6157894736842106, 'recall': 0.9241706161137441, 'f1-score': 0.7391029690461149, 'support': 633.0}	{'precision': 0.4108967082860386, 'recall': 0.2714321419645089, 'f1-score': 0.32691149909692957, 'support': 4001.0}	{'precision': 0.7886393659180978, 'recall': 0.2965722801788376, 'f1-score': 0.43104693140794226, 'support': 2013.0}	{'precision': 0.7819985825655563, 'recall': 0.9733592095977417, 'f1-score': 0.8672482904975242, 'support': 11336.0}	{'precision': 0.9980435749221876, 'recall': 0.9921322489391796, 'f1-score': 0.9950791328634125, 'support': 11312.0}	0.8256	{'precision': 0.5136239579108701, 'recall': 0.4939523566848588, 'f1-score': 0.4799126889874176, 'support': 29705.0}	{'precision': 0.8004015607396266, 'recall': 0.8256185827301801, 'f1-score': 0.7988885484682058, 'support': 29705.0}
No log	2.0	82	0.3017	{'precision': 0.3691275167785235, 'recall': 0.2029520295202952, 'f1-score': 0.2619047619047619, 'support': 271.0}	{'precision': 0.8205128205128205, 'recall': 0.2302158273381295, 'f1-score': 0.3595505617977528, 'support': 139.0}	{'precision': 0.740521327014218, 'recall': 0.9873617693522907, 'f1-score': 0.8463100880162492, 'support': 633.0}	{'precision': 0.5660936573799644, 'recall': 0.23869032741814547, 'f1-score': 0.3357946554149086, 'support': 4001.0}	{'precision': 0.7739499764039641, 'recall': 0.8147044212617983, 'f1-score': 0.7938044530493708, 'support': 2013.0}	{'precision': 0.8062661260597125, 'recall': 0.964802399435427, 'f1-score': 0.8784386169230152, 'support': 11336.0}	{'precision': 0.9999115200849407, 'recall': 0.9990275813295615, 'f1-score': 0.9994693552666489, 'support': 11312.0}	0.8600	{'precision': 0.7251975634620206, 'recall': 0.6339649079508068, 'f1-score': 0.6393246417675295, 'support': 29705.0}	{'precision': 0.8401470723780652, 'recall': 0.8599562363238512, 'f1-score': 0.8369664464837379, 'support': 29705.0}
No log	3.0	123	0.2706	{'precision': 0.6294820717131474, 'recall': 0.5830258302583026, 'f1-score': 0.6053639846743295, 'support': 271.0}	{'precision': 0.7352941176470589, 'recall': 0.8992805755395683, 'f1-score': 0.8090614886731392, 'support': 139.0}	{'precision': 0.887459807073955, 'recall': 0.8720379146919431, 'f1-score': 0.8796812749003984, 'support': 633.0}	{'precision': 0.6167864476386037, 'recall': 0.6005998500374906, 'f1-score': 0.6085855388122071, 'support': 4001.0}	{'precision': 0.7098312545854732, 'recall': 0.9612518628912071, 'f1-score': 0.8166279805866218, 'support': 2013.0}	{'precision': 0.9201155853840418, 'recall': 0.8707657021877205, 'f1-score': 0.8947606961566353, 'support': 11336.0}	{'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}	0.8873	{'precision': 0.7855670405774686, 'recall': 0.826708819372319, 'f1-score': 0.8020115662576188, 'support': 29705.0}	{'precision': 0.8912183386655823, 'recall': 0.8872580373674466, 'f1-score': 0.8876344571999458, 'support': 29705.0}
No log	4.0	164	0.2668	{'precision': 0.6441947565543071, 'recall': 0.6346863468634686, 'f1-score': 0.6394052044609665, 'support': 271.0}	{'precision': 0.7716049382716049, 'recall': 0.8992805755395683, 'f1-score': 0.8305647840531561, 'support': 139.0}	{'precision': 0.8972267536704731, 'recall': 0.8688783570300158, 'f1-score': 0.882825040128411, 'support': 633.0}	{'precision': 0.6395285087719298, 'recall': 0.583104223944014, 'f1-score': 0.6100143809648318, 'support': 4001.0}	{'precision': 0.7777310924369748, 'recall': 0.9195230998509687, 'f1-score': 0.8427043022991123, 'support': 2013.0}	{'precision': 0.8976329270446918, 'recall': 0.8965243472124206, 'f1-score': 0.897078294642069, 'support': 11336.0}	{'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0}	0.8923	{'precision': 0.8039757975524056, 'recall': 0.8288567072057794, 'f1-score': 0.8146496868076456, 'support': 29705.0}	{'precision': 0.8907816058765868, 'recall': 0.8923076923076924, 'f1-score': 0.8909397890834521, 'support': 29705.0}
No log	5.0	205	0.2914	{'precision': 0.6290322580645161, 'recall': 0.7195571955719557, 'f1-score': 0.6712564543889845, 'support': 271.0}	{'precision': 0.7839506172839507, 'recall': 0.9136690647482014, 'f1-score': 0.8438538205980067, 'support': 139.0}	{'precision': 0.9229422066549913, 'recall': 0.8325434439178515, 'f1-score': 0.8754152823920265, 'support': 633.0}	{'precision': 0.5971382507053608, 'recall': 0.7405648587853036, 'f1-score': 0.6611625571795158, 'support': 4001.0}	{'precision': 0.8306781975421028, 'recall': 0.9066070541480378, 'f1-score': 0.8669833729216152, 'support': 2013.0}	{'precision': 0.9331241419886253, 'recall': 0.8394495412844036, 'f1-score': 0.8838116466982447, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9993811881188119, 'f1-score': 0.9996904982977407, 'support': 11312.0}	0.8907	{'precision': 0.8138379531770781, 'recall': 0.8502531923677951, 'f1-score': 0.8288819474965906, 'support': 29705.0}	{'precision': 0.9027052658090564, 'recall': 0.8906918027268137, 'f1-score': 0.8945053263301965, 'support': 29705.0}
No log	6.0	246	0.2926	{'precision': 0.6426116838487973, 'recall': 0.6900369003690037, 'f1-score': 0.6654804270462633, 'support': 271.0}	{'precision': 0.8805970149253731, 'recall': 0.8489208633093526, 'f1-score': 0.8644688644688644, 'support': 139.0}	{'precision': 0.8883495145631068, 'recall': 0.8672985781990521, 'f1-score': 0.8776978417266188, 'support': 633.0}	{'precision': 0.6389367462466158, 'recall': 0.6488377905523619, 'f1-score': 0.6438492063492063, 'support': 4001.0}	{'precision': 0.9025026068821689, 'recall': 0.8599105812220567, 'f1-score': 0.8806919358941745, 'support': 2013.0}	{'precision': 0.8926310235666549, 'recall': 0.8954657727593507, 'f1-score': 0.8940461511361635, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0}	0.8969	{'precision': 0.8350897985761023, 'recall': 0.8300293259027484, 'f1-score': 0.8323002580699363, 'support': 29705.0}	{'precision': 0.8975884513265179, 'recall': 0.8968523817539135, 'f1-score': 0.8971678038583121, 'support': 29705.0}
No log	7.0	287	0.3022	{'precision': 0.6482758620689655, 'recall': 0.6937269372693727, 'f1-score': 0.6702317290552585, 'support': 271.0}	{'precision': 0.8863636363636364, 'recall': 0.841726618705036, 'f1-score': 0.8634686346863468, 'support': 139.0}	{'precision': 0.8903225806451613, 'recall': 0.8720379146919431, 'f1-score': 0.8810853950518756, 'support': 633.0}	{'precision': 0.6188143067123959, 'recall': 0.6313421644588852, 'f1-score': 0.6250154645552394, 'support': 4001.0}	{'precision': 0.909915014164306, 'recall': 0.7978142076502732, 'f1-score': 0.8501852832186342, 'support': 2013.0}	{'precision': 0.8855180806675939, 'recall': 0.8986414961185604, 'f1-score': 0.892031523642732, 'support': 11336.0}	{'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}	0.8917	{'precision': 0.8341727829460084, 'recall': 0.8193270484134387, 'f1-score': 0.826002575744298, 'support': 29705.0}	{'precision': 0.8927867168739442, 'recall': 0.8917017337148628, 'f1-score': 0.8919562765420584, 'support': 29705.0}
No log	8.0	328	0.3373	{'precision': 0.6413043478260869, 'recall': 0.6531365313653137, 'f1-score': 0.6471663619744058, 'support': 271.0}	{'precision': 0.8581560283687943, 'recall': 0.8705035971223022, 'f1-score': 0.8642857142857142, 'support': 139.0}	{'precision': 0.8801916932907349, 'recall': 0.8704581358609794, 'f1-score': 0.8752978554408261, 'support': 633.0}	{'precision': 0.626970227670753, 'recall': 0.6263434141464633, 'f1-score': 0.6266566641660415, 'support': 4001.0}	{'precision': 0.8939316675165732, 'recall': 0.8708395429706905, 'f1-score': 0.8822345244086564, 'support': 2013.0}	{'precision': 0.8870259831460674, 'recall': 0.891407904022583, 'f1-score': 0.8892115452305527, 'support': 11336.0}	{'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}	0.8929	{'precision': 0.8267971354027156, 'recall': 0.8260984464983332, 'f1-score': 0.8264075236437424, 'support': 29705.0}	{'precision': 0.8929660791322179, 'recall': 0.89294731526679, 'f1-score': 0.8929431014499688, 'support': 29705.0}
No log	9.0	369	0.3516	{'precision': 0.6514657980456026, 'recall': 0.7380073800738007, 'f1-score': 0.6920415224913494, 'support': 271.0}	{'precision': 0.8671328671328671, 'recall': 0.8920863309352518, 'f1-score': 0.8794326241134752, 'support': 139.0}	{'precision': 0.9089376053962901, 'recall': 0.8515007898894155, 'f1-score': 0.8792822185970636, 'support': 633.0}	{'precision': 0.6081597960050998, 'recall': 0.7153211697075731, 'f1-score': 0.6574020902721948, 'support': 4001.0}	{'precision': 0.8942753996905621, 'recall': 0.8614008941877794, 'f1-score': 0.8775303643724697, 'support': 2013.0}	{'precision': 0.9124871579340619, 'recall': 0.8618560338743825, 'f1-score': 0.8864492129020551, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9998231966053748, 'f1-score': 0.9999115904871364, 'support': 11312.0}	0.8934	{'precision': 0.8346369463149262, 'recall': 0.8457136850390825, 'f1-score': 0.8388642318908206, 'support': 29705.0}	{'precision': 0.9009195202744221, 'recall': 0.8934186163945463, 'f1-score': 0.8962428078582019, 'support': 29705.0}
No log	10.0	410	0.3603	{'precision': 0.6612903225806451, 'recall': 0.6051660516605166, 'f1-score': 0.631984585741811, 'support': 271.0}	{'precision': 0.8482758620689655, 'recall': 0.8848920863309353, 'f1-score': 0.8661971830985916, 'support': 139.0}	{'precision': 0.8661538461538462, 'recall': 0.8894154818325435, 'f1-score': 0.8776305533904911, 'support': 633.0}	{'precision': 0.6538879825992387, 'recall': 0.6010997250687328, 'f1-score': 0.6263836437036072, 'support': 4001.0}	{'precision': 0.8966733870967742, 'recall': 0.8837555886736215, 'f1-score': 0.8901676257192895, 'support': 2013.0}	{'precision': 0.8804617357845232, 'recall': 0.9083450952717008, 'f1-score': 0.8941860969996961, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9993811881188119, 'f1-score': 0.9996904982977407, 'support': 11312.0}	0.8967	{'precision': 0.8295347337548561, 'recall': 0.8245793167081231, 'f1-score': 0.8266057409930324, 'support': 29705.0}	{'precision': 0.8941093752001698, 'recall': 0.8966840599225719, 'f1-score': 0.8951445976383295, 'support': 29705.0}
No log	11.0	451	0.3775	{'precision': 0.6398601398601399, 'recall': 0.6752767527675276, 'f1-score': 0.6570915619389587, 'support': 271.0}	{'precision': 0.8333333333333334, 'recall': 0.8992805755395683, 'f1-score': 0.8650519031141869, 'support': 139.0}	{'precision': 0.8945634266886326, 'recall': 0.8578199052132701, 'f1-score': 0.8758064516129032, 'support': 633.0}	{'precision': 0.6256672081689487, 'recall': 0.6738315421144714, 'f1-score': 0.6488567990373044, 'support': 4001.0}	{'precision': 0.8632189463508941, 'recall': 0.8872329855936413, 'f1-score': 0.875061244487996, 'support': 2013.0}	{'precision': 0.9064892453518046, 'recall': 0.8773817925194072, 'f1-score': 0.8916980455441994, 'support': 11336.0}	{'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}	0.8952	{'precision': 0.8233046142505361, 'recall': 0.8386890791068409, 'f1-score': 0.8305094293907926, 'support': 29705.0}	{'precision': 0.8983137655117223, 'recall': 0.8951691634404982, 'f1-score': 0.8965010231974729, 'support': 29705.0}
No log	12.0	492	0.3852	{'precision': 0.6535433070866141, 'recall': 0.6125461254612546, 'f1-score': 0.6323809523809524, 'support': 271.0}	{'precision': 0.8652482269503546, 'recall': 0.8776978417266187, 'f1-score': 0.8714285714285713, 'support': 139.0}	{'precision': 0.864406779661017, 'recall': 0.8862559241706162, 'f1-score': 0.875195007800312, 'support': 633.0}	{'precision': 0.6240760295670539, 'recall': 0.590852286928268, 'f1-score': 0.6070098857362948, 'support': 4001.0}	{'precision': 0.8983761131482452, 'recall': 0.8519622454048683, 'f1-score': 0.874553799082101, 'support': 2013.0}	{'precision': 0.8760940449631028, 'recall': 0.9006704304869443, 'f1-score': 0.8882122662026969, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9998231966053748, 'f1-score': 0.9999115904871364, 'support': 11312.0}	0.8904	{'precision': 0.8259635001966268, 'recall': 0.8171154358262779, 'f1-score': 0.8212417247311521, 'support': 29705.0}	{'precision': 0.8885140762517735, 'recall': 0.8903551590641307, 'f1-score': 0.8892576926816647, 'support': 29705.0}

Framework versions

Transformers 4.37.2
Pytorch 2.2.0+cu121
Datasets 2.17.0
Tokenizers 0.15.2