meta_data/README_s42_e14.md

metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-sep_tok_full_labels
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: sep_tok_full_labels
          split: train[80%:100%]
          args: sep_tok_full_labels
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.89294731526679

longformer-sep_tok_full_labels

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

• Loss: 0.4381
• B-claim: {'precision': 0.6666666666666666, 'recall': 0.6125461254612546, 'f1-score': 0.6384615384615385, 'support': 271.0}
• B-majorclaim: {'precision': 0.8776978417266187, 'recall': 0.8776978417266187, 'f1-score': 0.8776978417266187, 'support': 139.0}
• B-premise: {'precision': 0.8641221374045801, 'recall': 0.8941548183254344, 'f1-score': 0.8788819875776396, 'support': 633.0}
• I-claim: {'precision': 0.643542019176537, 'recall': 0.5703574106473381, 'f1-score': 0.6047436067311515, 'support': 4001.0}
• I-majorclaim: {'precision': 0.9049608355091384, 'recall': 0.8609041231992052, 'f1-score': 0.8823828920570266, 'support': 2013.0}
• I-premise: {'precision': 0.8700479434771637, 'recall': 0.9124911785462244, 'f1-score': 0.8907642626480086, 'support': 11336.0}
• O: {'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}
• Accuracy: 0.8929
• Macro avg: {'precision': 0.8324339205658149, 'recall': 0.818307356843725, 'f1-score': 0.8247045898859976, 'support': 29705.0}
• Weighted avg: {'precision': 0.8894468018012494, 'recall': 0.89294731526679, 'f1-score': 0.8906539299336704, 'support': 29705.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 14

Training results

Training Loss	Epoch	Step	Validation Loss	B-claim	B-majorclaim	B-premise	I-claim	I-majorclaim	I-premise	O	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.3796	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 271.0}	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 139.0}	{'precision': 0.6347736625514403, 'recall': 0.9747235387045814, 'f1-score': 0.768847352024922, 'support': 633.0}	{'precision': 0.4812992125984252, 'recall': 0.36665833541614595, 'f1-score': 0.41622925237622355, 'support': 4001.0}	{'precision': 0.720292504570384, 'recall': 0.587183308494784, 'f1-score': 0.6469622331691298, 'support': 2013.0}	{'precision': 0.8275591785546323, 'recall': 0.9313690896259703, 'f1-score': 0.8764007636756038, 'support': 11336.0}	{'precision': 0.9976076555023924, 'recall': 0.9953147100424328, 'f1-score': 0.9964598637047527, 'support': 11312.0}	0.8444	{'precision': 0.5230760305396106, 'recall': 0.5507498546119878, 'f1-score': 0.5292713521358046, 'support': 29705.0}	{'precision': 0.822877877018681, 'recall': 0.8444032991078942, 'f1-score': 0.8302030507730453, 'support': 29705.0}
No log	2.0	82	0.3267	{'precision': 0.3563218390804598, 'recall': 0.22878228782287824, 'f1-score': 0.27865168539325846, 'support': 271.0}	{'precision': 0.9411764705882353, 'recall': 0.11510791366906475, 'f1-score': 0.20512820512820515, 'support': 139.0}	{'precision': 0.7385159010600707, 'recall': 0.990521327014218, 'f1-score': 0.8461538461538463, 'support': 633.0}	{'precision': 0.5734567901234567, 'recall': 0.232191952011997, 'f1-score': 0.33054616616260457, 'support': 4001.0}	{'precision': 0.7658168083097262, 'recall': 0.8057625434674615, 'f1-score': 0.7852820140401839, 'support': 2013.0}	{'precision': 0.8067097342534136, 'recall': 0.9693895553987297, 'f1-score': 0.8805994070037663, 'support': 11336.0}	{'precision': 0.9995577178239717, 'recall': 0.998939179632249, 'f1-score': 0.9992483530087988, 'support': 11312.0}	0.8600	{'precision': 0.7402221801770478, 'recall': 0.6200992512880854, 'f1-score': 0.6179442395558091, 'support': 29705.0}	{'precision': 0.8410272889111762, 'recall': 0.8599562363238512, 'f1-score': 0.8358492834196086, 'support': 29705.0}
No log	3.0	123	0.2787	{'precision': 0.625531914893617, 'recall': 0.5424354243542435, 'f1-score': 0.5810276679841898, 'support': 271.0}	{'precision': 0.725609756097561, 'recall': 0.8561151079136691, 'f1-score': 0.7854785478547854, 'support': 139.0}	{'precision': 0.8738317757009346, 'recall': 0.8862559241706162, 'f1-score': 0.88, 'support': 633.0}	{'precision': 0.6308089500860585, 'recall': 0.5496125968507873, 'f1-score': 0.5874181915319888, 'support': 4001.0}	{'precision': 0.682969739619986, 'recall': 0.9642324888226528, 'f1-score': 0.7995880535530382, 'support': 2013.0}	{'precision': 0.9101449275362319, 'recall': 0.8863796753705011, 'f1-score': 0.8981051126206651, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9985855728429985, 'f1-score': 0.9992922859164897, 'support': 11312.0}	0.8857	{'precision': 0.7784138662763412, 'recall': 0.8119452557607811, 'f1-score': 0.7901299799230225, 'support': 29705.0}	{'precision': 0.8871099819139034, 'recall': 0.8857431408853729, 'f1-score': 0.8843098753493511, 'support': 29705.0}
No log	4.0	164	0.2814	{'precision': 0.6200716845878136, 'recall': 0.6383763837638377, 'f1-score': 0.629090909090909, 'support': 271.0}	{'precision': 0.6931216931216931, 'recall': 0.9424460431654677, 'f1-score': 0.798780487804878, 'support': 139.0}	{'precision': 0.9234782608695652, 'recall': 0.8388625592417062, 'f1-score': 0.8791390728476821, 'support': 633.0}	{'precision': 0.6318435754189944, 'recall': 0.5653586603349162, 'f1-score': 0.5967550455085081, 'support': 4001.0}	{'precision': 0.7173252279635258, 'recall': 0.9379036264282166, 'f1-score': 0.8129171151776103, 'support': 2013.0}	{'precision': 0.9049542272482499, 'recall': 0.8894671841919548, 'f1-score': 0.8971438740101432, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9996463932107497, 'f1-score': 0.9998231653404068, 'support': 11312.0}	0.8879	{'precision': 0.7843992384585489, 'recall': 0.8302944071909784, 'f1-score': 0.801949952825734, 'support': 29705.0}	{'precision': 0.888452719415108, 'recall': 0.8879313246928127, 'f1-score': 0.8867884480499186, 'support': 29705.0}
No log	5.0	205	0.2656	{'precision': 0.6468401486988847, 'recall': 0.6420664206642066, 'f1-score': 0.6444444444444445, 'support': 271.0}	{'precision': 0.7839506172839507, 'recall': 0.9136690647482014, 'f1-score': 0.8438538205980067, 'support': 139.0}	{'precision': 0.8937908496732027, 'recall': 0.8641390205371248, 'f1-score': 0.878714859437751, 'support': 633.0}	{'precision': 0.6425884123401053, 'recall': 0.6403399150212447, 'f1-score': 0.6414621932899348, 'support': 4001.0}	{'precision': 0.8270196257416704, 'recall': 0.9001490312965723, 'f1-score': 0.8620361560418649, 'support': 2013.0}	{'precision': 0.9015117631272922, 'recall': 0.8890261115031757, 'f1-score': 0.8952254052853653, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9993811881188119, 'f1-score': 0.9996904982977407, 'support': 11312.0}	0.8956	{'precision': 0.8136716309807294, 'recall': 0.8355386788413338, 'f1-score': 0.8236324824850155, 'support': 29705.0}	{'precision': 0.8960564388198866, 'recall': 0.8956404645682545, 'f1-score': 0.8956980225570077, 'support': 29705.0}
No log	6.0	246	0.2903	{'precision': 0.6375, 'recall': 0.7527675276752768, 'f1-score': 0.6903553299492385, 'support': 271.0}	{'precision': 0.8840579710144928, 'recall': 0.8776978417266187, 'f1-score': 0.8808664259927799, 'support': 139.0}	{'precision': 0.910958904109589, 'recall': 0.8404423380726699, 'f1-score': 0.8742810188989318, 'support': 633.0}	{'precision': 0.5979050876442924, 'recall': 0.6990752311922019, 'f1-score': 0.6445443023389792, 'support': 4001.0}	{'precision': 0.9013713080168776, 'recall': 0.8489816194734228, 'f1-score': 0.8743924277308773, 'support': 2013.0}	{'precision': 0.9049605934167826, 'recall': 0.8609738884968243, 'f1-score': 0.8824194204601963, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9992927864214993, 'f1-score': 0.9996462681287585, 'support': 11312.0}	0.8897	{'precision': 0.8338219806002906, 'recall': 0.8398901761512161, 'f1-score': 0.8352150276428231, 'support': 29705.0}	{'precision': 0.8971417448223471, 'recall': 0.8896818717387646, 'f1-score': 0.8925441999219684, 'support': 29705.0}
No log	7.0	287	0.2984	{'precision': 0.6534653465346535, 'recall': 0.7306273062730627, 'f1-score': 0.6898954703832753, 'support': 271.0}	{'precision': 0.8872180451127819, 'recall': 0.8489208633093526, 'f1-score': 0.8676470588235295, 'support': 139.0}	{'precision': 0.9029605263157895, 'recall': 0.8672985781990521, 'f1-score': 0.8847703464947623, 'support': 633.0}	{'precision': 0.6086666666666667, 'recall': 0.6845788552861785, 'f1-score': 0.6443947770850488, 'support': 4001.0}	{'precision': 0.9126898047722343, 'recall': 0.8360655737704918, 'f1-score': 0.872698988851439, 'support': 2013.0}	{'precision': 0.8997184122081933, 'recall': 0.8737649964714185, 'f1-score': 0.8865518012978295, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9996463932107497, 'f1-score': 0.9998231653404068, 'support': 11312.0}	0.8921	{'precision': 0.8378169716586169, 'recall': 0.8344146523600438, 'f1-score': 0.8351116583251846, 'support': 29705.0}	{'precision': 0.8973477280414537, 'recall': 0.8921057061100824, 'f1-score': 0.8942111320147997, 'support': 29705.0}
No log	8.0	328	0.3357	{'precision': 0.6494464944649446, 'recall': 0.6494464944649446, 'f1-score': 0.6494464944649446, 'support': 271.0}	{'precision': 0.8581560283687943, 'recall': 0.8705035971223022, 'f1-score': 0.8642857142857142, 'support': 139.0}	{'precision': 0.8811410459587956, 'recall': 0.8783570300157978, 'f1-score': 0.8797468354430379, 'support': 633.0}	{'precision': 0.6352490421455939, 'recall': 0.6215946013496626, 'f1-score': 0.6283476503284486, 'support': 4001.0}	{'precision': 0.8910485933503837, 'recall': 0.8653750620963736, 'f1-score': 0.8780241935483872, 'support': 2013.0}	{'precision': 0.8866724738675958, 'recall': 0.8979357798165137, 'f1-score': 0.8922685834502103, 'support': 11336.0}	{'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}	0.8946	{'precision': 0.8288162397365868, 'recall': 0.8261732235522278, 'f1-score': 0.8274456387886776, 'support': 29705.0}	{'precision': 0.893845392016061, 'recall': 0.8945632048476687, 'f1-score': 0.8941677755828906, 'support': 29705.0}
No log	9.0	369	0.3380	{'precision': 0.6728624535315985, 'recall': 0.6678966789667896, 'f1-score': 0.6703703703703703, 'support': 271.0}	{'precision': 0.8571428571428571, 'recall': 0.9064748201438849, 'f1-score': 0.881118881118881, 'support': 139.0}	{'precision': 0.8853503184713376, 'recall': 0.8783570300157978, 'f1-score': 0.8818398096748612, 'support': 633.0}	{'precision': 0.6546069755335763, 'recall': 0.6285928517870533, 'f1-score': 0.6413362233839093, 'support': 4001.0}	{'precision': 0.8853439680957128, 'recall': 0.8822652757078987, 'f1-score': 0.8838019407812889, 'support': 2013.0}	{'precision': 0.8904728789986092, 'recall': 0.9036697247706422, 'f1-score': 0.8970227670753064, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0}	0.8991	{'precision': 0.8351113502533846, 'recall': 0.8381415966143041, 'f1-score': 0.8364796246104117, 'support': 29705.0}	{'precision': 0.8978150414696858, 'recall': 0.8990742299276216, 'f1-score': 0.8983868008603707, 'support': 29705.0}
No log	10.0	410	0.3903	{'precision': 0.6616541353383458, 'recall': 0.6494464944649446, 'f1-score': 0.6554934823091246, 'support': 271.0}	{'precision': 0.8523489932885906, 'recall': 0.9136690647482014, 'f1-score': 0.8819444444444444, 'support': 139.0}	{'precision': 0.8807631160572337, 'recall': 0.8751974723538705, 'f1-score': 0.87797147385103, 'support': 633.0}	{'precision': 0.6461460188247266, 'recall': 0.6348412896775806, 'f1-score': 0.6404437720625316, 'support': 4001.0}	{'precision': 0.8767795778105056, 'recall': 0.8872329855936413, 'f1-score': 0.8819753086419753, 'support': 2013.0}	{'precision': 0.8930077301475755, 'recall': 0.8967889908256881, 'f1-score': 0.894894366197183, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0}	0.8974	{'precision': 0.8300999387809967, 'recall': 0.8367015846531413, 'f1-score': 0.8332271753392219, 'support': 29705.0}	{'precision': 0.8968399587143993, 'recall': 0.8974246759804747, 'f1-score': 0.8971162807281875, 'support': 29705.0}
No log	11.0	451	0.3933	{'precision': 0.6537102473498233, 'recall': 0.6826568265682657, 'f1-score': 0.667870036101083, 'support': 271.0}	{'precision': 0.8141025641025641, 'recall': 0.9136690647482014, 'f1-score': 0.8610169491525423, 'support': 139.0}	{'precision': 0.902317880794702, 'recall': 0.8609794628751974, 'f1-score': 0.8811641067097818, 'support': 633.0}	{'precision': 0.6348090887795736, 'recall': 0.6773306673331667, 'f1-score': 0.6553808948004837, 'support': 4001.0}	{'precision': 0.8714492055849783, 'recall': 0.8991554893194238, 'f1-score': 0.8850855745721272, 'support': 2013.0}	{'precision': 0.906034169392948, 'recall': 0.879498941425547, 'f1-score': 0.8925693822739482, 'support': 11336.0}	{'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}	0.8975	{'precision': 0.8260604508577984, 'recall': 0.844755778895686, 'f1-score': 0.8347267062299952, 'support': 29705.0}	{'precision': 0.9001307763431476, 'recall': 0.897458340346743, 'f1-score': 0.8985852130762765, 'support': 29705.0}
No log	12.0	492	0.4298	{'precision': 0.6666666666666666, 'recall': 0.5682656826568265, 'f1-score': 0.6135458167330677, 'support': 271.0}	{'precision': 0.8897058823529411, 'recall': 0.8705035971223022, 'f1-score': 0.88, 'support': 139.0}	{'precision': 0.8463810930576071, 'recall': 0.9052132701421801, 'f1-score': 0.8748091603053435, 'support': 633.0}	{'precision': 0.6477407847800237, 'recall': 0.5446138465383654, 'f1-score': 0.5917175831636118, 'support': 4001.0}	{'precision': 0.9117330462863293, 'recall': 0.8415300546448088, 'f1-score': 0.8752260397830018, 'support': 2013.0}	{'precision': 0.8632915567282322, 'recall': 0.923606210303458, 'f1-score': 0.8924309580634162, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0}	0.8922	{'precision': 0.8322170042674, 'recall': 0.807663465672947, 'f1-score': 0.8182407650351405, 'support': 29705.0}	{'precision': 0.8875710353600299, 'recall': 0.8921730348426191, 'f1-score': 0.8887303316490228, 'support': 29705.0}
0.1988	13.0	533	0.4540	{'precision': 0.6622807017543859, 'recall': 0.5571955719557196, 'f1-score': 0.6052104208416833, 'support': 271.0}	{'precision': 0.8768115942028986, 'recall': 0.8705035971223022, 'f1-score': 0.8736462093862816, 'support': 139.0}	{'precision': 0.8451327433628318, 'recall': 0.9052132701421801, 'f1-score': 0.8741418764302059, 'support': 633.0}	{'precision': 0.6564341085271318, 'recall': 0.5291177205698575, 'f1-score': 0.5859396623304732, 'support': 4001.0}	{'precision': 0.9048870204939569, 'recall': 0.8554396423248882, 'f1-score': 0.8794688457609805, 'support': 2013.0}	{'precision': 0.8600065455735559, 'recall': 0.9272230063514467, 'f1-score': 0.8923507937855506, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0}	0.8923	{'precision': 0.8293646734163944, 'recall': 0.8063720581098688, 'f1-score': 0.8158162293903883, 'support': 29705.0}	{'precision': 0.8868974431715715, 'recall': 0.8923076923076924, 'f1-score': 0.8880890290081059, 'support': 29705.0}
0.1988	14.0	574	0.4381	{'precision': 0.6666666666666666, 'recall': 0.6125461254612546, 'f1-score': 0.6384615384615385, 'support': 271.0}	{'precision': 0.8776978417266187, 'recall': 0.8776978417266187, 'f1-score': 0.8776978417266187, 'support': 139.0}	{'precision': 0.8641221374045801, 'recall': 0.8941548183254344, 'f1-score': 0.8788819875776396, 'support': 633.0}	{'precision': 0.643542019176537, 'recall': 0.5703574106473381, 'f1-score': 0.6047436067311515, 'support': 4001.0}	{'precision': 0.9049608355091384, 'recall': 0.8609041231992052, 'f1-score': 0.8823828920570266, 'support': 2013.0}	{'precision': 0.8700479434771637, 'recall': 0.9124911785462244, 'f1-score': 0.8907642626480086, 'support': 11336.0}	{'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}	0.8929	{'precision': 0.8324339205658149, 'recall': 0.818307356843725, 'f1-score': 0.8247045898859976, 'support': 29705.0}	{'precision': 0.8894468018012494, 'recall': 0.89294731526679, 'f1-score': 0.8906539299336704, 'support': 29705.0}

Framework versions

• Transformers 4.37.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.0
• Tokenizers 0.15.2