meta_data/README_s42_e15.md

metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-sep_tok_full_labels
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: sep_tok_full_labels
          split: train[80%:100%]
          args: sep_tok_full_labels
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8972563541491332

longformer-sep_tok_full_labels

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

• Loss: 0.4731
• B-claim: {'precision': 0.678714859437751, 'recall': 0.6236162361623616, 'f1-score': 0.65, 'support': 271.0}
• B-majorclaim: {'precision': 0.8732394366197183, 'recall': 0.8920863309352518, 'f1-score': 0.8825622775800712, 'support': 139.0}
• B-premise: {'precision': 0.8680981595092024, 'recall': 0.8941548183254344, 'f1-score': 0.8809338521400777, 'support': 633.0}
• I-claim: {'precision': 0.6616605270614905, 'recall': 0.5836040989752562, 'f1-score': 0.6201859229747676, 'support': 4001.0}
• I-majorclaim: {'precision': 0.9088082901554404, 'recall': 0.8713363139592648, 'f1-score': 0.8896779102206441, 'support': 2013.0}
• I-premise: {'precision': 0.8740223698595576, 'recall': 0.9168136908962597, 'f1-score': 0.8949067895122056, 'support': 11336.0}
• O: {'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}
• Accuracy: 0.8973
• Macro avg: {'precision': 0.8377919489490228, 'recall': 0.8259444984648326, 'f1-score': 0.831180964632538, 'support': 29705.0}
• Weighted avg: {'precision': 0.8938384307406486, 'recall': 0.8972563541491332, 'f1-score': 0.8949808504290477, 'support': 29705.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 15

Training results

Training Loss	Epoch	Step	Validation Loss	B-claim	B-majorclaim	B-premise	I-claim	I-majorclaim	I-premise	O	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.4045	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 271.0}	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 139.0}	{'precision': 0.6537250786988458, 'recall': 0.9842022116903634, 'f1-score': 0.7856242118537201, 'support': 633.0}	{'precision': 0.47217068645640076, 'recall': 0.5088727818045489, 'f1-score': 0.4898351978828341, 'support': 4001.0}	{'precision': 0.6434329065908013, 'recall': 0.6741182314952807, 'f1-score': 0.658418243571082, 'support': 2013.0}	{'precision': 0.874659894794123, 'recall': 0.8507410021171489, 'f1-score': 0.8625346570074233, 'support': 11336.0}	{'precision': 0.9963732861565678, 'recall': 0.9957567185289957, 'f1-score': 0.9960649069284166, 'support': 11312.0}	0.8391	{'precision': 0.5200516932423913, 'recall': 0.5733844208051911, 'f1-score': 0.541782459606211, 'support': 29705.0}	{'precision': 0.8343481741351619, 'recall': 0.8390506648712338, 'f1-score': 0.8358089808500795, 'support': 29705.0}
No log	2.0	82	0.2974	{'precision': 0.437125748502994, 'recall': 0.2693726937269373, 'f1-score': 0.33333333333333337, 'support': 271.0}	{'precision': 0.84, 'recall': 0.302158273381295, 'f1-score': 0.4444444444444445, 'support': 139.0}	{'precision': 0.7591687041564792, 'recall': 0.981042654028436, 'f1-score': 0.855961405926947, 'support': 633.0}	{'precision': 0.59072375127421, 'recall': 0.2896775806048488, 'f1-score': 0.3887305047794734, 'support': 4001.0}	{'precision': 0.7283744254074384, 'recall': 0.8658718330849479, 'f1-score': 0.7911938266000907, 'support': 2013.0}	{'precision': 0.8273635664873175, 'recall': 0.9495412844036697, 'f1-score': 0.8842520331882034, 'support': 11336.0}	{'precision': 0.9998230871295887, 'recall': 0.9992043847241867, 'f1-score': 0.9995136401821638, 'support': 11312.0}	0.8653	{'precision': 0.740368468994004, 'recall': 0.6652669577077601, 'f1-score': 0.6710613126363796, 'support': 29705.0}	{'precision': 0.8495024563568015, 'recall': 0.86534253492678, 'f1-score': 0.8474094579900232, 'support': 29705.0}
No log	3.0	123	0.2611	{'precision': 0.5560538116591929, 'recall': 0.4575645756457565, 'f1-score': 0.5020242914979757, 'support': 271.0}	{'precision': 0.8165137614678899, 'recall': 0.6402877697841727, 'f1-score': 0.717741935483871, 'support': 139.0}	{'precision': 0.819971870604782, 'recall': 0.9210110584518167, 'f1-score': 0.8675595238095238, 'support': 633.0}	{'precision': 0.6249006622516556, 'recall': 0.5896025993501625, 'f1-score': 0.6067386831275721, 'support': 4001.0}	{'precision': 0.7860696517412935, 'recall': 0.8633879781420765, 'f1-score': 0.8229166666666666, 'support': 2013.0}	{'precision': 0.8943110876637651, 'recall': 0.8972300635144672, 'f1-score': 0.8957681976308953, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9992043847241867, 'f1-score': 0.9996020340481981, 'support': 11312.0}	0.8876	{'precision': 0.7854029779126541, 'recall': 0.7668983470875198, 'f1-score': 0.7731930474663862, 'support': 29705.0}	{'precision': 0.8859021861059433, 'recall': 0.8876283453963979, 'f1-score': 0.8864163982255546, 'support': 29705.0}
No log	4.0	164	0.2731	{'precision': 0.6354679802955665, 'recall': 0.47601476014760147, 'f1-score': 0.5443037974683544, 'support': 271.0}	{'precision': 0.7345679012345679, 'recall': 0.8561151079136691, 'f1-score': 0.7906976744186047, 'support': 139.0}	{'precision': 0.8468335787923417, 'recall': 0.9083728278041074, 'f1-score': 0.8765243902439023, 'support': 633.0}	{'precision': 0.6369000786782061, 'recall': 0.4046488377905524, 'f1-score': 0.4948800244536146, 'support': 4001.0}	{'precision': 0.8040909090909091, 'recall': 0.8787878787878788, 'f1-score': 0.8397816282933775, 'support': 2013.0}	{'precision': 0.8434451582203188, 'recall': 0.9381616090331687, 'f1-score': 0.8882856546251827, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9998231966053748, 'f1-score': 0.9999115904871364, 'support': 11312.0}	0.8805	{'precision': 0.7859008009017014, 'recall': 0.7802748882974789, 'f1-score': 0.7763406799985961, 'support': 29705.0}	{'precision': 0.870241674623272, 'recall': 0.8805251641137856, 'f1-score': 0.8706734466553624, 'support': 29705.0}
No log	5.0	205	0.2609	{'precision': 0.6553191489361702, 'recall': 0.5682656826568265, 'f1-score': 0.608695652173913, 'support': 271.0}	{'precision': 0.8571428571428571, 'recall': 0.8201438848920863, 'f1-score': 0.8382352941176471, 'support': 139.0}	{'precision': 0.8518518518518519, 'recall': 0.9083728278041074, 'f1-score': 0.8792048929663608, 'support': 633.0}	{'precision': 0.6405764340209099, 'recall': 0.5666083479130217, 'f1-score': 0.6013262599469497, 'support': 4001.0}	{'precision': 0.867027027027027, 'recall': 0.7968206656731247, 'f1-score': 0.8304426611441884, 'support': 2013.0}	{'precision': 0.8750939928147715, 'recall': 0.9239590684544813, 'f1-score': 0.8988629049560181, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9992927864214993, 'f1-score': 0.9996462681287585, 'support': 11312.0}	0.8918	{'precision': 0.8210016159705126, 'recall': 0.7976376091164497, 'f1-score': 0.8080591333476909, 'support': 29705.0}	{'precision': 0.8879412149199831, 'recall': 0.8918363911799361, 'f1-score': 0.8891804020688141, 'support': 29705.0}
No log	6.0	246	0.2863	{'precision': 0.6537102473498233, 'recall': 0.6826568265682657, 'f1-score': 0.667870036101083, 'support': 271.0}	{'precision': 0.8461538461538461, 'recall': 0.8705035971223022, 'f1-score': 0.8581560283687943, 'support': 139.0}	{'precision': 0.8946515397082658, 'recall': 0.8720379146919431, 'f1-score': 0.8832, 'support': 633.0}	{'precision': 0.604336675215668, 'recall': 0.6478380404898775, 'f1-score': 0.6253317249698431, 'support': 4001.0}	{'precision': 0.8833592534992224, 'recall': 0.8464977645305514, 'f1-score': 0.8645357686453577, 'support': 2013.0}	{'precision': 0.8935501257635645, 'recall': 0.877470007057163, 'f1-score': 0.8854370660494927, 'support': 11336.0}	{'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}	0.8892	{'precision': 0.8251088125271986, 'recall': 0.8281434500657289, 'f1-score': 0.8263615177335101, 'support': 29705.0}	{'precision': 0.892055974642993, 'recall': 0.88917690624474, 'f1-score': 0.8904534382208771, 'support': 29705.0}
No log	7.0	287	0.3057	{'precision': 0.6242038216560509, 'recall': 0.7232472324723247, 'f1-score': 0.6700854700854701, 'support': 271.0}	{'precision': 0.8682170542635659, 'recall': 0.8057553956834532, 'f1-score': 0.835820895522388, 'support': 139.0}	{'precision': 0.9033333333333333, 'recall': 0.8562401263823065, 'f1-score': 0.8791565287915653, 'support': 633.0}	{'precision': 0.599781181619256, 'recall': 0.6850787303174206, 'f1-score': 0.6395986465989966, 'support': 4001.0}	{'precision': 0.900281690140845, 'recall': 0.793840039741679, 'f1-score': 0.843717001055966, 'support': 2013.0}	{'precision': 0.9015991277485008, 'recall': 0.8753528581510233, 'f1-score': 0.8882821591621162, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0}	0.8895	{'precision': 0.8282023155373646, 'recall': 0.819917997292985, 'f1-score': 0.8223737854877207, 'support': 29705.0}	{'precision': 0.8956798743740312, 'recall': 0.889513549907423, 'f1-score': 0.8918626962438791, 'support': 29705.0}
No log	8.0	328	0.3326	{'precision': 0.6555555555555556, 'recall': 0.6531365313653137, 'f1-score': 0.6543438077634011, 'support': 271.0}	{'precision': 0.9015151515151515, 'recall': 0.8561151079136691, 'f1-score': 0.8782287822878229, 'support': 139.0}	{'precision': 0.8736349453978159, 'recall': 0.8846761453396524, 'f1-score': 0.8791208791208791, 'support': 633.0}	{'precision': 0.6485290288987243, 'recall': 0.6225943514121469, 'f1-score': 0.635297118082122, 'support': 4001.0}	{'precision': 0.9106474050294275, 'recall': 0.8455042225534029, 'f1-score': 0.8768675940236991, 'support': 2013.0}	{'precision': 0.8846319044755605, 'recall': 0.9084333098094566, 'f1-score': 0.8963746355050702, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0}	0.8974	{'precision': 0.8392162844103194, 'recall': 0.8243387523851898, 'f1-score': 0.8314555162829338, 'support': 29705.0}	{'precision': 0.8962821821961274, 'recall': 0.8974246759804747, 'f1-score': 0.8966721283058662, 'support': 29705.0}
No log	9.0	369	0.3502	{'precision': 0.6589147286821705, 'recall': 0.6273062730627307, 'f1-score': 0.6427221172022685, 'support': 271.0}	{'precision': 0.8705035971223022, 'recall': 0.8705035971223022, 'f1-score': 0.8705035971223022, 'support': 139.0}	{'precision': 0.871517027863777, 'recall': 0.8894154818325435, 'f1-score': 0.8803752931978107, 'support': 633.0}	{'precision': 0.6614194973764154, 'recall': 0.5986003499125219, 'f1-score': 0.628443977958541, 'support': 4001.0}	{'precision': 0.8998435054773083, 'recall': 0.856929955290611, 'f1-score': 0.8778625954198472, 'support': 2013.0}	{'precision': 0.8803013629052738, 'recall': 0.9173429781227946, 'f1-score': 0.898440537388224, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0}	0.8983	{'precision': 0.8346428170610353, 'recall': 0.8228586048065988, 'f1-score': 0.8283291307837909, 'support': 29705.0}	{'precision': 0.895474055059205, 'recall': 0.8982999495034506, 'f1-score': 0.8964894436788355, 'support': 29705.0}
No log	10.0	410	0.4286	{'precision': 0.6652542372881356, 'recall': 0.5793357933579336, 'f1-score': 0.6193293885601577, 'support': 271.0}	{'precision': 0.8714285714285714, 'recall': 0.8776978417266187, 'f1-score': 0.8745519713261649, 'support': 139.0}	{'precision': 0.856071964017991, 'recall': 0.9020537124802528, 'f1-score': 0.8784615384615385, 'support': 633.0}	{'precision': 0.6511834319526627, 'recall': 0.5501124718820295, 'f1-score': 0.5963961522828884, 'support': 4001.0}	{'precision': 0.9091852620653866, 'recall': 0.8703427719821163, 'f1-score': 0.8893401015228427, 'support': 2013.0}	{'precision': 0.8648626213995185, 'recall': 0.919107268877911, 'f1-score': 0.8911602446221614, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9996463932107497, 'f1-score': 0.9998231653404068, 'support': 11312.0}	0.8931	{'precision': 0.831140869736038, 'recall': 0.8140423219310874, 'f1-score': 0.82129465173088, 'support': 29705.0}	{'precision': 0.8885697203474908, 'recall': 0.8931156370981316, 'f1-score': 0.8898866167708168, 'support': 29705.0}
No log	11.0	451	0.3918	{'precision': 0.6324503311258278, 'recall': 0.7047970479704797, 'f1-score': 0.6666666666666666, 'support': 271.0}	{'precision': 0.8590604026845637, 'recall': 0.920863309352518, 'f1-score': 0.888888888888889, 'support': 139.0}	{'precision': 0.8969594594594594, 'recall': 0.8388625592417062, 'f1-score': 0.8669387755102039, 'support': 633.0}	{'precision': 0.6174621653084983, 'recall': 0.6628342914271432, 'f1-score': 0.6393442622950819, 'support': 4001.0}	{'precision': 0.8781190019193857, 'recall': 0.9090909090909091, 'f1-score': 0.893336587747132, 'support': 2013.0}	{'precision': 0.8994622185762464, 'recall': 0.870501058574453, 'f1-score': 0.8847446989734163, 'support': 11336.0}	{'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}	0.8925	{'precision': 0.8262162255819973, 'recall': 0.8438498822367441, 'f1-score': 0.8342742685830558, 'support': 29705.0}	{'precision': 0.8956405372021359, 'recall': 0.8925096785053022, 'f1-score': 0.8938146210826604, 'support': 29705.0}
No log	12.0	492	0.4418	{'precision': 0.6735537190082644, 'recall': 0.6014760147601476, 'f1-score': 0.6354775828460039, 'support': 271.0}	{'precision': 0.8680555555555556, 'recall': 0.8992805755395683, 'f1-score': 0.8833922261484098, 'support': 139.0}	{'precision': 0.863013698630137, 'recall': 0.8957345971563981, 'f1-score': 0.8790697674418606, 'support': 633.0}	{'precision': 0.6681084198385236, 'recall': 0.5791052236940765, 'f1-score': 0.6204311152764761, 'support': 4001.0}	{'precision': 0.9092783505154639, 'recall': 0.8763040238450075, 'f1-score': 0.8924867189476349, 'support': 2013.0}	{'precision': 0.8735661056685925, 'recall': 0.9203422724064926, 'f1-score': 0.8963443446883457, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0}	0.8982	{'precision': 0.8365108356023624, 'recall': 0.8245934722434826, 'f1-score': 0.829593936078039, 'support': 29705.0}	{'precision': 0.8943849497568445, 'recall': 0.8981652920383774, 'f1-score': 0.8955676564189546, 'support': 29705.0}
0.2042	13.0	533	0.4919	{'precision': 0.6954545454545454, 'recall': 0.5645756457564576, 'f1-score': 0.6232179226069247, 'support': 271.0}	{'precision': 0.863013698630137, 'recall': 0.9064748201438849, 'f1-score': 0.8842105263157894, 'support': 139.0}	{'precision': 0.8522895125553914, 'recall': 0.9115323854660348, 'f1-score': 0.8809160305343511, 'support': 633.0}	{'precision': 0.6722793888369193, 'recall': 0.5388652836790803, 'f1-score': 0.5982241953385128, 'support': 4001.0}	{'precision': 0.8951612903225806, 'recall': 0.8822652757078987, 'f1-score': 0.8886664998749062, 'support': 2013.0}	{'precision': 0.8646381578947369, 'recall': 0.9274876499647142, 'f1-score': 0.8949608443990467, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0}	0.8959	{'precision': 0.8346909419563301, 'recall': 0.8187303798601082, 'f1-score': 0.8243074023238676, 'support': 29705.0}	{'precision': 0.8905306856564867, 'recall': 0.8959097794984009, 'f1-score': 0.8917210552875402, 'support': 29705.0}
0.2042	14.0	574	0.4502	{'precision': 0.6653992395437263, 'recall': 0.6457564575645757, 'f1-score': 0.6554307116104869, 'support': 271.0}	{'precision': 0.8551724137931035, 'recall': 0.8920863309352518, 'f1-score': 0.8732394366197184, 'support': 139.0}	{'precision': 0.878740157480315, 'recall': 0.8815165876777251, 'f1-score': 0.8801261829652995, 'support': 633.0}	{'precision': 0.659796573875803, 'recall': 0.6160959760059985, 'f1-score': 0.6371978803153676, 'support': 4001.0}	{'precision': 0.9004570848146267, 'recall': 0.8807749627421758, 'f1-score': 0.890507282772476, 'support': 2013.0}	{'precision': 0.8842421640188922, 'recall': 0.9083450952717008, 'f1-score': 0.8961315869631434, 'support': 11336.0}	{'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}	0.8990	{'precision': 0.8348296619323524, 'recall': 0.8320822014567755, 'f1-score': 0.8332332973209274, 'support': 29705.0}	{'precision': 0.8969422674379979, 'recall': 0.8989732368288167, 'f1-score': 0.8977845287840629, 'support': 29705.0}
0.2042	15.0	615	0.4731	{'precision': 0.678714859437751, 'recall': 0.6236162361623616, 'f1-score': 0.65, 'support': 271.0}	{'precision': 0.8732394366197183, 'recall': 0.8920863309352518, 'f1-score': 0.8825622775800712, 'support': 139.0}	{'precision': 0.8680981595092024, 'recall': 0.8941548183254344, 'f1-score': 0.8809338521400777, 'support': 633.0}	{'precision': 0.6616605270614905, 'recall': 0.5836040989752562, 'f1-score': 0.6201859229747676, 'support': 4001.0}	{'precision': 0.9088082901554404, 'recall': 0.8713363139592648, 'f1-score': 0.8896779102206441, 'support': 2013.0}	{'precision': 0.8740223698595576, 'recall': 0.9168136908962597, 'f1-score': 0.8949067895122056, 'support': 11336.0}	{'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}	0.8973	{'precision': 0.8377919489490228, 'recall': 0.8259444984648326, 'f1-score': 0.831180964632538, 'support': 29705.0}	{'precision': 0.8938384307406486, 'recall': 0.8972563541491332, 'f1-score': 0.8949808504290477, 'support': 29705.0}

Framework versions

• Transformers 4.37.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.0
• Tokenizers 0.15.2