meta_data/README_s42_e16.md

metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-sep_tok_full_labels
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: sep_tok_full_labels
          split: train[80%:100%]
          args: sep_tok_full_labels
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8987712506312069

longformer-sep_tok_full_labels

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

• Loss: 0.4995
• B-claim: {'precision': 0.6529850746268657, 'recall': 0.6457564575645757, 'f1-score': 0.6493506493506495, 'support': 271.0}
• B-majorclaim: {'precision': 0.8591549295774648, 'recall': 0.8776978417266187, 'f1-score': 0.8683274021352312, 'support': 139.0}
• B-premise: {'precision': 0.8767772511848341, 'recall': 0.8767772511848341, 'f1-score': 0.876777251184834, 'support': 633.0}
• I-claim: {'precision': 0.6591271953166578, 'recall': 0.6190952261934516, 'f1-score': 0.6384843407655625, 'support': 4001.0}
• I-majorclaim: {'precision': 0.8964467005076142, 'recall': 0.877297565822156, 'f1-score': 0.8867687672608586, 'support': 2013.0}
• I-premise: {'precision': 0.885505376344086, 'recall': 0.9080804516584333, 'f1-score': 0.8966508427333304, 'support': 11336.0}
• O: {'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0}
• Accuracy: 0.8988
• Macro avg: {'precision': 0.8328566467939318, 'recall': 0.829205655579733, 'f1-score': 0.8308895190426757, 'support': 29705.0}
• Weighted avg: {'precision': 0.896925957019205, 'recall': 0.8987712506312069, 'f1-score': 0.8977022947337076, 'support': 29705.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 16

Training results

Training Loss	Epoch	Step	Validation Loss	B-claim	B-majorclaim	B-premise	I-claim	I-majorclaim	I-premise	O	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.4083	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 271.0}	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 139.0}	{'precision': 0.643595041322314, 'recall': 0.9842022116903634, 'f1-score': 0.778263585259213, 'support': 633.0}	{'precision': 0.4563708012760368, 'recall': 0.6078480379905024, 'f1-score': 0.5213290460878884, 'support': 4001.0}	{'precision': 0.7011784511784511, 'recall': 0.4138102334823646, 'f1-score': 0.520462355513902, 'support': 2013.0}	{'precision': 0.8778050331607159, 'recall': 0.8523288637967537, 'f1-score': 0.8648793805666204, 'support': 11336.0}	{'precision': 0.9991081780076697, 'recall': 0.9903642149929278, 'f1-score': 0.9947169811320755, 'support': 11312.0}	0.8333	{'precision': 0.5254367864207411, 'recall': 0.5497933659932731, 'f1-score': 0.5256644783656713, 'support': 29705.0}	{'precision': 0.8381591322948091, 'recall': 0.8332940582393537, 'f1-score': 0.830926787853407, 'support': 29705.0}
No log	2.0	82	0.2959	{'precision': 0.38823529411764707, 'recall': 0.24354243542435425, 'f1-score': 0.29931972789115646, 'support': 271.0}	{'precision': 0.8620689655172413, 'recall': 0.17985611510791366, 'f1-score': 0.2976190476190476, 'support': 139.0}	{'precision': 0.7422802850356295, 'recall': 0.9873617693522907, 'f1-score': 0.8474576271186441, 'support': 633.0}	{'precision': 0.5749261291684254, 'recall': 0.340414896275931, 'f1-score': 0.42762951334379906, 'support': 4001.0}	{'precision': 0.7699071812408402, 'recall': 0.7829110779930452, 'f1-score': 0.7763546798029557, 'support': 2013.0}	{'precision': 0.8290439755777108, 'recall': 0.9462773465067043, 'f1-score': 0.8837899073120494, 'support': 11336.0}	{'precision': 0.9999115748518879, 'recall': 0.9996463932107497, 'f1-score': 0.9997789664471067, 'support': 11312.0}	0.8648	{'precision': 0.7380533436441974, 'recall': 0.6400014334101413, 'f1-score': 0.6474213527906798, 'support': 29705.0}	{'precision': 0.850161508562683, 'recall': 0.8648039050664871, 'f1-score': 0.8503893311871605, 'support': 29705.0}
No log	3.0	123	0.2633	{'precision': 0.61328125, 'recall': 0.5793357933579336, 'f1-score': 0.5958254269449716, 'support': 271.0}	{'precision': 0.7655172413793103, 'recall': 0.7985611510791367, 'f1-score': 0.7816901408450705, 'support': 139.0}	{'precision': 0.8738317757009346, 'recall': 0.8862559241706162, 'f1-score': 0.88, 'support': 633.0}	{'precision': 0.6150895140664961, 'recall': 0.6010997250687328, 'f1-score': 0.6080141575022121, 'support': 4001.0}	{'precision': 0.7346859149434257, 'recall': 0.9354197714853453, 'f1-score': 0.8229895104895104, 'support': 2013.0}	{'precision': 0.9111703104905383, 'recall': 0.875, 'f1-score': 0.8927189271892718, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9992043847241867, 'f1-score': 0.9996020340481981, 'support': 11312.0}	0.8867	{'precision': 0.7876537152258151, 'recall': 0.8106966785551358, 'f1-score': 0.7972628852884621, 'support': 29705.0}	{'precision': 0.8889636142603039, 'recall': 0.8866857431408853, 'f1-score': 0.8868495578802252, 'support': 29705.0}
No log	4.0	164	0.2767	{'precision': 0.6180555555555556, 'recall': 0.6568265682656826, 'f1-score': 0.6368515205724509, 'support': 271.0}	{'precision': 0.7368421052631579, 'recall': 0.9064748201438849, 'f1-score': 0.8129032258064516, 'support': 139.0}	{'precision': 0.9126712328767124, 'recall': 0.8420221169036335, 'f1-score': 0.8759244042728019, 'support': 633.0}	{'precision': 0.6039107545750815, 'recall': 0.6020994751312172, 'f1-score': 0.6030037546933668, 'support': 4001.0}	{'precision': 0.7743306417339566, 'recall': 0.905116741182315, 'f1-score': 0.8346312414109024, 'support': 2013.0}	{'precision': 0.9007175946952494, 'recall': 0.8747353563867325, 'f1-score': 0.8875363616021481, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0}	0.8852	{'precision': 0.7923611263856734, 'recall': 0.8267409537594504, 'f1-score': 0.8072580436508092, 'support': 29705.0}	{'precision': 0.8868925824921324, 'recall': 0.8852045110250799, 'f1-score': 0.8855540596827394, 'support': 29705.0}
No log	5.0	205	0.2875	{'precision': 0.6273885350318471, 'recall': 0.7269372693726938, 'f1-score': 0.6735042735042734, 'support': 271.0}	{'precision': 0.7865853658536586, 'recall': 0.9280575539568345, 'f1-score': 0.8514851485148515, 'support': 139.0}	{'precision': 0.9257950530035336, 'recall': 0.8278041074249605, 'f1-score': 0.8740617180984154, 'support': 633.0}	{'precision': 0.5824691841126981, 'recall': 0.744063984003999, 'f1-score': 0.6534240561896401, 'support': 4001.0}	{'precision': 0.8339432753888381, 'recall': 0.9056135121708893, 'f1-score': 0.8683019766611098, 'support': 2013.0}	{'precision': 0.9326903957049115, 'recall': 0.8275405786873676, 'f1-score': 0.8769748527624568, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9994695898161244, 'f1-score': 0.9997347245556637, 'support': 11312.0}	0.8866	{'precision': 0.8126959727279266, 'recall': 0.8513552279189812, 'f1-score': 0.8282123928980587, 'support': 29705.0}	{'precision': 0.90084333519953, 'recall': 0.8866184144083488, 'f1-score': 0.8909875382676978, 'support': 29705.0}
No log	6.0	246	0.2945	{'precision': 0.6326530612244898, 'recall': 0.6863468634686347, 'f1-score': 0.6584070796460177, 'support': 271.0}	{'precision': 0.8818897637795275, 'recall': 0.8057553956834532, 'f1-score': 0.8421052631578947, 'support': 139.0}	{'precision': 0.8858520900321544, 'recall': 0.8704581358609794, 'f1-score': 0.8780876494023905, 'support': 633.0}	{'precision': 0.6304404482668752, 'recall': 0.6045988502874281, 'f1-score': 0.6172492982903802, 'support': 4001.0}	{'precision': 0.8976501305483029, 'recall': 0.8539493293591655, 'f1-score': 0.8752545824847251, 'support': 2013.0}	{'precision': 0.8802859357505813, 'recall': 0.9016407904022583, 'f1-score': 0.8908354033206956, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9988507779349364, 'f1-score': 0.9994250585997965, 'support': 11312.0}	0.8923	{'precision': 0.829824489943133, 'recall': 0.8173714489995507, 'f1-score': 0.8230520478431285, 'support': 29705.0}	{'precision': 0.8912660947222903, 'recall': 0.8923413566739606, 'f1-score': 0.8916619674856285, 'support': 29705.0}
No log	7.0	287	0.3037	{'precision': 0.6442953020134228, 'recall': 0.7084870848708487, 'f1-score': 0.6748681898066783, 'support': 271.0}	{'precision': 0.8705035971223022, 'recall': 0.8705035971223022, 'f1-score': 0.8705035971223022, 'support': 139.0}	{'precision': 0.8991735537190083, 'recall': 0.8593996840442338, 'f1-score': 0.8788368336025849, 'support': 633.0}	{'precision': 0.6310387984981226, 'recall': 0.6300924768807799, 'f1-score': 0.6305652826413206, 'support': 4001.0}	{'precision': 0.891640866873065, 'recall': 0.8584202682563339, 'f1-score': 0.8747152619589977, 'support': 2013.0}	{'precision': 0.8886845331932037, 'recall': 0.8951129146083274, 'f1-score': 0.8918871407225103, 'support': 11336.0}	{'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0}	0.8943	{'precision': 0.832190950202732, 'recall': 0.8317165751118323, 'f1-score': 0.8316251865506278, 'support': 29705.0}	{'precision': 0.8944813348740749, 'recall': 0.8942938899175223, 'f1-score': 0.894338297946804, 'support': 29705.0}
No log	8.0	328	0.3549	{'precision': 0.6498054474708171, 'recall': 0.6162361623616236, 'f1-score': 0.6325757575757576, 'support': 271.0}	{'precision': 0.8796992481203008, 'recall': 0.841726618705036, 'f1-score': 0.8602941176470588, 'support': 139.0}	{'precision': 0.863914373088685, 'recall': 0.8925750394944708, 'f1-score': 0.8780108780108781, 'support': 633.0}	{'precision': 0.644374282433984, 'recall': 0.5611097225693576, 'f1-score': 0.5998663994655979, 'support': 4001.0}	{'precision': 0.9037546271813856, 'recall': 0.8489816194734228, 'f1-score': 0.8755122950819673, 'support': 2013.0}	{'precision': 0.8685594989561587, 'recall': 0.9175194071983063, 'f1-score': 0.8923684097636309, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0}	0.8926	{'precision': 0.8300153538930474, 'recall': 0.811151452586415, 'f1-score': 0.8197976649632047, 'support': 29705.0}	{'precision': 0.8887602531095763, 'recall': 0.892610671604107, 'f1-score': 0.8899730612246367, 'support': 29705.0}
No log	9.0	369	0.3716	{'precision': 0.6468531468531469, 'recall': 0.6826568265682657, 'f1-score': 0.6642728904847396, 'support': 271.0}	{'precision': 0.8333333333333334, 'recall': 0.8633093525179856, 'f1-score': 0.8480565371024734, 'support': 139.0}	{'precision': 0.8941368078175895, 'recall': 0.8672985781990521, 'f1-score': 0.8805132317562149, 'support': 633.0}	{'precision': 0.6205917159763313, 'recall': 0.6553361659585104, 'f1-score': 0.637490882567469, 'support': 4001.0}	{'precision': 0.8750640040962622, 'recall': 0.8489816194734228, 'f1-score': 0.8618255168935955, 'support': 2013.0}	{'precision': 0.8960551033187226, 'recall': 0.8836450247000706, 'f1-score': 0.8898067954696869, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9992927864214993, 'f1-score': 0.9996462681287585, 'support': 11312.0}	0.8923	{'precision': 0.8237191587707694, 'recall': 0.8286457648341152, 'f1-score': 0.8259445889147053, 'support': 29705.0}	{'precision': 0.8945056752252882, 'recall': 0.8923076923076924, 'f1-score': 0.8933030430182268, 'support': 29705.0}
No log	10.0	410	0.4143	{'precision': 0.6746987951807228, 'recall': 0.6199261992619927, 'f1-score': 0.6461538461538461, 'support': 271.0}	{'precision': 0.8157894736842105, 'recall': 0.8920863309352518, 'f1-score': 0.852233676975945, 'support': 139.0}	{'precision': 0.8785046728971962, 'recall': 0.8909952606635071, 'f1-score': 0.8847058823529412, 'support': 633.0}	{'precision': 0.6611758023288838, 'recall': 0.5818545363659086, 'f1-score': 0.6189843126827972, 'support': 4001.0}	{'precision': 0.8256503879507074, 'recall': 0.8986587183308494, 'f1-score': 0.8606089438629877, 'support': 2013.0}	{'precision': 0.8856627437505369, 'recall': 0.9094918842625265, 'f1-score': 0.8974191582887234, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0}	0.8959	{'precision': 0.8202116965417511, 'recall': 0.8275353892468712, 'f1-score': 0.822853314312215, 'support': 29705.0}	{'precision': 0.8924963153509083, 'recall': 0.8958761151321326, 'f1-score': 0.8936607445011815, 'support': 29705.0}
No log	11.0	451	0.4242	{'precision': 0.6442953020134228, 'recall': 0.7084870848708487, 'f1-score': 0.6748681898066783, 'support': 271.0}	{'precision': 0.8661971830985915, 'recall': 0.8848920863309353, 'f1-score': 0.8754448398576513, 'support': 139.0}	{'precision': 0.8988391376451078, 'recall': 0.8562401263823065, 'f1-score': 0.8770226537216829, 'support': 633.0}	{'precision': 0.6332931242460796, 'recall': 0.6560859785053736, 'f1-score': 0.6444880923152466, 'support': 4001.0}	{'precision': 0.8797595190380761, 'recall': 0.8723298559364133, 'f1-score': 0.8760289348964829, 'support': 2013.0}	{'precision': 0.8979136947218259, 'recall': 0.8884086097388849, 'f1-score': 0.8931358637814828, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9993811881188119, 'f1-score': 0.9996904982977407, 'support': 11312.0}	0.8959	{'precision': 0.8314711372518719, 'recall': 0.8379749899833678, 'f1-score': 0.8343827246681379, 'support': 29705.0}	{'precision': 0.8974745650471141, 'recall': 0.8959434438646693, 'f1-score': 0.8966457372110589, 'support': 29705.0}
No log	12.0	492	0.4911	{'precision': 0.6926605504587156, 'recall': 0.5571955719557196, 'f1-score': 0.6175869120654396, 'support': 271.0}	{'precision': 0.8759124087591241, 'recall': 0.8633093525179856, 'f1-score': 0.8695652173913043, 'support': 139.0}	{'precision': 0.8461538461538461, 'recall': 0.9210110584518167, 'f1-score': 0.8819969742813918, 'support': 633.0}	{'precision': 0.6817567567567567, 'recall': 0.5043739065233691, 'f1-score': 0.5798017526217497, 'support': 4001.0}	{'precision': 0.9035639412997903, 'recall': 0.8564331843020367, 'f1-score': 0.8793675082887019, 'support': 2013.0}	{'precision': 0.8544101168560909, 'recall': 0.9416901905434015, 'f1-score': 0.8959295006294586, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9988507779349364, 'f1-score': 0.9994250585997965, 'support': 11312.0}	0.8945	{'precision': 0.8363510886120462, 'recall': 0.8061234346041807, 'f1-score': 0.8176675605539775, 'support': 29705.0}	{'precision': 0.888377522333214, 'recall': 0.8944622117488639, 'f1-score': 0.8886802353660483, 'support': 29705.0}
0.1948	13.0	533	0.4856	{'precision': 0.6967213114754098, 'recall': 0.6273062730627307, 'f1-score': 0.6601941747572815, 'support': 271.0}	{'precision': 0.8689655172413793, 'recall': 0.9064748201438849, 'f1-score': 0.8873239436619718, 'support': 139.0}	{'precision': 0.8715596330275229, 'recall': 0.9004739336492891, 'f1-score': 0.8857808857808857, 'support': 633.0}	{'precision': 0.6689675007190107, 'recall': 0.5813546613346663, 'f1-score': 0.622091468307034, 'support': 4001.0}	{'precision': 0.8964467005076142, 'recall': 0.877297565822156, 'f1-score': 0.8867687672608586, 'support': 2013.0}	{'precision': 0.8748636172891313, 'recall': 0.9195483415666902, 'f1-score': 0.8966496064685389, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.998939179632249, 'f1-score': 0.9994693083318592, 'support': 11312.0}	0.8982	{'precision': 0.8396463257514384, 'recall': 0.8301992536016665, 'f1-score': 0.8340397363669184, 'support': 29705.0}	{'precision': 0.8945239883555292, 'recall': 0.898232620770914, 'f1-score': 0.8957219390068148, 'support': 29705.0}
0.1948	14.0	574	0.4750	{'precision': 0.6655052264808362, 'recall': 0.7047970479704797, 'f1-score': 0.6845878136200716, 'support': 271.0}	{'precision': 0.8661971830985915, 'recall': 0.8848920863309353, 'f1-score': 0.8754448398576513, 'support': 139.0}	{'precision': 0.8973941368078175, 'recall': 0.8704581358609794, 'f1-score': 0.8837209302325582, 'support': 633.0}	{'precision': 0.6448780487804878, 'recall': 0.6608347913021745, 'f1-score': 0.6527589186520183, 'support': 4001.0}	{'precision': 0.8922610015174507, 'recall': 0.8763040238450075, 'f1-score': 0.8842105263157896, 'support': 2013.0}	{'precision': 0.8977292886287032, 'recall': 0.8928193366266761, 'f1-score': 0.8952675807164971, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0}	0.8990	{'precision': 0.8377092693305553, 'recall': 0.8414310028912771, 'f1-score': 0.839420915227447, 'support': 29705.0}	{'precision': 0.899974465529261, 'recall': 0.899006901195085, 'f1-score': 0.8994601237155254, 'support': 29705.0}
0.1948	15.0	615	0.5226	{'precision': 0.6761133603238867, 'recall': 0.6162361623616236, 'f1-score': 0.6447876447876448, 'support': 271.0}	{'precision': 0.8695652173913043, 'recall': 0.8633093525179856, 'f1-score': 0.8664259927797834, 'support': 139.0}	{'precision': 0.8677811550151976, 'recall': 0.9020537124802528, 'f1-score': 0.8845855925639039, 'support': 633.0}	{'precision': 0.6685476685476686, 'recall': 0.5626093476630842, 'f1-score': 0.6110206297502714, 'support': 4001.0}	{'precision': 0.8988186954288649, 'recall': 0.8693492300049677, 'f1-score': 0.8838383838383838, 'support': 2013.0}	{'precision': 0.8697276652274992, 'recall': 0.9240472829922372, 'f1-score': 0.8960650128314799, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9992927864214993, 'f1-score': 0.9996462681287585, 'support': 11312.0}	0.8968	{'precision': 0.8357933945620601, 'recall': 0.8195568392059501, 'f1-score': 0.8266242178114608, 'support': 29705.0}	{'precision': 0.8924024852976349, 'recall': 0.8967513886551086, 'f1-score': 0.8936126955615038, 'support': 29705.0}
0.1948	16.0	656	0.4995	{'precision': 0.6529850746268657, 'recall': 0.6457564575645757, 'f1-score': 0.6493506493506495, 'support': 271.0}	{'precision': 0.8591549295774648, 'recall': 0.8776978417266187, 'f1-score': 0.8683274021352312, 'support': 139.0}	{'precision': 0.8767772511848341, 'recall': 0.8767772511848341, 'f1-score': 0.876777251184834, 'support': 633.0}	{'precision': 0.6591271953166578, 'recall': 0.6190952261934516, 'f1-score': 0.6384843407655625, 'support': 4001.0}	{'precision': 0.8964467005076142, 'recall': 0.877297565822156, 'f1-score': 0.8867687672608586, 'support': 2013.0}	{'precision': 0.885505376344086, 'recall': 0.9080804516584333, 'f1-score': 0.8966508427333304, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0}	0.8988	{'precision': 0.8328566467939318, 'recall': 0.829205655579733, 'f1-score': 0.8308895190426757, 'support': 29705.0}	{'precision': 0.896925957019205, 'recall': 0.8987712506312069, 'f1-score': 0.8977022947337076, 'support': 29705.0}

Framework versions

• Transformers 4.37.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.0
• Tokenizers 0.15.2