meta_data/README_s42_e13.md

metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-sep_tok_full_labels
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: sep_tok_full_labels
          split: train[80%:100%]
          args: sep_tok_full_labels
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8991078942938899

longformer-sep_tok_full_labels

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

• Loss: 0.3891
• B-claim: {'precision': 0.6781609195402298, 'recall': 0.6531365313653137, 'f1-score': 0.6654135338345865, 'support': 271.0}
• B-majorclaim: {'precision': 0.8840579710144928, 'recall': 0.8776978417266187, 'f1-score': 0.8808664259927799, 'support': 139.0}
• B-premise: {'precision': 0.8773291925465838, 'recall': 0.8925750394944708, 'f1-score': 0.8848864526233359, 'support': 633.0}
• I-claim: {'precision': 0.6596830513027129, 'recall': 0.6138465383654087, 'f1-score': 0.6359399274987053, 'support': 4001.0}
• I-majorclaim: {'precision': 0.9052631578947369, 'recall': 0.8544461003477397, 'f1-score': 0.8791208791208791, 'support': 2013.0}
• I-premise: {'precision': 0.8835423598669055, 'recall': 0.9135497529992943, 'f1-score': 0.898295528472915, 'support': 11336.0}
• O: {'precision': 0.9994698710019438, 'recall': 1.0, 'f1-score': 0.9997348652231551, 'support': 11312.0}
• Accuracy: 0.8991
• Macro avg: {'precision': 0.8410723604525151, 'recall': 0.8293216863284066, 'f1-score': 0.8348939446809082, 'support': 29705.0}
• Weighted avg: {'precision': 0.8970052531014272, 'recall': 0.8991078942938899, 'f1-score': 0.8977965161137413, 'support': 29705.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 13

Training results

Training Loss	Epoch	Step	Validation Loss	B-claim	B-majorclaim	B-premise	I-claim	I-majorclaim	I-premise	O	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.3929	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 271.0}	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 139.0}	{'precision': 0.639412997903564, 'recall': 0.9636650868878357, 'f1-score': 0.7687460617517329, 'support': 633.0}	{'precision': 0.46799191374663074, 'recall': 0.34716320919770055, 'f1-score': 0.39862247094274644, 'support': 4001.0}	{'precision': 0.7141909814323607, 'recall': 0.5350223546944859, 'f1-score': 0.6117580232888384, 'support': 2013.0}	{'precision': 0.8186293362049073, 'recall': 0.9388673253352152, 'f1-score': 0.8746353289230391, 'support': 11336.0}	{'precision': 0.99654071314529, 'recall': 0.9931930693069307, 'f1-score': 0.9948640750907642, 'support': 11312.0}	0.8401	{'precision': 0.5195379917761075, 'recall': 0.5397015779174525, 'f1-score': 0.5212322799995887, 'support': 29705.0}	{'precision': 0.8169567824445351, 'recall': 0.8400605958592829, 'f1-score': 0.8241623353075073, 'support': 29705.0}
No log	2.0	82	0.3184	{'precision': 0.39226519337016574, 'recall': 0.26199261992619927, 'f1-score': 0.3141592920353982, 'support': 271.0}	{'precision': 0.6842105263157895, 'recall': 0.09352517985611511, 'f1-score': 0.16455696202531647, 'support': 139.0}	{'precision': 0.75453446191052, 'recall': 0.985781990521327, 'f1-score': 0.8547945205479451, 'support': 633.0}	{'precision': 0.5480972515856237, 'recall': 0.2591852036990752, 'f1-score': 0.3519429832004073, 'support': 4001.0}	{'precision': 0.7032640949554896, 'recall': 0.8241430700447094, 'f1-score': 0.7589204025617566, 'support': 2013.0}	{'precision': 0.8212545676004872, 'recall': 0.9516584333098095, 'f1-score': 0.8816606734226872, 'support': 11336.0}	{'precision': 0.9998228677707909, 'recall': 0.9979667609618105, 'f1-score': 0.9988939521302481, 'support': 11312.0}	0.8578	{'precision': 0.7004927090726952, 'recall': 0.6248933226170067, 'f1-score': 0.6178469694176798, 'support': 29705.0}	{'precision': 0.8384910025326497, 'recall': 0.8578017168826797, 'f1-score': 0.837533073352249, 'support': 29705.0}
No log	3.0	123	0.2601	{'precision': 0.55, 'recall': 0.44649446494464945, 'f1-score': 0.4928716904276986, 'support': 271.0}	{'precision': 0.8260869565217391, 'recall': 0.5467625899280576, 'f1-score': 0.658008658008658, 'support': 139.0}	{'precision': 0.8153214774281806, 'recall': 0.9415481832543444, 'f1-score': 0.873900293255132, 'support': 633.0}	{'precision': 0.6178135239797439, 'recall': 0.5183704073981504, 'f1-score': 0.5637401467790161, 'support': 4001.0}	{'precision': 0.8206656731246895, 'recall': 0.8206656731246895, 'f1-score': 0.8206656731246895, 'support': 2013.0}	{'precision': 0.8689263650025046, 'recall': 0.918136908962597, 'f1-score': 0.8928540790941066, 'support': 11336.0}	{'precision': 0.9998232278592893, 'recall': 1.0, 'f1-score': 0.9999116061168567, 'support': 11312.0}	0.8833	{'precision': 0.7855196034165923, 'recall': 0.7417111753732126, 'f1-score': 0.7574217352580226, 'support': 29705.0}	{'precision': 0.8774279117597774, 'recall': 0.8833193065140549, 'f1-score': 0.8792502465398797, 'support': 29705.0}
No log	4.0	164	0.2654	{'precision': 0.6285714285714286, 'recall': 0.6494464944649446, 'f1-score': 0.6388384754990926, 'support': 271.0}	{'precision': 0.735632183908046, 'recall': 0.920863309352518, 'f1-score': 0.8178913738019169, 'support': 139.0}	{'precision': 0.9134125636672326, 'recall': 0.8499210110584519, 'f1-score': 0.8805237315875614, 'support': 633.0}	{'precision': 0.6259168704156479, 'recall': 0.5758560359910022, 'f1-score': 0.5998437906795105, 'support': 4001.0}	{'precision': 0.7702251876563804, 'recall': 0.9175360158966717, 'f1-score': 0.8374518249829971, 'support': 2013.0}	{'precision': 0.8953766971337297, 'recall': 0.8900846859562456, 'f1-score': 0.8927228489272284, 'support': 11336.0}	{'precision': 0.9998232278592893, 'recall': 1.0, 'f1-score': 0.9999116061168567, 'support': 11312.0}	0.8886	{'precision': 0.795565451315965, 'recall': 0.8291010789599762, 'f1-score': 0.8095976645135947, 'support': 29705.0}	{'precision': 0.8875789657311893, 'recall': 0.8885709476519105, 'f1-score': 0.8874214941574174, 'support': 29705.0}
No log	5.0	205	0.2796	{'precision': 0.625, 'recall': 0.6642066420664207, 'f1-score': 0.6440071556350626, 'support': 271.0}	{'precision': 0.7398843930635838, 'recall': 0.920863309352518, 'f1-score': 0.8205128205128205, 'support': 139.0}	{'precision': 0.9123711340206185, 'recall': 0.8388625592417062, 'f1-score': 0.8740740740740741, 'support': 633.0}	{'precision': 0.6159355416293644, 'recall': 0.6878280429892527, 'f1-score': 0.6498996339591451, 'support': 4001.0}	{'precision': 0.8207674943566592, 'recall': 0.9031296572280179, 'f1-score': 0.8599810785241249, 'support': 2013.0}	{'precision': 0.9187289088863893, 'recall': 0.864590684544813, 'f1-score': 0.8908380294491911, 'support': 11336.0}	{'precision': 0.9997347714614092, 'recall': 0.9996463932107497, 'f1-score': 0.9996905803827962, 'support': 11312.0}	0.8927	{'precision': 0.8046317490597178, 'recall': 0.8398753269476398, 'f1-score': 0.8198576246481736, 'support': 29705.0}	{'precision': 0.8985027292569432, 'recall': 0.892711664702912, 'f1-score': 0.8948088269241911, 'support': 29705.0}
No log	6.0	246	0.2896	{'precision': 0.6379928315412187, 'recall': 0.6568265682656826, 'f1-score': 0.6472727272727272, 'support': 271.0}	{'precision': 0.8656716417910447, 'recall': 0.8345323741007195, 'f1-score': 0.8498168498168498, 'support': 139.0}	{'precision': 0.8807631160572337, 'recall': 0.8751974723538705, 'f1-score': 0.87797147385103, 'support': 633.0}	{'precision': 0.6119942196531792, 'recall': 0.6350912271932017, 'f1-score': 0.6233288360112841, 'support': 4001.0}	{'precision': 0.9029810298102982, 'recall': 0.8276204669647292, 'f1-score': 0.8636599274235356, 'support': 2013.0}	{'precision': 0.8871935085553008, 'recall': 0.8873500352858151, 'f1-score': 0.8872717650172003, 'support': 11336.0}	{'precision': 0.998587570621469, 'recall': 1.0, 'f1-score': 0.9992932862190813, 'support': 11312.0}	0.8896	{'precision': 0.8264548454328206, 'recall': 0.816659734880574, 'f1-score': 0.8212306950873868, 'support': 29705.0}	{'precision': 0.89110538177796, 'recall': 0.8896145430062279, 'f1-score': 0.8902169965626361, 'support': 29705.0}
No log	7.0	287	0.3125	{'precision': 0.631578947368421, 'recall': 0.6642066420664207, 'f1-score': 0.6474820143884892, 'support': 271.0}	{'precision': 0.8974358974358975, 'recall': 0.7553956834532374, 'f1-score': 0.8203124999999999, 'support': 139.0}	{'precision': 0.8753894080996885, 'recall': 0.8878357030015798, 'f1-score': 0.8815686274509804, 'support': 633.0}	{'precision': 0.6153071500503524, 'recall': 0.6108472881779555, 'f1-score': 0.6130691082403109, 'support': 4001.0}	{'precision': 0.9138906348208674, 'recall': 0.7223050173869846, 'f1-score': 0.806881243063263, 'support': 2013.0}	{'precision': 0.8795109110979027, 'recall': 0.9137261820748059, 'f1-score': 0.8962921299701466, 'support': 11336.0}	{'precision': 0.998940022966169, 'recall': 0.9997347949080623, 'f1-score': 0.9993372509168028, 'support': 11312.0}	0.8891	{'precision': 0.8302932816913283, 'recall': 0.7934359015812923, 'f1-score': 0.809277553432856, 'support': 29705.0}	{'precision': 0.8894689097139381, 'recall': 0.8891432418784716, 'f1-score': 0.8883870061149315, 'support': 29705.0}
No log	8.0	328	0.3276	{'precision': 0.6532846715328468, 'recall': 0.6605166051660517, 'f1-score': 0.6568807339449542, 'support': 271.0}	{'precision': 0.8633093525179856, 'recall': 0.8633093525179856, 'f1-score': 0.8633093525179856, 'support': 139.0}	{'precision': 0.8827258320126783, 'recall': 0.8799368088467614, 'f1-score': 0.8813291139240506, 'support': 633.0}	{'precision': 0.6458757637474541, 'recall': 0.6340914771307173, 'f1-score': 0.6399293731870349, 'support': 4001.0}	{'precision': 0.8905033731188375, 'recall': 0.8524590163934426, 'f1-score': 0.8710659898477157, 'support': 2013.0}	{'precision': 0.8909391591957525, 'recall': 0.9029640084685956, 'f1-score': 0.8969112814895948, 'support': 11336.0}	{'precision': 0.9993814615180702, 'recall': 0.9998231966053748, 'f1-score': 0.9996022802598435, 'support': 11312.0}	0.8973	{'precision': 0.8322885162348036, 'recall': 0.8275857807327042, 'f1-score': 0.8298611607387398, 'support': 29705.0}	{'precision': 0.8967253735003088, 'recall': 0.8973236828816697, 'f1-score': 0.8969734572955119, 'support': 29705.0}
No log	9.0	369	0.3595	{'precision': 0.6528301886792452, 'recall': 0.6383763837638377, 'f1-score': 0.6455223880597014, 'support': 271.0}	{'precision': 0.855072463768116, 'recall': 0.8489208633093526, 'f1-score': 0.851985559566787, 'support': 139.0}	{'precision': 0.8765625, 'recall': 0.8862559241706162, 'f1-score': 0.8813825608798115, 'support': 633.0}	{'precision': 0.6564171122994652, 'recall': 0.6135966008497875, 'f1-score': 0.6342849761012789, 'support': 4001.0}	{'precision': 0.8883170355120947, 'recall': 0.8574267262791853, 'f1-score': 0.8725985844287159, 'support': 2013.0}	{'precision': 0.8857240905971173, 'recall': 0.910726887791108, 'f1-score': 0.8980514961725817, 'support': 11336.0}	{'precision': 0.9990285260090083, 'recall': 1.0, 'f1-score': 0.9995140269494146, 'support': 11312.0}	0.8978	{'precision': 0.8305645595521495, 'recall': 0.8221861980234124, 'f1-score': 0.8261913703083273, 'support': 29705.0}	{'precision': 0.8956984397278125, 'recall': 0.897794984009426, 'f1-score': 0.8965631137613624, 'support': 29705.0}
No log	10.0	410	0.3676	{'precision': 0.6591760299625468, 'recall': 0.6494464944649446, 'f1-score': 0.6542750929368029, 'support': 271.0}	{'precision': 0.8591549295774648, 'recall': 0.8776978417266187, 'f1-score': 0.8683274021352312, 'support': 139.0}	{'precision': 0.8817034700315457, 'recall': 0.8830963665086888, 'f1-score': 0.8823993685872139, 'support': 633.0}	{'precision': 0.657699205739175, 'recall': 0.6415896025993502, 'f1-score': 0.6495445344129553, 'support': 4001.0}	{'precision': 0.8908252178370066, 'recall': 0.8633879781420765, 'f1-score': 0.8768920282542885, 'support': 2013.0}	{'precision': 0.893792983372508, 'recall': 0.9056986591390261, 'f1-score': 0.8997064364895061, 'support': 11336.0}	{'precision': 0.999028354385655, 'recall': 0.9998231966053748, 'f1-score': 0.9994256174612292, 'support': 11312.0}	0.9002	{'precision': 0.8344828844151289, 'recall': 0.8315343055980114, 'f1-score': 0.8329386400396039, 'support': 29705.0}	{'precision': 0.8993068246372901, 'recall': 0.9001514896482073, 'f1-score': 0.8996852699285292, 'support': 29705.0}
No log	11.0	451	0.3859	{'precision': 0.6321070234113713, 'recall': 0.6974169741697417, 'f1-score': 0.6631578947368422, 'support': 271.0}	{'precision': 0.8289473684210527, 'recall': 0.9064748201438849, 'f1-score': 0.8659793814432989, 'support': 139.0}	{'precision': 0.9037162162162162, 'recall': 0.8451816745655608, 'f1-score': 0.873469387755102, 'support': 633.0}	{'precision': 0.6186218927169647, 'recall': 0.7090727318170458, 'f1-score': 0.6607662746011413, 'support': 4001.0}	{'precision': 0.8550519357884797, 'recall': 0.899652260307998, 'f1-score': 0.8767852820140403, 'support': 2013.0}	{'precision': 0.9189468735307945, 'recall': 0.86212067748765, 'f1-score': 0.8896272359018706, 'support': 11336.0}	{'precision': 0.9990285260090083, 'recall': 1.0, 'f1-score': 0.9995140269494146, 'support': 11312.0}	0.8949	{'precision': 0.8223456908705552, 'recall': 0.8457027340702687, 'f1-score': 0.8327570690573871, 'support': 29705.0}	{'precision': 0.90129928417713, 'recall': 0.8948998485103518, 'f1-score': 0.8972565124508365, 'support': 29705.0}
No log	12.0	492	0.3883	{'precision': 0.6756756756756757, 'recall': 0.6457564575645757, 'f1-score': 0.660377358490566, 'support': 271.0}	{'precision': 0.8880597014925373, 'recall': 0.8561151079136691, 'f1-score': 0.8717948717948718, 'support': 139.0}	{'precision': 0.8753846153846154, 'recall': 0.8988941548183255, 'f1-score': 0.8869836321122371, 'support': 633.0}	{'precision': 0.656190734218369, 'recall': 0.6053486628342915, 'f1-score': 0.6297451898075924, 'support': 4001.0}	{'precision': 0.9079365079365079, 'recall': 0.8524590163934426, 'f1-score': 0.8793235972328978, 'support': 2013.0}	{'precision': 0.8805576808637252, 'recall': 0.9137261820748059, 'f1-score': 0.8968353608381315, 'support': 11336.0}	{'precision': 0.9994698710019438, 'recall': 1.0, 'f1-score': 0.9997348652231551, 'support': 11312.0}	0.8979	{'precision': 0.8404678266533392, 'recall': 0.82461422594273, 'f1-score': 0.832113553642779, 'support': 29705.0}	{'precision': 0.895531635660094, 'recall': 0.8978623127419626, 'f1-score': 0.8963749450251765, 'support': 29705.0}
0.2074	13.0	533	0.3891	{'precision': 0.6781609195402298, 'recall': 0.6531365313653137, 'f1-score': 0.6654135338345865, 'support': 271.0}	{'precision': 0.8840579710144928, 'recall': 0.8776978417266187, 'f1-score': 0.8808664259927799, 'support': 139.0}	{'precision': 0.8773291925465838, 'recall': 0.8925750394944708, 'f1-score': 0.8848864526233359, 'support': 633.0}	{'precision': 0.6596830513027129, 'recall': 0.6138465383654087, 'f1-score': 0.6359399274987053, 'support': 4001.0}	{'precision': 0.9052631578947369, 'recall': 0.8544461003477397, 'f1-score': 0.8791208791208791, 'support': 2013.0}	{'precision': 0.8835423598669055, 'recall': 0.9135497529992943, 'f1-score': 0.898295528472915, 'support': 11336.0}	{'precision': 0.9994698710019438, 'recall': 1.0, 'f1-score': 0.9997348652231551, 'support': 11312.0}	0.8991	{'precision': 0.8410723604525151, 'recall': 0.8293216863284066, 'f1-score': 0.8348939446809082, 'support': 29705.0}	{'precision': 0.8970052531014272, 'recall': 0.8991078942938899, 'f1-score': 0.8977965161137413, 'support': 29705.0}

Framework versions

• Transformers 4.37.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.0
• Tokenizers 0.15.2