--- license: apache-2.0 base_model: allenai/longformer-base-4096 tags: - generated_from_trainer datasets: - essays_su_g metrics: - accuracy model-index: - name: longformer-sep_tok_full_labels results: - task: name: Token Classification type: token-classification dataset: name: essays_su_g type: essays_su_g config: sep_tok_full_labels split: train[80%:100%] args: sep_tok_full_labels metrics: - name: Accuracy type: accuracy value: 0.8938225887897661 --- # longformer-sep_tok_full_labels This model is a fine-tuned version of [allenai/longformer-base-4096](https://huggingface.co./allenai/longformer-base-4096) on the essays_su_g dataset. It achieves the following results on the evaluation set: - Loss: 0.2832 - B-claim: {'precision': 0.65, 'recall': 0.5756457564575646, 'f1-score': 0.6105675146771037, 'support': 271.0} - B-majorclaim: {'precision': 0.8206896551724138, 'recall': 0.8561151079136691, 'f1-score': 0.8380281690140844, 'support': 139.0} - B-premise: {'precision': 0.8632218844984803, 'recall': 0.8973143759873617, 'f1-score': 0.8799380325329202, 'support': 633.0} - I-claim: {'precision': 0.6347290007706139, 'recall': 0.6175956010997251, 'f1-score': 0.6260450975424373, 'support': 4001.0} - I-majorclaim: {'precision': 0.8701298701298701, 'recall': 0.8653750620963736, 'f1-score': 0.8677459526774596, 'support': 2013.0} - I-premise: {'precision': 0.8889858614068773, 'recall': 0.8985532815808045, 'f1-score': 0.893743967710801, 'support': 11336.0} - O: {'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0} - Accuracy: 0.8938 - Macro avg: {'precision': 0.8182508959968936, 'recall': 0.81576199714908, 'f1-score': 0.8165623020032957, 'support': 29705.0} - Weighted avg: {'precision': 0.8926884767049156, 'recall': 0.8938225887897661, 'f1-score': 0.8932001664235063, 'support': 29705.0} ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 8 - eval_batch_size: 8 - seed: 42 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - num_epochs: 7 ### Training results | Training Loss | Epoch | Step | Validation Loss | B-claim | B-majorclaim | B-premise | I-claim | I-majorclaim | I-premise | O | Accuracy | Macro avg | Weighted avg | |:-------------:|:-----:|:----:|:---------------:|:-------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|:--------:|:-------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:| | No log | 1.0 | 41 | 0.3941 | {'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 271.0} | {'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 139.0} | {'precision': 0.6378035902851109, 'recall': 0.9541864139020537, 'f1-score': 0.7645569620253164, 'support': 633.0} | {'precision': 0.44665194996320823, 'recall': 0.303424143964009, 'f1-score': 0.3613632981098378, 'support': 4001.0} | {'precision': 0.7471544715447155, 'recall': 0.4565325384997516, 'f1-score': 0.5667591736046871, 'support': 2013.0} | {'precision': 0.8023273050696709, 'recall': 0.9549223712067749, 'f1-score': 0.871999355566296, 'support': 11336.0} | {'precision': 0.9959356776815692, 'recall': 0.9964639321074965, 'f1-score': 0.9961997348652232, 'support': 11312.0} | 0.8360 | {'precision': 0.5185532849348964, 'recall': 0.5236470570971551, 'f1-score': 0.5086969320244801, 'support': 29705.0} | {'precision': 0.8098304255807603, 'recall': 0.8360208719070863, 'f1-score': 0.8155078749254555, 'support': 29705.0} | | No log | 2.0 | 82 | 0.3007 | {'precision': 0.3785310734463277, 'recall': 0.24723247232472326, 'f1-score': 0.29910714285714285, 'support': 271.0} | {'precision': 0.8823529411764706, 'recall': 0.1079136690647482, 'f1-score': 0.1923076923076923, 'support': 139.0} | {'precision': 0.7395957193816884, 'recall': 0.9826224328593997, 'f1-score': 0.8439620081411127, 'support': 633.0} | {'precision': 0.566340160284951, 'recall': 0.3179205198700325, 'f1-score': 0.40723547302705304, 'support': 4001.0} | {'precision': 0.7801980198019802, 'recall': 0.7829110779930452, 'f1-score': 0.7815521943962312, 'support': 2013.0} | {'precision': 0.8230264162467552, 'recall': 0.9509527170077628, 'f1-score': 0.8823770156339527, 'support': 11336.0} | {'precision': 0.9998231027772864, 'recall': 0.9992927864214993, 'f1-score': 0.9995578742594394, 'support': 11312.0} | 0.8630 | {'precision': 0.7385524904450655, 'recall': 0.6269779536487444, 'f1-score': 0.6294427715175177, 'support': 29705.0} | {'precision': 0.8473214966654586, 'recall': 0.863019693654267, 'f1-score': 0.8468020526459685, 'support': 29705.0} | | No log | 3.0 | 123 | 0.2656 | {'precision': 0.5555555555555556, 'recall': 0.44280442804428044, 'f1-score': 0.4928131416837782, 'support': 271.0} | {'precision': 0.8367346938775511, 'recall': 0.5899280575539568, 'f1-score': 0.6919831223628692, 'support': 139.0} | {'precision': 0.8120713305898491, 'recall': 0.9352290679304898, 'f1-score': 0.8693098384728339, 'support': 633.0} | {'precision': 0.6189608021877848, 'recall': 0.50912271932017, 'f1-score': 0.5586944596818432, 'support': 4001.0} | {'precision': 0.8277078085642318, 'recall': 0.8161947342275211, 'f1-score': 0.821910955477739, 'support': 2013.0} | {'precision': 0.8641668736031786, 'recall': 0.9209597741707833, 'f1-score': 0.8916599051970789, 'support': 11336.0} | {'precision': 1.0, 'recall': 0.9993811881188119, 'f1-score': 0.9996904982977407, 'support': 11312.0} | 0.8826 | {'precision': 0.7878852949111643, 'recall': 0.7448028527665732, 'f1-score': 0.7608659887391263, 'support': 29705.0} | {'precision': 0.8763419120561073, 'recall': 0.8826460191886888, 'f1-score': 0.8781757290617714, 'support': 29705.0} | | No log | 4.0 | 164 | 0.2673 | {'precision': 0.6179245283018868, 'recall': 0.4833948339483395, 'f1-score': 0.5424430641821946, 'support': 271.0} | {'precision': 0.7857142857142857, 'recall': 0.7913669064748201, 'f1-score': 0.7885304659498209, 'support': 139.0} | {'precision': 0.8393632416787264, 'recall': 0.9162717219589257, 'f1-score': 0.8761329305135951, 'support': 633.0} | {'precision': 0.6405669929125886, 'recall': 0.47438140464883777, 'f1-score': 0.5450890292935094, 'support': 4001.0} | {'precision': 0.8220742150333016, 'recall': 0.8584202682563339, 'f1-score': 0.839854191980559, 'support': 2013.0} | {'precision': 0.8581196581196581, 'recall': 0.9299576570218772, 'f1-score': 0.8925955717370136, 'support': 11336.0} | {'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0} | 0.8854 | {'precision': 0.7948232745372067, 'recall': 0.7791132560441619, 'f1-score': 0.7835207505223847, 'support': 29705.0} | {'precision': 0.8774744030572791, 'recall': 0.8854064972226898, 'f1-score': 0.8790840278083316, 'support': 29705.0} | | No log | 5.0 | 205 | 0.2698 | {'precision': 0.631578947368421, 'recall': 0.6642066420664207, 'f1-score': 0.6474820143884892, 'support': 271.0} | {'precision': 0.8013245033112583, 'recall': 0.8705035971223022, 'f1-score': 0.8344827586206897, 'support': 139.0} | {'precision': 0.8976897689768977, 'recall': 0.8593996840442338, 'f1-score': 0.8781275221953189, 'support': 633.0} | {'precision': 0.6005734450816056, 'recall': 0.680579855036241, 'f1-score': 0.6380785002929115, 'support': 4001.0} | {'precision': 0.8148654810761514, 'recall': 0.8877297565822156, 'f1-score': 0.849738468854018, 'support': 2013.0} | {'precision': 0.9169016732468509, 'recall': 0.8604446012702893, 'f1-score': 0.8877764630927459, 'support': 11336.0} | {'precision': 1.0, 'recall': 0.9987623762376238, 'f1-score': 0.9993808049535603, 'support': 11312.0} | 0.8890 | {'precision': 0.8089905455801693, 'recall': 0.8316609303370467, 'f1-score': 0.819295218913962, 'support': 29705.0} | {'precision': 0.8954719932524351, 'recall': 0.8889749200471301, 'f1-score': 0.8914196505512982, 'support': 29705.0} | | No log | 6.0 | 246 | 0.2851 | {'precision': 0.61328125, 'recall': 0.5793357933579336, 'f1-score': 0.5958254269449716, 'support': 271.0} | {'precision': 0.8717948717948718, 'recall': 0.7338129496402878, 'f1-score': 0.796875, 'support': 139.0} | {'precision': 0.8522388059701492, 'recall': 0.9020537124802528, 'f1-score': 0.8764389869531849, 'support': 633.0} | {'precision': 0.6405070118662352, 'recall': 0.5936015996001, 'f1-score': 0.6161629264496044, 'support': 4001.0} | {'precision': 0.8946784922394678, 'recall': 0.8017883755588674, 'f1-score': 0.845690332722033, 'support': 2013.0} | {'precision': 0.8785780629907962, 'recall': 0.9178722653493295, 'f1-score': 0.8977954182665343, 'support': 11336.0} | {'precision': 1.0, 'recall': 0.999557991513437, 'f1-score': 0.9997789469030461, 'support': 11312.0} | 0.8931 | {'precision': 0.8215826421230743, 'recall': 0.789717526785744, 'f1-score': 0.8040810054627677, 'support': 29705.0} | {'precision': 0.890828586148026, 'recall': 0.8931493014644, 'f1-score': 0.8914853059005039, 'support': 29705.0} | | No log | 7.0 | 287 | 0.2832 | {'precision': 0.65, 'recall': 0.5756457564575646, 'f1-score': 0.6105675146771037, 'support': 271.0} | {'precision': 0.8206896551724138, 'recall': 0.8561151079136691, 'f1-score': 0.8380281690140844, 'support': 139.0} | {'precision': 0.8632218844984803, 'recall': 0.8973143759873617, 'f1-score': 0.8799380325329202, 'support': 633.0} | {'precision': 0.6347290007706139, 'recall': 0.6175956010997251, 'f1-score': 0.6260450975424373, 'support': 4001.0} | {'precision': 0.8701298701298701, 'recall': 0.8653750620963736, 'f1-score': 0.8677459526774596, 'support': 2013.0} | {'precision': 0.8889858614068773, 'recall': 0.8985532815808045, 'f1-score': 0.893743967710801, 'support': 11336.0} | {'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0} | 0.8938 | {'precision': 0.8182508959968936, 'recall': 0.81576199714908, 'f1-score': 0.8165623020032957, 'support': 29705.0} | {'precision': 0.8926884767049156, 'recall': 0.8938225887897661, 'f1-score': 0.8932001664235063, 'support': 29705.0} | ### Framework versions - Transformers 4.37.2 - Pytorch 2.2.0+cu121 - Datasets 2.17.0 - Tokenizers 0.15.2