metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-sep_tok_full_labels
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: sep_tok_full_labels
          split: train[80%:100%]
          args: sep_tok_full_labels
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8965157380912304

longformer-sep_tok_full_labels

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

Loss: 0.3135
B-claim: {'precision': 0.6573705179282868, 'recall': 0.6088560885608856, 'f1-score': 0.632183908045977, 'support': 271.0}
B-majorclaim: {'precision': 0.8489208633093526, 'recall': 0.8489208633093526, 'f1-score': 0.8489208633093525, 'support': 139.0}
B-premise: {'precision': 0.8698315467075038, 'recall': 0.8973143759873617, 'f1-score': 0.8833592534992224, 'support': 633.0}
I-claim: {'precision': 0.6493333333333333, 'recall': 0.6085978505373657, 'f1-score': 0.6283060250290285, 'support': 4001.0}
I-majorclaim: {'precision': 0.8869202709744659, 'recall': 0.8455042225534029, 'f1-score': 0.8657171922685657, 'support': 2013.0}
I-premise: {'precision': 0.884437596302003, 'recall': 0.9114326040931545, 'f1-score': 0.897732209575115, 'support': 11336.0}
O: {'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0}
Accuracy: 0.8965
Macro avg: {'precision': 0.8281163040792779, 'recall': 0.8172196576206014, 'f1-score': 0.8223107498464006, 'support': 29705.0}
Weighted avg: {'precision': 0.8943978637684629, 'recall': 0.8965157380912304, 'f1-score': 0.8952439544307468, 'support': 29705.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 9

Training results

Training Loss	Epoch	Step	Validation Loss	B-claim	B-majorclaim	B-premise	I-claim	I-majorclaim	I-premise	O	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.4270	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 271.0}	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 139.0}	{'precision': 0.6227795193312434, 'recall': 0.9415481832543444, 'f1-score': 0.7496855345911949, 'support': 633.0}	{'precision': 0.4119028974158183, 'recall': 0.26293426643339163, 'f1-score': 0.32097635392829904, 'support': 4001.0}	{'precision': 0.7732342007434945, 'recall': 0.30998509687034276, 'f1-score': 0.4425531914893617, 'support': 2013.0}	{'precision': 0.7799929303640862, 'recall': 0.9732709950599859, 'f1-score': 0.8659785722695342, 'support': 11336.0}	{'precision': 0.9988436221312934, 'recall': 0.9926626591230552, 'f1-score': 0.9957435488161747, 'support': 11312.0}	0.8259	{'precision': 0.5123933099979908, 'recall': 0.49720017153444573, 'f1-score': 0.4821338858706521, 'support': 29705.0}	{'precision': 0.7991813595876281, 'recall': 0.8259215620265948, 'f1-score': 0.7988628523611326, 'support': 29705.0}
No log	2.0	82	0.3021	{'precision': 0.39226519337016574, 'recall': 0.26199261992619927, 'f1-score': 0.3141592920353982, 'support': 271.0}	{'precision': 0.8275862068965517, 'recall': 0.17266187050359713, 'f1-score': 0.28571428571428575, 'support': 139.0}	{'precision': 0.7603911980440098, 'recall': 0.9826224328593997, 'f1-score': 0.8573397656788421, 'support': 633.0}	{'precision': 0.5958279009126467, 'recall': 0.34266433391652085, 'f1-score': 0.4350999682640432, 'support': 4001.0}	{'precision': 0.756928668786915, 'recall': 0.8276204669647292, 'f1-score': 0.7906976744186046, 'support': 2013.0}	{'precision': 0.8333980281034081, 'recall': 0.9469830628087509, 'f1-score': 0.8865672874427056, 'support': 11336.0}	{'precision': 0.9999114574110147, 'recall': 0.9983203677510608, 'f1-score': 0.9991152791294347, 'support': 11312.0}	0.8679	{'precision': 0.7380440933606731, 'recall': 0.6475521649614654, 'f1-score': 0.6526705075261877, 'support': 29705.0}	{'precision': 0.8540201741510516, 'recall': 0.8679346911294394, 'f1-score': 0.8534649293089497, 'support': 29705.0}
No log	3.0	123	0.2667	{'precision': 0.5930232558139535, 'recall': 0.5645756457564576, 'f1-score': 0.5784499054820417, 'support': 271.0}	{'precision': 0.7983193277310925, 'recall': 0.6834532374100719, 'f1-score': 0.7364341085271319, 'support': 139.0}	{'precision': 0.8586466165413534, 'recall': 0.9020537124802528, 'f1-score': 0.8798151001540832, 'support': 633.0}	{'precision': 0.6084305408271474, 'recall': 0.5736065983504124, 'f1-score': 0.5905055962948668, 'support': 4001.0}	{'precision': 0.7296868806975823, 'recall': 0.914555389965226, 'f1-score': 0.8117283950617283, 'support': 2013.0}	{'precision': 0.9037331646027298, 'recall': 0.8819689484827099, 'f1-score': 0.8927184249296843, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9993811881188119, 'f1-score': 0.9996904982977407, 'support': 11312.0}	0.8840	{'precision': 0.784548540887694, 'recall': 0.7885135315091346, 'f1-score': 0.7841917183924682, 'support': 29705.0}	{'precision': 0.8845348741215646, 'recall': 0.8839589294731527, 'f1-score': 0.8833874551641274, 'support': 29705.0}
No log	4.0	164	0.2619	{'precision': 0.6512605042016807, 'recall': 0.5719557195571956, 'f1-score': 0.6090373280943026, 'support': 271.0}	{'precision': 0.7908496732026143, 'recall': 0.8705035971223022, 'f1-score': 0.8287671232876712, 'support': 139.0}	{'precision': 0.8696319018404908, 'recall': 0.8957345971563981, 'f1-score': 0.8824902723735409, 'support': 633.0}	{'precision': 0.6461988304093568, 'recall': 0.5523619095226193, 'f1-score': 0.5956070610429861, 'support': 4001.0}	{'precision': 0.809009009009009, 'recall': 0.892200695479384, 'f1-score': 0.8485707536026459, 'support': 2013.0}	{'precision': 0.8816800409766092, 'recall': 0.9110797459421313, 'f1-score': 0.8961388286334057, 'support': 11336.0}	{'precision': 0.9999115670321896, 'recall': 0.999557991513437, 'f1-score': 0.9997347480106101, 'support': 11312.0}	0.8916	{'precision': 0.8069345038102786, 'recall': 0.8133420366133525, 'f1-score': 0.8086208735778804, 'support': 29705.0}	{'precision': 0.8872781666877897, 'recall': 0.8915670762497896, 'f1-score': 0.8886615080822224, 'support': 29705.0}
No log	5.0	205	0.2628	{'precision': 0.6589147286821705, 'recall': 0.6273062730627307, 'f1-score': 0.6427221172022685, 'support': 271.0}	{'precision': 0.8187919463087249, 'recall': 0.8776978417266187, 'f1-score': 0.8472222222222222, 'support': 139.0}	{'precision': 0.8820754716981132, 'recall': 0.8862559241706162, 'f1-score': 0.8841607565011821, 'support': 633.0}	{'precision': 0.6446451612903226, 'recall': 0.6243439140214946, 'f1-score': 0.6343321482986287, 'support': 4001.0}	{'precision': 0.8322550831792976, 'recall': 0.8946845504222554, 'f1-score': 0.8623413933445057, 'support': 2013.0}	{'precision': 0.8981178757621278, 'recall': 0.8966125617501765, 'f1-score': 0.8973645874718581, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9994695898161244, 'f1-score': 0.9997347245556637, 'support': 11312.0}	0.8962	{'precision': 0.8192571809886795, 'recall': 0.8294815221385737, 'f1-score': 0.823982564228047, 'support': 29705.0}	{'precision': 0.895416622050048, 'recall': 0.8962127587948157, 'f1-score': 0.8957076471777784, 'support': 29705.0}
No log	6.0	246	0.2929	{'precision': 0.6412213740458015, 'recall': 0.6199261992619927, 'f1-score': 0.6303939962476548, 'support': 271.0}	{'precision': 0.8688524590163934, 'recall': 0.762589928057554, 'f1-score': 0.8122605363984674, 'support': 139.0}	{'precision': 0.8679817905918058, 'recall': 0.9036334913112164, 'f1-score': 0.8854489164086687, 'support': 633.0}	{'precision': 0.6434381778741866, 'recall': 0.5931017245688578, 'f1-score': 0.6172454155286773, 'support': 4001.0}	{'precision': 0.9013683010262258, 'recall': 0.7853949329359166, 'f1-score': 0.8393947438279799, 'support': 2013.0}	{'precision': 0.8776658270361041, 'recall': 0.922106563161609, 'f1-score': 0.8993375204336229, 'support': 11336.0}	{'precision': 0.9999115826702034, 'recall': 0.9997347949080623, 'f1-score': 0.9998231809742728, 'support': 11312.0}	0.8942	{'precision': 0.8286342160372459, 'recall': 0.7980696620293155, 'f1-score': 0.811986329974192, 'support': 29705.0}	{'precision': 0.8918715413466195, 'recall': 0.8941928968187174, 'f1-score': 0.8923891813937611, 'support': 29705.0}
No log	7.0	287	0.3174	{'precision': 0.6396761133603239, 'recall': 0.5830258302583026, 'f1-score': 0.61003861003861, 'support': 271.0}	{'precision': 0.8974358974358975, 'recall': 0.7553956834532374, 'f1-score': 0.8203124999999999, 'support': 139.0}	{'precision': 0.8512518409425626, 'recall': 0.9131121642969984, 'f1-score': 0.8810975609756098, 'support': 633.0}	{'precision': 0.6256590509666081, 'recall': 0.5338665333666583, 'f1-score': 0.5761294672960215, 'support': 4001.0}	{'precision': 0.9229813664596274, 'recall': 0.7382016890213612, 'f1-score': 0.8203146563621309, 'support': 2013.0}	{'precision': 0.8586727243225701, 'recall': 0.9336626676076217, 'f1-score': 0.8945989350012677, 'support': 11336.0}	{'precision': 0.9999115983026874, 'recall': 0.9999115983026874, 'f1-score': 0.9999115983026874, 'support': 11312.0}	0.8873	{'precision': 0.827941227398611, 'recall': 0.7795965951866952, 'f1-score': 0.8003433325680467, 'support': 29705.0}	{'precision': 0.8834565086114177, 'recall': 0.8873253660999831, 'f1-score': 0.8835428239690539, 'support': 29705.0}
No log	8.0	328	0.3181	{'precision': 0.6652719665271967, 'recall': 0.5867158671586716, 'f1-score': 0.6235294117647059, 'support': 271.0}	{'precision': 0.8344827586206897, 'recall': 0.8705035971223022, 'f1-score': 0.852112676056338, 'support': 139.0}	{'precision': 0.8649468892261002, 'recall': 0.9004739336492891, 'f1-score': 0.8823529411764706, 'support': 633.0}	{'precision': 0.6542391604529135, 'recall': 0.5921019745063734, 'f1-score': 0.6216216216216215, 'support': 4001.0}	{'precision': 0.8813219829744617, 'recall': 0.8743169398907104, 'f1-score': 0.8778054862842893, 'support': 2013.0}	{'precision': 0.881084306538232, 'recall': 0.9117854622441779, 'f1-score': 0.8961720206355399, 'support': 11336.0}	{'precision': 0.9999116061168567, 'recall': 1.0, 'f1-score': 0.9999558011049724, 'support': 11312.0}	0.8964	{'precision': 0.8258940957794929, 'recall': 0.8194139677959321, 'f1-score': 0.8219357083777054, 'support': 29705.0}	{'precision': 0.8932663527105956, 'recall': 0.8963810806261572, 'f1-score': 0.8944819439268501, 'support': 29705.0}
No log	9.0	369	0.3135	{'precision': 0.6573705179282868, 'recall': 0.6088560885608856, 'f1-score': 0.632183908045977, 'support': 271.0}	{'precision': 0.8489208633093526, 'recall': 0.8489208633093526, 'f1-score': 0.8489208633093525, 'support': 139.0}	{'precision': 0.8698315467075038, 'recall': 0.8973143759873617, 'f1-score': 0.8833592534992224, 'support': 633.0}	{'precision': 0.6493333333333333, 'recall': 0.6085978505373657, 'f1-score': 0.6283060250290285, 'support': 4001.0}	{'precision': 0.8869202709744659, 'recall': 0.8455042225534029, 'f1-score': 0.8657171922685657, 'support': 2013.0}	{'precision': 0.884437596302003, 'recall': 0.9114326040931545, 'f1-score': 0.897732209575115, 'support': 11336.0}	{'precision': 1.0, 'recall': 0.9999115983026874, 'f1-score': 0.9999557971975424, 'support': 11312.0}	0.8965	{'precision': 0.8281163040792779, 'recall': 0.8172196576206014, 'f1-score': 0.8223107498464006, 'support': 29705.0}	{'precision': 0.8943978637684629, 'recall': 0.8965157380912304, 'f1-score': 0.8952439544307468, 'support': 29705.0}

Framework versions

Transformers 4.37.2
Pytorch 2.2.0+cu121
Datasets 2.17.0
Tokenizers 0.15.2