File size: 16,136 Bytes
d7e7d38 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 |
---
license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
- generated_from_trainer
datasets:
- essays_su_g
metrics:
- accuracy
model-index:
- name: longformer-sep_tok_full_labels
results:
- task:
name: Token Classification
type: token-classification
dataset:
name: essays_su_g
type: essays_su_g
config: sep_tok_full_labels
split: train[80%:100%]
args: sep_tok_full_labels
metrics:
- name: Accuracy
type: accuracy
value: 0.8970543679515233
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# longformer-sep_tok_full_labels
This model is a fine-tuned version of [allenai/longformer-base-4096](https://huggingface.co./allenai/longformer-base-4096) on the essays_su_g dataset.
It achieves the following results on the evaluation set:
- Loss: 0.3490
- B-claim: {'precision': 0.6679841897233202, 'recall': 0.6236162361623616, 'f1-score': 0.6450381679389313, 'support': 271.0}
- B-majorclaim: {'precision': 0.851063829787234, 'recall': 0.8633093525179856, 'f1-score': 0.8571428571428571, 'support': 139.0}
- B-premise: {'precision': 0.8736517719568567, 'recall': 0.8957345971563981, 'f1-score': 0.8845553822152886, 'support': 633.0}
- I-claim: {'precision': 0.6537938536850694, 'recall': 0.6008497875531117, 'f1-score': 0.6262047408179214, 'support': 4001.0}
- I-majorclaim: {'precision': 0.9007751937984496, 'recall': 0.8658718330849479, 'f1-score': 0.8829787234042552, 'support': 2013.0}
- I-premise: {'precision': 0.8802487011327825, 'recall': 0.911697247706422, 'f1-score': 0.8956970143432854, 'support': 11336.0}
- O: {'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0}
- Accuracy: 0.8971
- Macro avg: {'precision': 0.8325025057262447, 'recall': 0.8229734070127556, 'f1-score': 0.8273548951044003, 'support': 29705.0}
- Weighted avg: {'precision': 0.8945272549791482, 'recall': 0.8970543679515233, 'f1-score': 0.8955015738591867, 'support': 29705.0}
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10
### Training results
| Training Loss | Epoch | Step | Validation Loss | B-claim | B-majorclaim | B-premise | I-claim | I-majorclaim | I-premise | O | Accuracy | Macro avg | Weighted avg |
|:-------------:|:-----:|:----:|:---------------:|:------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|:--------:|:-------------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------------:|
| No log | 1.0 | 41 | 0.4114 | {'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 271.0} | {'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 139.0} | {'precision': 0.6441908713692946, 'recall': 0.981042654028436, 'f1-score': 0.7777082028804008, 'support': 633.0} | {'precision': 0.45472716370177896, 'recall': 0.5686078480379905, 'f1-score': 0.5053309640159929, 'support': 4001.0} | {'precision': 0.7067901234567902, 'recall': 0.4550422255340288, 'f1-score': 0.5536415835599879, 'support': 2013.0} | {'precision': 0.8708348262271587, 'recall': 0.857621736062103, 'f1-score': 0.8641777777777776, 'support': 11336.0} | {'precision': 0.9955665898208902, 'recall': 0.9925742574257426, 'f1-score': 0.9940681717574147, 'support': 11312.0} | 0.8336 | {'precision': 0.5245870820822732, 'recall': 0.5506983887269001, 'f1-score': 0.5278466714273676, 'support': 29705.0} | {'precision': 0.834322086390543, 'recall': 0.8335970375357684, 'f1-score': 0.830493769902922, 'support': 29705.0} |
| No log | 2.0 | 82 | 0.2953 | {'precision': 0.4044943820224719, 'recall': 0.2656826568265683, 'f1-score': 0.32071269487750553, 'support': 271.0} | {'precision': 0.875, 'recall': 0.2014388489208633, 'f1-score': 0.327485380116959, 'support': 139.0} | {'precision': 0.7609756097560976, 'recall': 0.985781990521327, 'f1-score': 0.8589125946317963, 'support': 633.0} | {'precision': 0.5836245631552671, 'recall': 0.29217695576105973, 'f1-score': 0.38940706195869423, 'support': 4001.0} | {'precision': 0.7398819561551433, 'recall': 0.8718330849478391, 'f1-score': 0.8004561003420751, 'support': 2013.0} | {'precision': 0.8254749634643489, 'recall': 0.9467184191954834, 'f1-score': 0.8819492953116653, 'support': 11336.0} | {'precision': 0.9999114965926188, 'recall': 0.9987623762376238, 'f1-score': 0.9993366060766884, 'support': 11312.0} | 0.8644 | {'precision': 0.741337567306564, 'recall': 0.6517706189158234, 'f1-score': 0.6540371047593406, 'support': 29705.0} | {'precision': 0.8485436064203262, 'recall': 0.8644335970375358, 'f1-score': 0.8465825270165918, 'support': 29705.0} |
| No log | 3.0 | 123 | 0.2542 | {'precision': 0.5887445887445888, 'recall': 0.5018450184501845, 'f1-score': 0.5418326693227091, 'support': 271.0} | {'precision': 0.8333333333333334, 'recall': 0.6474820143884892, 'f1-score': 0.7287449392712549, 'support': 139.0} | {'precision': 0.8323863636363636, 'recall': 0.9257503949447078, 'f1-score': 0.8765893792071803, 'support': 633.0} | {'precision': 0.6355166572557877, 'recall': 0.5626093476630842, 'f1-score': 0.5968447567280923, 'support': 4001.0} | {'precision': 0.8084507042253521, 'recall': 0.8554396423248882, 'f1-score': 0.831281679942071, 'support': 2013.0} | {'precision': 0.8828767123287671, 'recall': 0.9096683133380381, 'f1-score': 0.8960722975321516, 'support': 11336.0} | {'precision': 1.0, 'recall': 0.9998231966053748, 'f1-score': 0.9999115904871364, 'support': 11312.0} | 0.8890 | {'precision': 0.7973297656463132, 'recall': 0.7718025611021095, 'f1-score': 0.7816110446415138, 'support': 29705.0} | {'precision': 0.8851266624914956, 'recall': 0.8889749200471301, 'f1-score': 0.886491656178783, 'support': 29705.0} |
| No log | 4.0 | 164 | 0.2729 | {'precision': 0.6325757575757576, 'recall': 0.6162361623616236, 'f1-score': 0.6242990654205607, 'support': 271.0} | {'precision': 0.75, 'recall': 0.8848920863309353, 'f1-score': 0.8118811881188119, 'support': 139.0} | {'precision': 0.8959349593495934, 'recall': 0.8704581358609794, 'f1-score': 0.8830128205128205, 'support': 633.0} | {'precision': 0.6295979469632165, 'recall': 0.5518620344913772, 'f1-score': 0.5881726158763986, 'support': 4001.0} | {'precision': 0.7551271534044298, 'recall': 0.914555389965226, 'f1-score': 0.8272298359919119, 'support': 2013.0} | {'precision': 0.8926786497150373, 'recall': 0.8981122088920254, 'f1-score': 0.8953871861395717, 'support': 11336.0} | {'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 11312.0} | 0.8882 | {'precision': 0.7937020667154335, 'recall': 0.8194451454145952, 'f1-score': 0.8042832445800107, 'support': 29705.0} | {'precision': 0.8858206489932313, 'recall': 0.8881669752566907, 'f1-score': 0.886099481074347, 'support': 29705.0} |
| No log | 5.0 | 205 | 0.2636 | {'precision': 0.6716417910447762, 'recall': 0.6642066420664207, 'f1-score': 0.6679035250463822, 'support': 271.0} | {'precision': 0.8116883116883117, 'recall': 0.8992805755395683, 'f1-score': 0.8532423208191127, 'support': 139.0} | {'precision': 0.8971061093247589, 'recall': 0.8815165876777251, 'f1-score': 0.8892430278884461, 'support': 633.0} | {'precision': 0.6369410569105691, 'recall': 0.6265933516620845, 'f1-score': 0.6317248330603503, 'support': 4001.0} | {'precision': 0.836166194523135, 'recall': 0.8797814207650273, 'f1-score': 0.8574195110142822, 'support': 2013.0} | {'precision': 0.8971772409521281, 'recall': 0.8944071983062809, 'f1-score': 0.8957900781905729, 'support': 11336.0} | {'precision': 1.0, 'recall': 0.9994695898161244, 'f1-score': 0.9997347245556637, 'support': 11312.0} | 0.8950 | {'precision': 0.82153152920624, 'recall': 0.8350364808333187, 'f1-score': 0.827865431510687, 'support': 29705.0} | {'precision': 0.8946881901904717, 'recall': 0.8950008416091567, 'f1-score': 0.8947882658752071, 'support': 29705.0} |
| No log | 6.0 | 246 | 0.2841 | {'precision': 0.6382252559726962, 'recall': 0.6900369003690037, 'f1-score': 0.6631205673758864, 'support': 271.0} | {'precision': 0.8671875, 'recall': 0.7985611510791367, 'f1-score': 0.8314606741573035, 'support': 139.0} | {'precision': 0.8903225806451613, 'recall': 0.8720379146919431, 'f1-score': 0.8810853950518756, 'support': 633.0} | {'precision': 0.6216798277099784, 'recall': 0.6493376655836041, 'f1-score': 0.6352078239608802, 'support': 4001.0} | {'precision': 0.9020251778872469, 'recall': 0.8186785891703925, 'f1-score': 0.8583333333333334, 'support': 2013.0} | {'precision': 0.891806167400881, 'recall': 0.8929075511644319, 'f1-score': 0.8923565194393016, 'support': 11336.0} | {'precision': 1.0, 'recall': 0.9996463932107497, 'f1-score': 0.9998231653404068, 'support': 11312.0} | 0.8930 | {'precision': 0.8301780728022805, 'recall': 0.8173151664670374, 'f1-score': 0.823055354094141, 'support': 29705.0} | {'precision': 0.8948562426469627, 'recall': 0.8929809796330584, 'f1-score': 0.8937232929438362, 'support': 29705.0} |
| No log | 7.0 | 287 | 0.3034 | {'precision': 0.6413043478260869, 'recall': 0.6531365313653137, 'f1-score': 0.6471663619744058, 'support': 271.0} | {'precision': 0.9051724137931034, 'recall': 0.7553956834532374, 'f1-score': 0.8235294117647058, 'support': 139.0} | {'precision': 0.8725038402457758, 'recall': 0.8973143759873617, 'f1-score': 0.8847352024922119, 'support': 633.0} | {'precision': 0.6278366111951589, 'recall': 0.6223444138965258, 'f1-score': 0.6250784486004769, 'support': 4001.0} | {'precision': 0.9178082191780822, 'recall': 0.7655240933929458, 'f1-score': 0.8347778981581798, 'support': 2013.0} | {'precision': 0.8827297574308165, 'recall': 0.911697247706422, 'f1-score': 0.8969796910258636, 'support': 11336.0} | {'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0} | 0.8929 | {'precision': 0.8353364556670034, 'recall': 0.8007353058156956, 'f1-score': 0.816019199126301, 'support': 29705.0} | {'precision': 0.8931174594003277, 'recall': 0.89294731526679, 'f1-score': 0.8924389300080743, 'support': 29705.0} |
| No log | 8.0 | 328 | 0.3248 | {'precision': 0.6555555555555556, 'recall': 0.6531365313653137, 'f1-score': 0.6543438077634011, 'support': 271.0} | {'precision': 0.8450704225352113, 'recall': 0.8633093525179856, 'f1-score': 0.8540925266903915, 'support': 139.0} | {'precision': 0.884310618066561, 'recall': 0.8815165876777251, 'f1-score': 0.8829113924050632, 'support': 633.0} | {'precision': 0.6295746785361028, 'recall': 0.6363409147713072, 'f1-score': 0.6329397141081416, 'support': 4001.0} | {'precision': 0.8876518218623481, 'recall': 0.8713363139592648, 'f1-score': 0.8794184006016544, 'support': 2013.0} | {'precision': 0.8911930815390046, 'recall': 0.890878616796048, 'f1-score': 0.8910358214222692, 'support': 11336.0} | {'precision': 1.0, 'recall': 0.9998231966053748, 'f1-score': 0.9999115904871364, 'support': 11312.0} | 0.8943 | {'precision': 0.8276223111563975, 'recall': 0.8280487876704313, 'f1-score': 0.8278076076397225, 'support': 29705.0} | {'precision': 0.894638112913048, 'recall': 0.894260225551254, 'f1-score': 0.8944410693249414, 'support': 29705.0} |
| No log | 9.0 | 369 | 0.3427 | {'precision': 0.6554307116104869, 'recall': 0.6457564575645757, 'f1-score': 0.650557620817844, 'support': 271.0} | {'precision': 0.896, 'recall': 0.8057553956834532, 'f1-score': 0.8484848484848485, 'support': 139.0} | {'precision': 0.8725038402457758, 'recall': 0.8973143759873617, 'f1-score': 0.8847352024922119, 'support': 633.0} | {'precision': 0.6360742705570291, 'recall': 0.5993501624593851, 'f1-score': 0.6171663878522712, 'support': 4001.0} | {'precision': 0.907469342251951, 'recall': 0.8087431693989071, 'f1-score': 0.8552666141318623, 'support': 2013.0} | {'precision': 0.8780322307039864, 'recall': 0.913196894848271, 'f1-score': 0.8952693937559457, 'support': 11336.0} | {'precision': 1.0, 'recall': 0.9996463932107497, 'f1-score': 0.9998231653404068, 'support': 11312.0} | 0.8935 | {'precision': 0.8350729136241756, 'recall': 0.809966121307529, 'f1-score': 0.8216147475536272, 'support': 29705.0} | {'precision': 0.8918196587589381, 'recall': 0.893485945127083, 'f1-score': 0.8922398854856441, 'support': 29705.0} |
| No log | 10.0 | 410 | 0.3490 | {'precision': 0.6679841897233202, 'recall': 0.6236162361623616, 'f1-score': 0.6450381679389313, 'support': 271.0} | {'precision': 0.851063829787234, 'recall': 0.8633093525179856, 'f1-score': 0.8571428571428571, 'support': 139.0} | {'precision': 0.8736517719568567, 'recall': 0.8957345971563981, 'f1-score': 0.8845553822152886, 'support': 633.0} | {'precision': 0.6537938536850694, 'recall': 0.6008497875531117, 'f1-score': 0.6262047408179214, 'support': 4001.0} | {'precision': 0.9007751937984496, 'recall': 0.8658718330849479, 'f1-score': 0.8829787234042552, 'support': 2013.0} | {'precision': 0.8802487011327825, 'recall': 0.911697247706422, 'f1-score': 0.8956970143432854, 'support': 11336.0} | {'precision': 1.0, 'recall': 0.9997347949080623, 'f1-score': 0.9998673798682641, 'support': 11312.0} | 0.8971 | {'precision': 0.8325025057262447, 'recall': 0.8229734070127556, 'f1-score': 0.8273548951044003, 'support': 29705.0} | {'precision': 0.8945272549791482, 'recall': 0.8970543679515233, 'f1-score': 0.8955015738591867, 'support': 29705.0} |
### Framework versions
- Transformers 4.37.2
- Pytorch 2.2.0+cu121
- Datasets 2.17.0
- Tokenizers 0.15.2
|