README.md · NDugar/debertav3-mnli-snli-anli at 264b0adbb606a73484da7f3319465f4a8f6cd99e

metadata

language: en
tags:
  - deberta-v3
  - deberta-v2`
  - deberta-mnli
tasks: mnli
thumbnail: https://huggingface.co./front/thumbnails/microsoft.png
license: mit
pipeline_tag: zero-shot-classification

We present the dev results on SQuAD 1.1/2.0 and several GLUE benchmark tasks. | Model | SQuAD 1.1 | SQuAD 2.0 | MNLI-m/mm | SST-2 | QNLI | CoLA | RTE | MRPC | QQP |STS-B | |---------------------------|-----------|-----------|-------------|-------|------|------|--------|-------|-------|------| | | F1/EM | F1/EM | Acc | Acc | Acc | MCC | Acc |Acc/F1 |Acc/F1 |P/S | | BERT-Large | 90.9/84.1 | 81.8/79.0 | 86.6/- | 93.2 | 92.3 | 60.6 | 70.4 | 88.0/- | 91.3/- |90.0/- | | RoBERTa-Large | 94.6/88.9 | 89.4/86.5 | 90.2/- | 96.4 | 93.9 | 68.0 | 86.6 | 90.9/- | 92.2/- |92.4/- | | XLNet-Large | 95.1/89.7 | 90.6/87.9 | 90.8/- | 97.0 | 94.9 | 69.0 | 85.9 | 90.8/- | 92.3/- |92.5/- | | DeBERTa-Large¹ | 95.5/90.1 | 90.7/88.0 | 91.3/91.1| 96.5|95.3| 69.5| 91.0| 92.6/94.6| 92.3/- |92.8/92.5 | | DeBERTa-XLarge¹ | -/- | -/- | 91.5/91.2| 97.0 | - | - | 93.1 | 92.1/94.3 | - |92.9/92.7| | DeBERTa-V2-XLarge¹|95.8/90.8| 91.4/88.9|91.7/91.6| 97.5| 95.8|71.1|93.9|92.0/94.2|92.3/89.8|92.9/92.9| |DeBERTa-V2-XXLarge^1,2|96.1/91.4|92.2/89.7|91.7/91.9|97.2|96.0|72.0| 93.5| 93.1/94.9|92.7/90.3 |93.2/93.1 |

Notes.

¹ Following RoBERTa, for RTE, MRPC, STS-B, we fine-tune the tasks based on DeBERTa-Large-MNLI, DeBERTa-XLarge-MNLI, DeBERTa-V2-XLarge-MNLI, DeBERTa-V2-XXLarge-MNLI. The results of SST-2/QQP/QNLI/SQuADv2 will also be slightly improved when start from MNLI fine-tuned models, however, we only report the numbers fine-tuned from pretrained base models for those 4 tasks.
² To try the XXLarge model with HF transformers, we recommand using deepspeed as it's faster and saves memory.