File size: 1,575 Bytes
7d12537
 
 
08a7542
 
 
7d12537
 
08a7542
 
 
7d12537
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
---
datasets:
- squad
language:
- en
license: apache-2.0
metrics:
- squad
tags:
- question-answering
thumbnail: https://github.com/karanchahal/distiller/blob/master/distiller.jpg
---

# DistilBERT with a second step of distillation

## Model description

This model replicates the "DistilBERT (D)" model from Table 2 of the [DistilBERT paper](https://arxiv.org/pdf/1910.01108.pdf). In this approach, a DistilBERT student is fine-tuned on SQuAD v1.1, but with a BERT model (also fine-tuned on SQuAD v1.1) acting as a teacher for a second step of task-specific distillation.

In this version, the following pre-trained models were used:

* Student: `distilbert-base-uncased`
* Teacher: `lewtun/bert-base-uncased-finetuned-squad-v1`

## Training data

This model was trained on the SQuAD v1.1 dataset which can be obtained from the `datasets` library as follows:

```python
from datasets import load_dataset
squad = load_dataset('squad')
```

## Training procedure

## Eval results

|                  | Exact Match | F1   |
|------------------|-------------|------|
| DistilBERT paper | 79.1        | 86.9 |
| Ours             | 78.4        | 86.5 |

The scores were calculated using the `squad` metric from `datasets`.

### BibTeX entry and citation info

```bibtex
@misc{sanh2020distilbert,
      title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter}, 
      author={Victor Sanh and Lysandre Debut and Julien Chaumond and Thomas Wolf},
      year={2020},
      eprint={1910.01108},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```