chainyo commited on
Commit
1093102
1 Parent(s): 7f5f2f2

Update README.md

Browse files

Add Optimum + ONNX usage and English corrections

Files changed (1) hide show
  1. README.md +21 -5
README.md CHANGED
@@ -14,19 +14,21 @@ widget:
14
  DistilCamemBERT-QA
15
  ==================
16
 
17
- We present DistilCamemBERT-QA which is [DistilCamemBERT](https://huggingface.co/cmarkea/distilcamembert-base) fine-tuned for the Question-Answering task for the french language. This model is constructed over two datasets FQuAD v1.0 and Piaf which are composed of contexts and questions with their answers inside the context.
18
 
19
- This modelization is close to [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) based on [CamemBERT](https://huggingface.co/camembert-base) model. The problem of the modelizations based on CamemBERT is at the scaling moment, for the production phase for example. Indeed, inference cost can be a technological issue especially as in a context of cross-encoding like for this task. To counteract this effect, we propose this modelization which divides the inference time by 2 with the same consumption power thanks to DistilCamemBERT.
20
 
21
  Dataset
22
  -------
23
 
24
- The dataset is composed of FQuAD v1.0 and Piaf with 24'566 questions and answers for the training set and 3'188 for the evaluation set.
25
 
26
  Evaluation results and benchmark
27
  --------------------------------
28
 
29
- We compare [DistilCamemBERT-QA](https://huggingface.co/cmarkea/distilcamembert-base-qa) to two other modelizations working on french language. The first one [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) is based on well named [CamemBERT](https://huggingface.co/camembert-base), the french RoBERTa model and the second one [fmikaelian/flaubert-base-uncased-squad](https://huggingface.co/fmikaelian/flaubert-base-uncased-squad) is based on [FlauBERT](https://huggingface.co/flaubert/flaubert_base_uncased) an other french model based on BERT architecture this time. To compare the models to each others, the exact match comparing character by character the prediected answer and the ground truth is used, f1-score which measures the quality of intersection between predicted answer words and ground truth is also used and finally inclusion score which measures if the ground truth answer is include in predicted answer. For the mean inference time measure, an **AMD Ryzen 5 4500U @ 2.3GHz with 6 cores** was used.
 
 
30
 
31
  | **model** | **time (ms)** | **exact match (%)** | **f1-score (%)** | **inclusion-score (%)** |
32
  | :--------------: | :-----------: | :--------------: | :------------: | :------------: |
@@ -34,7 +36,7 @@ We compare [DistilCamemBERT-QA](https://huggingface.co/cmarkea/distilcamembert-b
34
  | [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 432.17 | **59.76** | **79.57** | **69.23** |
35
  | [fmikaelian/flaubert-base-uncased-squad](https://huggingface.co/fmikaelian/flaubert-base-uncased-squad) | 875.84 | 0.22 | 5.21 | 3.68 |
36
 
37
- Do not take into account the results on the FlauBERT model, there seems to be a problem with the modelling as the results seem very low.
38
 
39
  How to use DistilCamemBERT-QA
40
  ------------------------------
@@ -71,6 +73,20 @@ result
71
  'answer': ' réalisateur et producteur américain.'}
72
  ```
73
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
74
  Citation
75
  --------
76
  ```bibtex
 
14
  DistilCamemBERT-QA
15
  ==================
16
 
17
+ We present DistilCamemBERT-QA, which is [DistilCamemBERT](https://huggingface.co/cmarkea/distilcamembert-base) fine-tuned for the Question-Answering task for the french language. This model is built using two datasets, FQuAD v1.0 and Piaf, composed of contexts and questions with their answers inside the context.
18
 
19
+ This modelization is close to [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) based on [CamemBERT](https://huggingface.co/camembert-base) model. The problem of the modelizations based on CamemBERT is at the scaling moment, for the production phase, for example. Indeed, inference cost can be a technological issue, especially in cross-encoding like this task. To counteract this effect, we propose this modelization which divides the inference time by 2 with the same consumption power, thanks to DistilCamemBERT.
20
 
21
  Dataset
22
  -------
23
 
24
+ The dataset comprises FQuAD v1.0 and Piaf with 24'566 questions and answers for the training set and 3'188 for the evaluation set.
25
 
26
  Evaluation results and benchmark
27
  --------------------------------
28
 
29
+ We compare [DistilCamemBERT-QA](https://huggingface.co/cmarkea/distilcamembert-base-qa) to two other modelizations working on the french language. The first one [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) is based on well named [CamemBERT](https://huggingface.co/camembert-base), the french RoBERTa model and the second one [fmikaelian/flaubert-base-uncased-squad](https://huggingface.co/fmikaelian/flaubert-base-uncased-squad) is based on [FlauBERT](https://huggingface.co/flaubert/flaubert_base_uncased) another french model based on BERT architecture this time.
30
+
31
+ For our benchmarks, we compare matching character by character between the predicted answer and the ground truth. We also use the f1-score, which measures the intersection quality between predicted responses and ground truth. Finally, we use the inclusion score, which measures if the ground truth answer is included in the predicted answer. An **AMD Ryzen 5 4500U @ 2.3GHz with 6 cores** was used for the mean inference time measure.
32
 
33
  | **model** | **time (ms)** | **exact match (%)** | **f1-score (%)** | **inclusion-score (%)** |
34
  | :--------------: | :-----------: | :--------------: | :------------: | :------------: |
 
36
  | [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 432.17 | **59.76** | **79.57** | **69.23** |
37
  | [fmikaelian/flaubert-base-uncased-squad](https://huggingface.co/fmikaelian/flaubert-base-uncased-squad) | 875.84 | 0.22 | 5.21 | 3.68 |
38
 
39
+ Do not take into account the results of the FlauBERT model. The modeling seems to be a problem, as the results seem very low.
40
 
41
  How to use DistilCamemBERT-QA
42
  ------------------------------
 
73
  'answer': ' réalisateur et producteur américain.'}
74
  ```
75
 
76
+ ### Optimum + ONNX
77
+
78
+ ```python
79
+ from optimum.onnxruntime import ORTModelForQuestionAnswering
80
+ from transformers import AutoTokenizer, pipeline
81
+
82
+ tokenizer = AutoTokenizer.from_pretrained("cmarkea/distilcamembert-base-qa")
83
+ model = ORTModelForQuestionAnswering.from_pretrained("cmarkea/distilcamembert-base-qa")
84
+ onnx_qa = pipeline("question-answering", model=model, tokenizer=tokenizer)
85
+
86
+ # Quantized onnx model
87
+ quantized_model = ORTModelForQuestionAnswering.from_pretrained("cmarkea/distilcamembert-base-qa", file_name="model_quantized.onnx")
88
+ ```
89
+
90
  Citation
91
  --------
92
  ```bibtex