update all ref of chinese to arabic
Browse files
README.md
CHANGED
@@ -77,18 +77,18 @@ We will add more details here.
|
|
77 |
|
78 |
<center>
|
79 |
<img src="./ar_metrics_2.png" width=120%/>
|
80 |
-
<b><p>Table 2: Detailed
|
81 |
|
82 |
</center>
|
83 |
|
84 |
Full set of evaluation numbers for our model
|
85 |
|
86 |
```python
|
87 |
-
{'NDCG@1': 0.
|
88 |
-
{'MAP@1': 0.
|
89 |
-
{'Recall@10': 0.
|
90 |
-
{'P@1': 0.
|
91 |
-
{'MRR@10': 0.
|
92 |
```
|
93 |
|
94 |
<br/>
|
@@ -109,7 +109,7 @@ from sentence_transformers import SentenceTransformer
|
|
109 |
import scipy.spatial
|
110 |
|
111 |
|
112 |
-
model = SentenceTransformer('prithivida/
|
113 |
|
114 |
corpus = [
|
115 |
'一个男人正在吃东西',
|
@@ -167,19 +167,19 @@ for query, query_embedding in zip(queries, query_embeddings):
|
|
167 |
|
168 |
#### How do I offer hybrid search to improve accuracy ?
|
169 |
MIRACL paper shows simply combining BM25 is a good starting point for a Hybrid option:
|
170 |
-
The below numbers are with mDPR model, but
|
171 |
|
172 |
| Language | ISO | nDCG@10 BM25 | nDCG@10 mDPR | nDCG@10 Hybrid |
|
173 |
|-----------|-----|--------------|--------------|----------------|
|
174 |
-
| **
|
175 |
|
176 |
-
*Note: MIRACL paper shows a different (higher) value for BM25
|
177 |
|
178 |
#### cMTEB numbers:
|
179 |
CMTEB is a general purpose embedding evaluation benchmark covering wide range of tasks, but like BGE-M3, miniMiracle models are predominantly tuned for retireval tasks aimed at search & IR based usecases.
|
180 |
We ran the retrieval slice of the cMTEB.
|
181 |
|
182 |
-
We compared the performance few top general purpose embedding models on the C-MTEB benchmark. please refer to the C-MTEB leaderboard. Almost all models below are bert-
|
183 |
|
184 |
<center>
|
185 |
<img src="./ar_metrics_3.png" width=150%/>
|
|
|
77 |
|
78 |
<center>
|
79 |
<img src="./ar_metrics_2.png" width=120%/>
|
80 |
+
<b><p>Table 2: Detailed Arabic retrieval performance on the MIRACL dev set (measured by nDCG@10)</p></b>
|
81 |
|
82 |
</center>
|
83 |
|
84 |
Full set of evaluation numbers for our model
|
85 |
|
86 |
```python
|
87 |
+
{'NDCG@1': 0.50449, 'NDCG@3': 0.52437, 'NDCG@5': 0.55649, 'NDCG@10': 0.60599, 'NDCG@100': 0.64745, 'NDCG@1000': 0.65717}
|
88 |
+
{'MAP@1': 0.34169, 'MAP@3': 0.45784, 'MAP@5': 0.48922, 'MAP@10': 0.51316, 'MAP@100': 0.53012, 'MAP@1000': 0.53069}
|
89 |
+
{'Recall@10': 0.72479, 'Recall@50': 0.87686, 'Recall@100': 0.91178, 'Recall@200': 0.93593, 'Recall@500': 0.96254, 'Recall@1000': 0.97557}
|
90 |
+
{'P@1': 0.50449, 'P@3': 0.29604, 'P@5': 0.21581, 'P@10': 0.13149, 'P@100': 0.01771, 'P@1000': 0.0019}
|
91 |
+
{'MRR@10': 0.61833, 'MRR@100': 0.62314, 'MRR@1000': 0.62329}
|
92 |
```
|
93 |
|
94 |
<br/>
|
|
|
109 |
import scipy.spatial
|
110 |
|
111 |
|
112 |
+
model = SentenceTransformer('prithivida/miniDense_arabic_v1')
|
113 |
|
114 |
corpus = [
|
115 |
'一个男人正在吃东西',
|
|
|
167 |
|
168 |
#### How do I offer hybrid search to improve accuracy ?
|
169 |
MIRACL paper shows simply combining BM25 is a good starting point for a Hybrid option:
|
170 |
+
The below numbers are with mDPR model, but miniDense_arabic_v1 should give a even better hybrid performance.
|
171 |
|
172 |
| Language | ISO | nDCG@10 BM25 | nDCG@10 mDPR | nDCG@10 Hybrid |
|
173 |
|-----------|-----|--------------|--------------|----------------|
|
174 |
+
| **Arabic** | **zh** | **0.175** | **0.512** | **0.526** |
|
175 |
|
176 |
+
*Note: MIRACL paper shows a different (higher) value for BM25 Arabic, So we are taking that value from BGE-M3 paper, rest all are form the MIRACL paper.*
|
177 |
|
178 |
#### cMTEB numbers:
|
179 |
CMTEB is a general purpose embedding evaluation benchmark covering wide range of tasks, but like BGE-M3, miniMiracle models are predominantly tuned for retireval tasks aimed at search & IR based usecases.
|
180 |
We ran the retrieval slice of the cMTEB.
|
181 |
|
182 |
+
We compared the performance few top general purpose embedding models on the C-MTEB benchmark. please refer to the C-MTEB leaderboard. Almost all models below are bert-Arabic based so they have no notion of any other languages.
|
183 |
|
184 |
<center>
|
185 |
<img src="./ar_metrics_3.png" width=150%/>
|