prithivida commited on
Commit
8887186
·
verified ·
1 Parent(s): 0b94bf6

update all ref of chinese to arabic

Browse files
Files changed (1) hide show
  1. README.md +11 -11
README.md CHANGED
@@ -77,18 +77,18 @@ We will add more details here.
77
 
78
  <center>
79
  <img src="./ar_metrics_2.png" width=120%/>
80
- <b><p>Table 2: Detailed Chinese retrieval performance on the MIRACL dev set (measured by nDCG@10)</p></b>
81
 
82
  </center>
83
 
84
  Full set of evaluation numbers for our model
85
 
86
  ```python
87
- {'NDCG@1': 0.43511, 'NDCG@3': 0.42434, 'NDCG@5': 0.45298, 'NDCG@10': 0.50914, 'NDCG@100': 0.5815, 'NDCG@1000': 0.59392}
88
- {'MAP@1': 0.21342, 'MAP@3': 0.32967, 'MAP@5': 0.36798, 'MAP@10': 0.39908, 'MAP@100': 0.42592, 'MAP@1000': 0.42686}
89
- {'Recall@10': 0.63258, 'Recall@50': 0.85, 'Recall@100': 0.91595, 'Recall@200': 0.942, 'Recall@500': 0.96924, 'Recall@1000': 0.9857}
90
- {'P@1': 0.43511, 'P@3': 0.29177, 'P@5': 0.22545, 'P@10': 0.14758, 'P@100': 0.02252, 'P@1000': 0.00249}
91
- {'MRR@10': 0.55448, 'MRR@100': 0.56288, 'MRR@1000': 0.56294}
92
  ```
93
 
94
  <br/>
@@ -109,7 +109,7 @@ from sentence_transformers import SentenceTransformer
109
  import scipy.spatial
110
 
111
 
112
- model = SentenceTransformer('prithivida/miniMiracle_zh_v1')
113
 
114
  corpus = [
115
  '一个男人正在吃东西',
@@ -167,19 +167,19 @@ for query, query_embedding in zip(queries, query_embeddings):
167
 
168
  #### How do I offer hybrid search to improve accuracy ?
169
  MIRACL paper shows simply combining BM25 is a good starting point for a Hybrid option:
170
- The below numbers are with mDPR model, but miniMiracle_zh_v1 should give a even better hybrid performance.
171
 
172
  | Language | ISO | nDCG@10 BM25 | nDCG@10 mDPR | nDCG@10 Hybrid |
173
  |-----------|-----|--------------|--------------|----------------|
174
- | **Chinese** | **zh** | **0.175** | **0.512** | **0.526** |
175
 
176
- *Note: MIRACL paper shows a different (higher) value for BM25 Chinese, So we are taking that value from BGE-M3 paper, rest all are form the MIRACL paper.*
177
 
178
  #### cMTEB numbers:
179
  CMTEB is a general purpose embedding evaluation benchmark covering wide range of tasks, but like BGE-M3, miniMiracle models are predominantly tuned for retireval tasks aimed at search & IR based usecases.
180
  We ran the retrieval slice of the cMTEB.
181
 
182
- We compared the performance few top general purpose embedding models on the C-MTEB benchmark. please refer to the C-MTEB leaderboard. Almost all models below are bert-chinese based so they have no notion of any other languages.
183
 
184
  <center>
185
  <img src="./ar_metrics_3.png" width=150%/>
 
77
 
78
  <center>
79
  <img src="./ar_metrics_2.png" width=120%/>
80
+ <b><p>Table 2: Detailed Arabic retrieval performance on the MIRACL dev set (measured by nDCG@10)</p></b>
81
 
82
  </center>
83
 
84
  Full set of evaluation numbers for our model
85
 
86
  ```python
87
+ {'NDCG@1': 0.50449, 'NDCG@3': 0.52437, 'NDCG@5': 0.55649, 'NDCG@10': 0.60599, 'NDCG@100': 0.64745, 'NDCG@1000': 0.65717}
88
+ {'MAP@1': 0.34169, 'MAP@3': 0.45784, 'MAP@5': 0.48922, 'MAP@10': 0.51316, 'MAP@100': 0.53012, 'MAP@1000': 0.53069}
89
+ {'Recall@10': 0.72479, 'Recall@50': 0.87686, 'Recall@100': 0.91178, 'Recall@200': 0.93593, 'Recall@500': 0.96254, 'Recall@1000': 0.97557}
90
+ {'P@1': 0.50449, 'P@3': 0.29604, 'P@5': 0.21581, 'P@10': 0.13149, 'P@100': 0.01771, 'P@1000': 0.0019}
91
+ {'MRR@10': 0.61833, 'MRR@100': 0.62314, 'MRR@1000': 0.62329}
92
  ```
93
 
94
  <br/>
 
109
  import scipy.spatial
110
 
111
 
112
+ model = SentenceTransformer('prithivida/miniDense_arabic_v1')
113
 
114
  corpus = [
115
  '一个男人正在吃东西',
 
167
 
168
  #### How do I offer hybrid search to improve accuracy ?
169
  MIRACL paper shows simply combining BM25 is a good starting point for a Hybrid option:
170
+ The below numbers are with mDPR model, but miniDense_arabic_v1 should give a even better hybrid performance.
171
 
172
  | Language | ISO | nDCG@10 BM25 | nDCG@10 mDPR | nDCG@10 Hybrid |
173
  |-----------|-----|--------------|--------------|----------------|
174
+ | **Arabic** | **zh** | **0.175** | **0.512** | **0.526** |
175
 
176
+ *Note: MIRACL paper shows a different (higher) value for BM25 Arabic, So we are taking that value from BGE-M3 paper, rest all are form the MIRACL paper.*
177
 
178
  #### cMTEB numbers:
179
  CMTEB is a general purpose embedding evaluation benchmark covering wide range of tasks, but like BGE-M3, miniMiracle models are predominantly tuned for retireval tasks aimed at search & IR based usecases.
180
  We ran the retrieval slice of the cMTEB.
181
 
182
+ We compared the performance few top general purpose embedding models on the C-MTEB benchmark. please refer to the C-MTEB leaderboard. Almost all models below are bert-Arabic based so they have no notion of any other languages.
183
 
184
  <center>
185
  <img src="./ar_metrics_3.png" width=150%/>