MANMEET75 commited on
Commit
a8c6ab8
1 Parent(s): e17cc08

Add new SentenceTransformer model.

Browse files
1_Pooling/config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "word_embedding_dimension": 768,
3
+ "pooling_mode_cls_token": false,
4
+ "pooling_mode_mean_tokens": true,
5
+ "pooling_mode_max_tokens": false,
6
+ "pooling_mode_mean_sqrt_len_tokens": false,
7
+ "pooling_mode_weightedmean_tokens": false,
8
+ "pooling_mode_lasttoken": false,
9
+ "include_prompt": true
10
+ }
README.md ADDED
@@ -0,0 +1,803 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: NeuML/pubmedbert-base-embeddings
3
+ datasets: []
4
+ language:
5
+ - en
6
+ library_name: sentence-transformers
7
+ license: apache-2.0
8
+ metrics:
9
+ - cosine_accuracy@1
10
+ - cosine_accuracy@3
11
+ - cosine_accuracy@5
12
+ - cosine_accuracy@10
13
+ - cosine_precision@1
14
+ - cosine_precision@3
15
+ - cosine_precision@5
16
+ - cosine_precision@10
17
+ - cosine_recall@1
18
+ - cosine_recall@3
19
+ - cosine_recall@5
20
+ - cosine_recall@10
21
+ - cosine_ndcg@10
22
+ - cosine_mrr@10
23
+ - cosine_map@100
24
+ pipeline_tag: sentence-similarity
25
+ tags:
26
+ - sentence-transformers
27
+ - sentence-similarity
28
+ - feature-extraction
29
+ - generated_from_trainer
30
+ - dataset_size:530
31
+ - loss:MatryoshkaLoss
32
+ - loss:MultipleNegativesRankingLoss
33
+ widget:
34
+ - source_sentence: If you receive a BharatPe speaker that you didn't order, please
35
+ contact BharatPe support immediately. They will assist in resolving the issue
36
+ and advise on the next steps.
37
+ sentences:
38
+ - Can I control multiple BharatPe speakers from one app?
39
+ - What to do if the BharatPe speaker's transaction announcements are intermittently
40
+ silent?
41
+ - What should I do if I receive a BharatPe speaker without ordering it?
42
+ - source_sentence: Remote control capabilities depend on the model of the BharatPe
43
+ speaker. Check if your model supports remote control through the BharatPe app
44
+ or a connected device.
45
+ sentences:
46
+ - How do I update my personal details in my Bharatpe account?
47
+ - What are the benefits of the BharatPe speaker?
48
+ - Can I control the BharatPe speaker remotely?
49
+ - source_sentence: If the announcements are not clear, check the speaker's volume
50
+ settings and ensure it's not placed near noisy equipment. If clarity doesn't improve,
51
+ the speaker may need servicing.
52
+ sentences:
53
+ - What to do if my BharatPe speaker is not syncing with the transaction history
54
+ in the app?
55
+ - What should I do if the speaker is not announcing payments clearly?
56
+ - The speaker doesn't produce any sound, what can be done?
57
+ - source_sentence: If the speaker is causing interference, try relocating it or other
58
+ devices to reduce the interference. Ensure there's a reasonable distance between
59
+ the speaker and other wireless equipment.
60
+ sentences:
61
+ - Can I use my Bharatpe device for international transactions?
62
+ - How do I know if my BharatPe speaker is under warranty?
63
+ - What should I do if the BharatPe speaker is causing interference with other wireless
64
+ devices?
65
+ - source_sentence: I can understand and respond in multiple Indian regional languages.
66
+ Feel free to communicate with me in the language you're most comfortable with.
67
+ sentences:
68
+ - How can I check if the BharatPe speaker is receiving a network signal?
69
+ - Bharti, can you provide tips for effective online communication?
70
+ - Bharti, what languages can you understand and respond to?
71
+ model-index:
72
+ - name: pubmedbert-base-embedding Chatbot Matryoshka
73
+ results:
74
+ - task:
75
+ type: information-retrieval
76
+ name: Information Retrieval
77
+ dataset:
78
+ name: dim 768
79
+ type: dim_768
80
+ metrics:
81
+ - type: cosine_accuracy@1
82
+ value: 0.7674418604651163
83
+ name: Cosine Accuracy@1
84
+ - type: cosine_accuracy@3
85
+ value: 0.9069767441860465
86
+ name: Cosine Accuracy@3
87
+ - type: cosine_accuracy@5
88
+ value: 0.9302325581395349
89
+ name: Cosine Accuracy@5
90
+ - type: cosine_accuracy@10
91
+ value: 0.9302325581395349
92
+ name: Cosine Accuracy@10
93
+ - type: cosine_precision@1
94
+ value: 0.7674418604651163
95
+ name: Cosine Precision@1
96
+ - type: cosine_precision@3
97
+ value: 0.3023255813953489
98
+ name: Cosine Precision@3
99
+ - type: cosine_precision@5
100
+ value: 0.18604651162790697
101
+ name: Cosine Precision@5
102
+ - type: cosine_precision@10
103
+ value: 0.09302325581395349
104
+ name: Cosine Precision@10
105
+ - type: cosine_recall@1
106
+ value: 0.7674418604651163
107
+ name: Cosine Recall@1
108
+ - type: cosine_recall@3
109
+ value: 0.9069767441860465
110
+ name: Cosine Recall@3
111
+ - type: cosine_recall@5
112
+ value: 0.9302325581395349
113
+ name: Cosine Recall@5
114
+ - type: cosine_recall@10
115
+ value: 0.9302325581395349
116
+ name: Cosine Recall@10
117
+ - type: cosine_ndcg@10
118
+ value: 0.8563596702043667
119
+ name: Cosine Ndcg@10
120
+ - type: cosine_mrr@10
121
+ value: 0.8313953488372093
122
+ name: Cosine Mrr@10
123
+ - type: cosine_map@100
124
+ value: 0.8349894291754757
125
+ name: Cosine Map@100
126
+ - task:
127
+ type: information-retrieval
128
+ name: Information Retrieval
129
+ dataset:
130
+ name: dim 512
131
+ type: dim_512
132
+ metrics:
133
+ - type: cosine_accuracy@1
134
+ value: 0.6976744186046512
135
+ name: Cosine Accuracy@1
136
+ - type: cosine_accuracy@3
137
+ value: 0.8837209302325582
138
+ name: Cosine Accuracy@3
139
+ - type: cosine_accuracy@5
140
+ value: 0.9302325581395349
141
+ name: Cosine Accuracy@5
142
+ - type: cosine_accuracy@10
143
+ value: 0.9302325581395349
144
+ name: Cosine Accuracy@10
145
+ - type: cosine_precision@1
146
+ value: 0.6976744186046512
147
+ name: Cosine Precision@1
148
+ - type: cosine_precision@3
149
+ value: 0.29457364341085274
150
+ name: Cosine Precision@3
151
+ - type: cosine_precision@5
152
+ value: 0.18604651162790697
153
+ name: Cosine Precision@5
154
+ - type: cosine_precision@10
155
+ value: 0.09302325581395349
156
+ name: Cosine Precision@10
157
+ - type: cosine_recall@1
158
+ value: 0.6976744186046512
159
+ name: Cosine Recall@1
160
+ - type: cosine_recall@3
161
+ value: 0.8837209302325582
162
+ name: Cosine Recall@3
163
+ - type: cosine_recall@5
164
+ value: 0.9302325581395349
165
+ name: Cosine Recall@5
166
+ - type: cosine_recall@10
167
+ value: 0.9302325581395349
168
+ name: Cosine Recall@10
169
+ - type: cosine_ndcg@10
170
+ value: 0.8320432881662091
171
+ name: Cosine Ndcg@10
172
+ - type: cosine_mrr@10
173
+ value: 0.7984496124031009
174
+ name: Cosine Mrr@10
175
+ - type: cosine_map@100
176
+ value: 0.8017447288993117
177
+ name: Cosine Map@100
178
+ - task:
179
+ type: information-retrieval
180
+ name: Information Retrieval
181
+ dataset:
182
+ name: dim 256
183
+ type: dim_256
184
+ metrics:
185
+ - type: cosine_accuracy@1
186
+ value: 0.7906976744186046
187
+ name: Cosine Accuracy@1
188
+ - type: cosine_accuracy@3
189
+ value: 0.8837209302325582
190
+ name: Cosine Accuracy@3
191
+ - type: cosine_accuracy@5
192
+ value: 0.9069767441860465
193
+ name: Cosine Accuracy@5
194
+ - type: cosine_accuracy@10
195
+ value: 0.9069767441860465
196
+ name: Cosine Accuracy@10
197
+ - type: cosine_precision@1
198
+ value: 0.7906976744186046
199
+ name: Cosine Precision@1
200
+ - type: cosine_precision@3
201
+ value: 0.29457364341085274
202
+ name: Cosine Precision@3
203
+ - type: cosine_precision@5
204
+ value: 0.1813953488372093
205
+ name: Cosine Precision@5
206
+ - type: cosine_precision@10
207
+ value: 0.09069767441860466
208
+ name: Cosine Precision@10
209
+ - type: cosine_recall@1
210
+ value: 0.7906976744186046
211
+ name: Cosine Recall@1
212
+ - type: cosine_recall@3
213
+ value: 0.8837209302325582
214
+ name: Cosine Recall@3
215
+ - type: cosine_recall@5
216
+ value: 0.9069767441860465
217
+ name: Cosine Recall@5
218
+ - type: cosine_recall@10
219
+ value: 0.9069767441860465
220
+ name: Cosine Recall@10
221
+ - type: cosine_ndcg@10
222
+ value: 0.8533147922143328
223
+ name: Cosine Ndcg@10
224
+ - type: cosine_mrr@10
225
+ value: 0.8352713178294573
226
+ name: Cosine Mrr@10
227
+ - type: cosine_map@100
228
+ value: 0.8392285023210497
229
+ name: Cosine Map@100
230
+ - task:
231
+ type: information-retrieval
232
+ name: Information Retrieval
233
+ dataset:
234
+ name: dim 128
235
+ type: dim_128
236
+ metrics:
237
+ - type: cosine_accuracy@1
238
+ value: 0.6744186046511628
239
+ name: Cosine Accuracy@1
240
+ - type: cosine_accuracy@3
241
+ value: 0.813953488372093
242
+ name: Cosine Accuracy@3
243
+ - type: cosine_accuracy@5
244
+ value: 0.8837209302325582
245
+ name: Cosine Accuracy@5
246
+ - type: cosine_accuracy@10
247
+ value: 0.9069767441860465
248
+ name: Cosine Accuracy@10
249
+ - type: cosine_precision@1
250
+ value: 0.6744186046511628
251
+ name: Cosine Precision@1
252
+ - type: cosine_precision@3
253
+ value: 0.2713178294573643
254
+ name: Cosine Precision@3
255
+ - type: cosine_precision@5
256
+ value: 0.17674418604651165
257
+ name: Cosine Precision@5
258
+ - type: cosine_precision@10
259
+ value: 0.09069767441860466
260
+ name: Cosine Precision@10
261
+ - type: cosine_recall@1
262
+ value: 0.6744186046511628
263
+ name: Cosine Recall@1
264
+ - type: cosine_recall@3
265
+ value: 0.813953488372093
266
+ name: Cosine Recall@3
267
+ - type: cosine_recall@5
268
+ value: 0.8837209302325582
269
+ name: Cosine Recall@5
270
+ - type: cosine_recall@10
271
+ value: 0.9069767441860465
272
+ name: Cosine Recall@10
273
+ - type: cosine_ndcg@10
274
+ value: 0.794152105183587
275
+ name: Cosine Ndcg@10
276
+ - type: cosine_mrr@10
277
+ value: 0.7575858250276855
278
+ name: Cosine Mrr@10
279
+ - type: cosine_map@100
280
+ value: 0.7600321150655651
281
+ name: Cosine Map@100
282
+ - task:
283
+ type: information-retrieval
284
+ name: Information Retrieval
285
+ dataset:
286
+ name: dim 64
287
+ type: dim_64
288
+ metrics:
289
+ - type: cosine_accuracy@1
290
+ value: 0.6046511627906976
291
+ name: Cosine Accuracy@1
292
+ - type: cosine_accuracy@3
293
+ value: 0.7441860465116279
294
+ name: Cosine Accuracy@3
295
+ - type: cosine_accuracy@5
296
+ value: 0.7906976744186046
297
+ name: Cosine Accuracy@5
298
+ - type: cosine_accuracy@10
299
+ value: 0.8604651162790697
300
+ name: Cosine Accuracy@10
301
+ - type: cosine_precision@1
302
+ value: 0.6046511627906976
303
+ name: Cosine Precision@1
304
+ - type: cosine_precision@3
305
+ value: 0.24806201550387597
306
+ name: Cosine Precision@3
307
+ - type: cosine_precision@5
308
+ value: 0.15813953488372093
309
+ name: Cosine Precision@5
310
+ - type: cosine_precision@10
311
+ value: 0.08604651162790698
312
+ name: Cosine Precision@10
313
+ - type: cosine_recall@1
314
+ value: 0.6046511627906976
315
+ name: Cosine Recall@1
316
+ - type: cosine_recall@3
317
+ value: 0.7441860465116279
318
+ name: Cosine Recall@3
319
+ - type: cosine_recall@5
320
+ value: 0.7906976744186046
321
+ name: Cosine Recall@5
322
+ - type: cosine_recall@10
323
+ value: 0.8604651162790697
324
+ name: Cosine Recall@10
325
+ - type: cosine_ndcg@10
326
+ value: 0.7220252449949186
327
+ name: Cosine Ndcg@10
328
+ - type: cosine_mrr@10
329
+ value: 0.6786083425618308
330
+ name: Cosine Mrr@10
331
+ - type: cosine_map@100
332
+ value: 0.6823125300680127
333
+ name: Cosine Map@100
334
+ ---
335
+
336
+ # pubmedbert-base-embedding Chatbot Matryoshka
337
+
338
+ This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [NeuML/pubmedbert-base-embeddings](https://huggingface.co/NeuML/pubmedbert-base-embeddings). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
339
+
340
+ ## Model Details
341
+
342
+ ### Model Description
343
+ - **Model Type:** Sentence Transformer
344
+ - **Base model:** [NeuML/pubmedbert-base-embeddings](https://huggingface.co/NeuML/pubmedbert-base-embeddings) <!-- at revision ba210f40b1b6d555d675c2d1ed6372e44570fc3c -->
345
+ - **Maximum Sequence Length:** 512 tokens
346
+ - **Output Dimensionality:** 768 tokens
347
+ - **Similarity Function:** Cosine Similarity
348
+ <!-- - **Training Dataset:** Unknown -->
349
+ - **Language:** en
350
+ - **License:** apache-2.0
351
+
352
+ ### Model Sources
353
+
354
+ - **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
355
+ - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
356
+ - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
357
+
358
+ ### Full Model Architecture
359
+
360
+ ```
361
+ SentenceTransformer(
362
+ (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
363
+ (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
364
+ )
365
+ ```
366
+
367
+ ## Usage
368
+
369
+ ### Direct Usage (Sentence Transformers)
370
+
371
+ First install the Sentence Transformers library:
372
+
373
+ ```bash
374
+ pip install -U sentence-transformers
375
+ ```
376
+
377
+ Then you can load this model and run inference.
378
+ ```python
379
+ from sentence_transformers import SentenceTransformer
380
+
381
+ # Download from the 🤗 Hub
382
+ model = SentenceTransformer("MANMEET75/pubmedbert-base-embedding-Chatbot-Matryoshk")
383
+ # Run inference
384
+ sentences = [
385
+ "I can understand and respond in multiple Indian regional languages. Feel free to communicate with me in the language you're most comfortable with.",
386
+ 'Bharti, what languages can you understand and respond to?',
387
+ 'Bharti, can you provide tips for effective online communication?',
388
+ ]
389
+ embeddings = model.encode(sentences)
390
+ print(embeddings.shape)
391
+ # [3, 768]
392
+
393
+ # Get the similarity scores for the embeddings
394
+ similarities = model.similarity(embeddings, embeddings)
395
+ print(similarities.shape)
396
+ # [3, 3]
397
+ ```
398
+
399
+ <!--
400
+ ### Direct Usage (Transformers)
401
+
402
+ <details><summary>Click to see the direct usage in Transformers</summary>
403
+
404
+ </details>
405
+ -->
406
+
407
+ <!--
408
+ ### Downstream Usage (Sentence Transformers)
409
+
410
+ You can finetune this model on your own dataset.
411
+
412
+ <details><summary>Click to expand</summary>
413
+
414
+ </details>
415
+ -->
416
+
417
+ <!--
418
+ ### Out-of-Scope Use
419
+
420
+ *List how the model may foreseeably be misused and address what users ought not to do with the model.*
421
+ -->
422
+
423
+ ## Evaluation
424
+
425
+ ### Metrics
426
+
427
+ #### Information Retrieval
428
+ * Dataset: `dim_768`
429
+ * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
430
+
431
+ | Metric | Value |
432
+ |:--------------------|:----------|
433
+ | cosine_accuracy@1 | 0.7674 |
434
+ | cosine_accuracy@3 | 0.907 |
435
+ | cosine_accuracy@5 | 0.9302 |
436
+ | cosine_accuracy@10 | 0.9302 |
437
+ | cosine_precision@1 | 0.7674 |
438
+ | cosine_precision@3 | 0.3023 |
439
+ | cosine_precision@5 | 0.186 |
440
+ | cosine_precision@10 | 0.093 |
441
+ | cosine_recall@1 | 0.7674 |
442
+ | cosine_recall@3 | 0.907 |
443
+ | cosine_recall@5 | 0.9302 |
444
+ | cosine_recall@10 | 0.9302 |
445
+ | cosine_ndcg@10 | 0.8564 |
446
+ | cosine_mrr@10 | 0.8314 |
447
+ | **cosine_map@100** | **0.835** |
448
+
449
+ #### Information Retrieval
450
+ * Dataset: `dim_512`
451
+ * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
452
+
453
+ | Metric | Value |
454
+ |:--------------------|:-----------|
455
+ | cosine_accuracy@1 | 0.6977 |
456
+ | cosine_accuracy@3 | 0.8837 |
457
+ | cosine_accuracy@5 | 0.9302 |
458
+ | cosine_accuracy@10 | 0.9302 |
459
+ | cosine_precision@1 | 0.6977 |
460
+ | cosine_precision@3 | 0.2946 |
461
+ | cosine_precision@5 | 0.186 |
462
+ | cosine_precision@10 | 0.093 |
463
+ | cosine_recall@1 | 0.6977 |
464
+ | cosine_recall@3 | 0.8837 |
465
+ | cosine_recall@5 | 0.9302 |
466
+ | cosine_recall@10 | 0.9302 |
467
+ | cosine_ndcg@10 | 0.832 |
468
+ | cosine_mrr@10 | 0.7984 |
469
+ | **cosine_map@100** | **0.8017** |
470
+
471
+ #### Information Retrieval
472
+ * Dataset: `dim_256`
473
+ * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
474
+
475
+ | Metric | Value |
476
+ |:--------------------|:-----------|
477
+ | cosine_accuracy@1 | 0.7907 |
478
+ | cosine_accuracy@3 | 0.8837 |
479
+ | cosine_accuracy@5 | 0.907 |
480
+ | cosine_accuracy@10 | 0.907 |
481
+ | cosine_precision@1 | 0.7907 |
482
+ | cosine_precision@3 | 0.2946 |
483
+ | cosine_precision@5 | 0.1814 |
484
+ | cosine_precision@10 | 0.0907 |
485
+ | cosine_recall@1 | 0.7907 |
486
+ | cosine_recall@3 | 0.8837 |
487
+ | cosine_recall@5 | 0.907 |
488
+ | cosine_recall@10 | 0.907 |
489
+ | cosine_ndcg@10 | 0.8533 |
490
+ | cosine_mrr@10 | 0.8353 |
491
+ | **cosine_map@100** | **0.8392** |
492
+
493
+ #### Information Retrieval
494
+ * Dataset: `dim_128`
495
+ * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
496
+
497
+ | Metric | Value |
498
+ |:--------------------|:---------|
499
+ | cosine_accuracy@1 | 0.6744 |
500
+ | cosine_accuracy@3 | 0.814 |
501
+ | cosine_accuracy@5 | 0.8837 |
502
+ | cosine_accuracy@10 | 0.907 |
503
+ | cosine_precision@1 | 0.6744 |
504
+ | cosine_precision@3 | 0.2713 |
505
+ | cosine_precision@5 | 0.1767 |
506
+ | cosine_precision@10 | 0.0907 |
507
+ | cosine_recall@1 | 0.6744 |
508
+ | cosine_recall@3 | 0.814 |
509
+ | cosine_recall@5 | 0.8837 |
510
+ | cosine_recall@10 | 0.907 |
511
+ | cosine_ndcg@10 | 0.7942 |
512
+ | cosine_mrr@10 | 0.7576 |
513
+ | **cosine_map@100** | **0.76** |
514
+
515
+ #### Information Retrieval
516
+ * Dataset: `dim_64`
517
+ * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
518
+
519
+ | Metric | Value |
520
+ |:--------------------|:-----------|
521
+ | cosine_accuracy@1 | 0.6047 |
522
+ | cosine_accuracy@3 | 0.7442 |
523
+ | cosine_accuracy@5 | 0.7907 |
524
+ | cosine_accuracy@10 | 0.8605 |
525
+ | cosine_precision@1 | 0.6047 |
526
+ | cosine_precision@3 | 0.2481 |
527
+ | cosine_precision@5 | 0.1581 |
528
+ | cosine_precision@10 | 0.086 |
529
+ | cosine_recall@1 | 0.6047 |
530
+ | cosine_recall@3 | 0.7442 |
531
+ | cosine_recall@5 | 0.7907 |
532
+ | cosine_recall@10 | 0.8605 |
533
+ | cosine_ndcg@10 | 0.722 |
534
+ | cosine_mrr@10 | 0.6786 |
535
+ | **cosine_map@100** | **0.6823** |
536
+
537
+ <!--
538
+ ## Bias, Risks and Limitations
539
+
540
+ *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
541
+ -->
542
+
543
+ <!--
544
+ ### Recommendations
545
+
546
+ *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
547
+ -->
548
+
549
+ ## Training Details
550
+
551
+ ### Training Dataset
552
+
553
+ #### Unnamed Dataset
554
+
555
+
556
+ * Size: 530 training samples
557
+ * Columns: <code>positive</code> and <code>anchor</code>
558
+ * Approximate statistics based on the first 1000 samples:
559
+ | | positive | anchor |
560
+ |:--------|:------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|
561
+ | type | string | string |
562
+ | details | <ul><li>min: 12 tokens</li><li>mean: 36.83 tokens</li><li>max: 107 tokens</li></ul> | <ul><li>min: 7 tokens</li><li>mean: 18.54 tokens</li><li>max: 30 tokens</li></ul> |
563
+ * Samples:
564
+ | positive | anchor |
565
+ |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------|
566
+ | <code>BharatPe Speaker comes with the following benefits: - Helps you avoid payment fraud - Lightweight & Easy installation process - Compatible with SIM & GPRS connectivity - Comes with a battery, no hassle of constant charging - Available in 10 Languages - Cashback Offers - Free replacement To Know more and place an order, tap below http://bharatpe.in/speaker.</code> | <code>What are the benefits of the BharatPe speaker?</code> |
567
+ | <code>BharatPe Speaker comes with the following benefits: - Helps you avoid payment fraud - Lightweight & Easy installation process - Compatible with SIM & GPRS connectivity - Comes with a battery, no hassle of constant charging - Available in 10 Languages - Cashback Offers - Free replacement To Know more and place an order, tap below http://bharatpe.in/speaker.</code> | <code>What advantages does the BharatPe speaker offer?</code> |
568
+ | <code>BharatPe Speaker comes with the following benefits: - Helps you avoid payment fraud - Lightweight & Easy installation process - Compatible with SIM & GPRS connectivity - Comes with a battery, no hassle of constant charging - Available in 10 Languages - Cashback Offers - Free replacement To Know more and place an order, tap below http://bharatpe.in/speaker.</code> | <code>Can you outline the benefits of using the BharatPe speaker?</code> |
569
+ * Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
570
+ ```json
571
+ {
572
+ "loss": "MultipleNegativesRankingLoss",
573
+ "matryoshka_dims": [
574
+ 768,
575
+ 512,
576
+ 256,
577
+ 128,
578
+ 64
579
+ ],
580
+ "matryoshka_weights": [
581
+ 1,
582
+ 1,
583
+ 1,
584
+ 1,
585
+ 1
586
+ ],
587
+ "n_dims_per_step": -1
588
+ }
589
+ ```
590
+
591
+ ### Training Hyperparameters
592
+ #### Non-Default Hyperparameters
593
+
594
+ - `eval_strategy`: epoch
595
+ - `per_device_train_batch_size`: 32
596
+ - `per_device_eval_batch_size`: 16
597
+ - `gradient_accumulation_steps`: 16
598
+ - `learning_rate`: 2e-05
599
+ - `num_train_epochs`: 10
600
+ - `lr_scheduler_type`: cosine
601
+ - `warmup_ratio`: 0.1
602
+ - `tf32`: False
603
+ - `load_best_model_at_end`: True
604
+ - `optim`: adamw_torch_fused
605
+ - `batch_sampler`: no_duplicates
606
+
607
+ #### All Hyperparameters
608
+ <details><summary>Click to expand</summary>
609
+
610
+ - `overwrite_output_dir`: False
611
+ - `do_predict`: False
612
+ - `eval_strategy`: epoch
613
+ - `prediction_loss_only`: True
614
+ - `per_device_train_batch_size`: 32
615
+ - `per_device_eval_batch_size`: 16
616
+ - `per_gpu_train_batch_size`: None
617
+ - `per_gpu_eval_batch_size`: None
618
+ - `gradient_accumulation_steps`: 16
619
+ - `eval_accumulation_steps`: None
620
+ - `learning_rate`: 2e-05
621
+ - `weight_decay`: 0.0
622
+ - `adam_beta1`: 0.9
623
+ - `adam_beta2`: 0.999
624
+ - `adam_epsilon`: 1e-08
625
+ - `max_grad_norm`: 1.0
626
+ - `num_train_epochs`: 10
627
+ - `max_steps`: -1
628
+ - `lr_scheduler_type`: cosine
629
+ - `lr_scheduler_kwargs`: {}
630
+ - `warmup_ratio`: 0.1
631
+ - `warmup_steps`: 0
632
+ - `log_level`: passive
633
+ - `log_level_replica`: warning
634
+ - `log_on_each_node`: True
635
+ - `logging_nan_inf_filter`: True
636
+ - `save_safetensors`: True
637
+ - `save_on_each_node`: False
638
+ - `save_only_model`: False
639
+ - `restore_callback_states_from_checkpoint`: False
640
+ - `no_cuda`: False
641
+ - `use_cpu`: False
642
+ - `use_mps_device`: False
643
+ - `seed`: 42
644
+ - `data_seed`: None
645
+ - `jit_mode_eval`: False
646
+ - `use_ipex`: False
647
+ - `bf16`: False
648
+ - `fp16`: False
649
+ - `fp16_opt_level`: O1
650
+ - `half_precision_backend`: auto
651
+ - `bf16_full_eval`: False
652
+ - `fp16_full_eval`: False
653
+ - `tf32`: False
654
+ - `local_rank`: 0
655
+ - `ddp_backend`: None
656
+ - `tpu_num_cores`: None
657
+ - `tpu_metrics_debug`: False
658
+ - `debug`: []
659
+ - `dataloader_drop_last`: False
660
+ - `dataloader_num_workers`: 0
661
+ - `dataloader_prefetch_factor`: None
662
+ - `past_index`: -1
663
+ - `disable_tqdm`: False
664
+ - `remove_unused_columns`: True
665
+ - `label_names`: None
666
+ - `load_best_model_at_end`: True
667
+ - `ignore_data_skip`: False
668
+ - `fsdp`: []
669
+ - `fsdp_min_num_params`: 0
670
+ - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
671
+ - `fsdp_transformer_layer_cls_to_wrap`: None
672
+ - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
673
+ - `deepspeed`: None
674
+ - `label_smoothing_factor`: 0.0
675
+ - `optim`: adamw_torch_fused
676
+ - `optim_args`: None
677
+ - `adafactor`: False
678
+ - `group_by_length`: False
679
+ - `length_column_name`: length
680
+ - `ddp_find_unused_parameters`: None
681
+ - `ddp_bucket_cap_mb`: None
682
+ - `ddp_broadcast_buffers`: False
683
+ - `dataloader_pin_memory`: True
684
+ - `dataloader_persistent_workers`: False
685
+ - `skip_memory_metrics`: True
686
+ - `use_legacy_prediction_loop`: False
687
+ - `push_to_hub`: False
688
+ - `resume_from_checkpoint`: None
689
+ - `hub_model_id`: None
690
+ - `hub_strategy`: every_save
691
+ - `hub_private_repo`: False
692
+ - `hub_always_push`: False
693
+ - `gradient_checkpointing`: False
694
+ - `gradient_checkpointing_kwargs`: None
695
+ - `include_inputs_for_metrics`: False
696
+ - `eval_do_concat_batches`: True
697
+ - `fp16_backend`: auto
698
+ - `push_to_hub_model_id`: None
699
+ - `push_to_hub_organization`: None
700
+ - `mp_parameters`:
701
+ - `auto_find_batch_size`: False
702
+ - `full_determinism`: False
703
+ - `torchdynamo`: None
704
+ - `ray_scope`: last
705
+ - `ddp_timeout`: 1800
706
+ - `torch_compile`: False
707
+ - `torch_compile_backend`: None
708
+ - `torch_compile_mode`: None
709
+ - `dispatch_batches`: None
710
+ - `split_batches`: None
711
+ - `include_tokens_per_second`: False
712
+ - `include_num_input_tokens_seen`: False
713
+ - `neftune_noise_alpha`: None
714
+ - `optim_target_modules`: None
715
+ - `batch_eval_metrics`: False
716
+ - `batch_sampler`: no_duplicates
717
+ - `multi_dataset_batch_sampler`: proportional
718
+
719
+ </details>
720
+
721
+ ### Training Logs
722
+ | Epoch | Step | Training Loss | dim_128_cosine_map@100 | dim_256_cosine_map@100 | dim_512_cosine_map@100 | dim_64_cosine_map@100 | dim_768_cosine_map@100 |
723
+ |:----------:|:-----:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|:----------------------:|
724
+ | 0.9412 | 1 | - | 0.4829 | 0.5338 | 0.5921 | 0.3235 | 0.6100 |
725
+ | 1.8824 | 2 | - | 0.5767 | 0.6175 | 0.6588 | 0.4176 | 0.6793 |
726
+ | 2.8235 | 3 | - | 0.6337 | 0.6776 | 0.6979 | 0.5083 | 0.7263 |
727
+ | 3.7647 | 4 | - | 0.6588 | 0.7257 | 0.7297 | 0.5840 | 0.7612 |
728
+ | 4.7059 | 5 | - | 0.7049 | 0.7766 | 0.7643 | 0.6151 | 0.7902 |
729
+ | 5.6471 | 6 | - | 0.7374 | 0.8257 | 0.7890 | 0.6519 | 0.7956 |
730
+ | 6.5882 | 7 | - | 0.7573 | 0.8261 | 0.7912 | 0.6689 | 0.7978 |
731
+ | 7.5294 | 8 | - | 0.7590 | 0.8275 | 0.7958 | 0.6811 | 0.8233 |
732
+ | **8.4706** | **9** | **-** | **0.76** | **0.8392** | **0.7998** | **0.6823** | **0.8234** |
733
+ | 9.4118 | 10 | 4.944 | 0.7600 | 0.8392 | 0.8017 | 0.6823 | 0.8350 |
734
+
735
+ * The bold row denotes the saved checkpoint.
736
+
737
+ ### Framework Versions
738
+ - Python: 3.10.12
739
+ - Sentence Transformers: 3.0.1
740
+ - Transformers: 4.41.2
741
+ - PyTorch: 2.1.2+cu121
742
+ - Accelerate: 0.32.1
743
+ - Datasets: 2.19.1
744
+ - Tokenizers: 0.19.1
745
+
746
+ ## Citation
747
+
748
+ ### BibTeX
749
+
750
+ #### Sentence Transformers
751
+ ```bibtex
752
+ @inproceedings{reimers-2019-sentence-bert,
753
+ title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
754
+ author = "Reimers, Nils and Gurevych, Iryna",
755
+ booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
756
+ month = "11",
757
+ year = "2019",
758
+ publisher = "Association for Computational Linguistics",
759
+ url = "https://arxiv.org/abs/1908.10084",
760
+ }
761
+ ```
762
+
763
+ #### MatryoshkaLoss
764
+ ```bibtex
765
+ @misc{kusupati2024matryoshka,
766
+ title={Matryoshka Representation Learning},
767
+ author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
768
+ year={2024},
769
+ eprint={2205.13147},
770
+ archivePrefix={arXiv},
771
+ primaryClass={cs.LG}
772
+ }
773
+ ```
774
+
775
+ #### MultipleNegativesRankingLoss
776
+ ```bibtex
777
+ @misc{henderson2017efficient,
778
+ title={Efficient Natural Language Response Suggestion for Smart Reply},
779
+ author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
780
+ year={2017},
781
+ eprint={1705.00652},
782
+ archivePrefix={arXiv},
783
+ primaryClass={cs.CL}
784
+ }
785
+ ```
786
+
787
+ <!--
788
+ ## Glossary
789
+
790
+ *Clearly define terms in order to be accessible across audiences.*
791
+ -->
792
+
793
+ <!--
794
+ ## Model Card Authors
795
+
796
+ *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
797
+ -->
798
+
799
+ <!--
800
+ ## Model Card Contact
801
+
802
+ *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
803
+ -->
config.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "NeuML/pubmedbert-base-embeddings",
3
+ "architectures": [
4
+ "BertModel"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "classifier_dropout": null,
8
+ "hidden_act": "gelu",
9
+ "hidden_dropout_prob": 0.1,
10
+ "hidden_size": 768,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 3072,
13
+ "layer_norm_eps": 1e-12,
14
+ "max_position_embeddings": 512,
15
+ "model_type": "bert",
16
+ "num_attention_heads": 12,
17
+ "num_hidden_layers": 12,
18
+ "pad_token_id": 0,
19
+ "position_embedding_type": "absolute",
20
+ "torch_dtype": "float32",
21
+ "transformers_version": "4.41.2",
22
+ "type_vocab_size": 2,
23
+ "use_cache": true,
24
+ "vocab_size": 30522
25
+ }
config_sentence_transformers.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "__version__": {
3
+ "sentence_transformers": "3.0.1",
4
+ "transformers": "4.41.2",
5
+ "pytorch": "2.1.2+cu121"
6
+ },
7
+ "prompts": {},
8
+ "default_prompt_name": null,
9
+ "similarity_fn_name": null
10
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dc564f9c865eac75fa610620772ae7b096b7a4f4b8e0937c1456227f12bc39a5
3
+ size 437951328
modules.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "idx": 0,
4
+ "name": "0",
5
+ "path": "",
6
+ "type": "sentence_transformers.models.Transformer"
7
+ },
8
+ {
9
+ "idx": 1,
10
+ "name": "1",
11
+ "path": "1_Pooling",
12
+ "type": "sentence_transformers.models.Pooling"
13
+ }
14
+ ]
sentence_bert_config.json ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {
2
+ "max_seq_length": 512,
3
+ "do_lower_case": false
4
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": {
3
+ "content": "[CLS]",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "mask_token": {
10
+ "content": "[MASK]",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "[PAD]",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "sep_token": {
24
+ "content": "[SEP]",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "unk_token": {
31
+ "content": "[UNK]",
32
+ "lstrip": false,
33
+ "normalized": false,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ }
37
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "[PAD]",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "[UNK]",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "[CLS]",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "3": {
28
+ "content": "[SEP]",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "4": {
36
+ "content": "[MASK]",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "additional_special_tokens": [],
45
+ "clean_up_tokenization_spaces": true,
46
+ "cls_token": "[CLS]",
47
+ "do_basic_tokenize": true,
48
+ "do_lower_case": true,
49
+ "mask_token": "[MASK]",
50
+ "max_length": 512,
51
+ "model_max_length": 512,
52
+ "never_split": null,
53
+ "pad_to_multiple_of": null,
54
+ "pad_token": "[PAD]",
55
+ "pad_token_type_id": 0,
56
+ "padding_side": "right",
57
+ "sep_token": "[SEP]",
58
+ "stride": 0,
59
+ "strip_accents": null,
60
+ "tokenize_chinese_chars": true,
61
+ "tokenizer_class": "BertTokenizer",
62
+ "truncation_side": "right",
63
+ "truncation_strategy": "longest_first",
64
+ "unk_token": "[UNK]"
65
+ }
vocab.txt ADDED
The diff for this file is too large to render. See raw diff