Hugging Face's logo

pavanmantha
/
bge-base-en-bioembed768

Sentence Similarity
sentence-transformers
Safetensors
English
bert
feature-extraction
Generated from Trainer
dataset_size:4247
loss:MatryoshkaLoss
loss:MultipleNegativesRankingLoss
Eval Results
text-embeddings-inference
Inference Endpoints
Model card Files Community
bge-base-en-bioembed768 / README.md
pavanmantha's picture
pavanmantha
Update README.md
de32358 verified
preview code
|
raw
history blame
71.1 kB
metadata
language:
  - en
license: apache-2.0
library_name: sentence-transformers
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:4247
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
model-index:
  - name: pavanmantha/bge-base-en-bioembed768
    results:
      - dataset:
          config: default
          name: MTEB MedicalQARetrieval (default)
          revision: ae763399273d8b20506b80cf6f6f9a31a6a2b238
          split: test
          type: mteb/medical_qa
        metrics:
          - type: main_score
            value: 68.65599999999999
          - type: map_at_1
            value: 51.025
          - type: map_at_10
            value: 62.946000000000005
          - type: map_at_100
            value: 63.392
          - type: map_at_1000
            value: 63.408
          - type: map_at_20
            value: 63.254
          - type: map_at_3
            value: 60.221000000000004
          - type: map_at_5
            value: 62.022999999999996
          - type: mrr_at_1
            value: 51.025390625
          - type: mrr_at_10
            value: 62.94629293774791
          - type: mrr_at_100
            value: 63.391942840845125
          - type: mrr_at_1000
            value: 63.40789553144974
          - type: mrr_at_20
            value: 63.25463963215003
          - type: mrr_at_3
            value: 60.22135416666657
          - type: mrr_at_5
            value: 62.02311197916664
          - type: nauc_map_at_1000_diff1
            value: 44.0289077012448
          - type: nauc_map_at_1000_max
            value: 40.196044082980045
          - type: nauc_map_at_1000_std
            value: -13.087939597961418
          - type: nauc_map_at_100_diff1
            value: 44.02100215716583
          - type: nauc_map_at_100_max
            value: 40.20884959562626
          - type: nauc_map_at_100_std
            value: -13.074614347446076
          - type: nauc_map_at_10_diff1
            value: 43.78028009647677
          - type: nauc_map_at_10_max
            value: 40.01480272155306
          - type: nauc_map_at_10_std
            value: -13.377550749542138
          - type: nauc_map_at_1_diff1
            value: 49.57988513879831
          - type: nauc_map_at_1_max
            value: 40.609452025451915
          - type: nauc_map_at_1_std
            value: -11.541103718714977
          - type: nauc_map_at_20_diff1
            value: 43.87022043596322
          - type: nauc_map_at_20_max
            value: 40.22321319847734
          - type: nauc_map_at_20_std
            value: -13.164262713651956
          - type: nauc_map_at_3_diff1
            value: 44.38171551545852
          - type: nauc_map_at_3_max
            value: 39.6459216129053
          - type: nauc_map_at_3_std
            value: -13.770104903701466
          - type: nauc_map_at_5_diff1
            value: 43.8534318488618
          - type: nauc_map_at_5_max
            value: 39.68004513144695
          - type: nauc_map_at_5_std
            value: -13.877556205848146
          - type: nauc_mrr_at_1000_diff1
            value: 44.028920021140976
          - type: nauc_mrr_at_1000_max
            value: 40.23569448794487
          - type: nauc_mrr_at_1000_std
            value: -12.956236343591568
          - type: nauc_mrr_at_100_diff1
            value: 44.02100738083268
          - type: nauc_mrr_at_100_max
            value: 40.24846931727988
          - type: nauc_mrr_at_100_std
            value: -12.942990526259535
          - type: nauc_mrr_at_10_diff1
            value: 43.78028009647677
          - type: nauc_mrr_at_10_max
            value: 40.05165903518751
          - type: nauc_mrr_at_10_std
            value: -13.249179890670767
          - type: nauc_mrr_at_1_diff1
            value: 49.57988513879831
          - type: nauc_mrr_at_1_max
            value: 40.64858186876722
          - type: nauc_mrr_at_1_std
            value: -11.320856258088295
          - type: nauc_mrr_at_20_diff1
            value: 43.87022486033709
          - type: nauc_mrr_at_20_max
            value: 40.26263859989682
          - type: nauc_mrr_at_20_std
            value: -13.033285163966218
          - type: nauc_mrr_at_3_diff1
            value: 44.38171551545852
          - type: nauc_mrr_at_3_max
            value: 39.69840946271301
          - type: nauc_mrr_at_3_std
            value: -13.633807447557226
          - type: nauc_mrr_at_5_diff1
            value: 43.8534318488618
          - type: nauc_mrr_at_5_max
            value: 39.72797855901197
          - type: nauc_mrr_at_5_std
            value: -13.747197261890934
          - type: nauc_ndcg_at_1000_diff1
            value: 42.6428514635724
          - type: nauc_ndcg_at_1000_max
            value: 40.48576100703463
          - type: nauc_ndcg_at_1000_std
            value: -12.705673993292077
          - type: nauc_ndcg_at_100_diff1
            value: 42.17588553015554
          - type: nauc_ndcg_at_100_max
            value: 40.80768178826183
          - type: nauc_ndcg_at_100_std
            value: -12.05166428600645
          - type: nauc_ndcg_at_10_diff1
            value: 40.65597689427392
          - type: nauc_ndcg_at_10_max
            value: 40.175863350340215
          - type: nauc_ndcg_at_10_std
            value: -13.557398725708799
          - type: nauc_ndcg_at_1_diff1
            value: 49.57988513879831
          - type: nauc_ndcg_at_1_max
            value: 40.609452025451915
          - type: nauc_ndcg_at_1_std
            value: -11.541103718714977
          - type: nauc_ndcg_at_20_diff1
            value: 40.913600886170634
          - type: nauc_ndcg_at_20_max
            value: 41.00724137368964
          - type: nauc_ndcg_at_20_std
            value: -12.593947509615822
          - type: nauc_ndcg_at_3_diff1
            value: 42.40019474173081
          - type: nauc_ndcg_at_3_max
            value: 39.2565483832548
          - type: nauc_ndcg_at_3_std
            value: -14.612074432976202
          - type: nauc_ndcg_at_5_diff1
            value: 41.15688939938367
          - type: nauc_ndcg_at_5_max
            value: 39.35405468594706
          - type: nauc_ndcg_at_5_std
            value: -14.899193680376587
          - type: nauc_precision_at_1000_diff1
            value: 29.214715925497835
          - type: nauc_precision_at_1000_max
            value: 66.23687386181616
          - type: nauc_precision_at_1000_std
            value: 20.38998945371634
          - type: nauc_precision_at_100_diff1
            value: 16.184818683990663
          - type: nauc_precision_at_100_max
            value: 57.004633019507764
          - type: nauc_precision_at_100_std
            value: 24.749447988511026
          - type: nauc_precision_at_10_diff1
            value: 20.14231168000379
          - type: nauc_precision_at_10_max
            value: 41.81505494607868
          - type: nauc_precision_at_10_std
            value: -13.588327036007039
          - type: nauc_precision_at_1_diff1
            value: 49.57988513879831
          - type: nauc_precision_at_1_max
            value: 40.609452025451915
          - type: nauc_precision_at_1_std
            value: -11.541103718714977
          - type: nauc_precision_at_20_diff1
            value: 13.936251711556528
          - type: nauc_precision_at_20_max
            value: 50.63499099834573
          - type: nauc_precision_at_20_std
            value: -3.4568556682579055
          - type: nauc_precision_at_3_diff1
            value: 35.342792247725725
          - type: nauc_precision_at_3_max
            value: 37.86162276112827
          - type: nauc_precision_at_3_std
            value: -17.607999329247022
          - type: nauc_precision_at_5_diff1
            value: 28.95330037767198
          - type: nauc_precision_at_5_max
            value: 37.98848038772262
          - type: nauc_precision_at_5_std
            value: -19.46990621093604
          - type: nauc_recall_at_1000_diff1
            value: 29.214715925499096
          - type: nauc_recall_at_1000_max
            value: 66.23687386181345
          - type: nauc_recall_at_1000_std
            value: 20.389989453717625
          - type: nauc_recall_at_100_diff1
            value: 16.184818683991136
          - type: nauc_recall_at_100_max
            value: 57.004633019507104
          - type: nauc_recall_at_100_std
            value: 24.74944798851128
          - type: nauc_recall_at_10_diff1
            value: 20.14231168000389
          - type: nauc_recall_at_10_max
            value: 41.81505494607885
          - type: nauc_recall_at_10_std
            value: -13.588327036006865
          - type: nauc_recall_at_1_diff1
            value: 49.57988513879831
          - type: nauc_recall_at_1_max
            value: 40.609452025451915
          - type: nauc_recall_at_1_std
            value: -11.541103718714977
          - type: nauc_recall_at_20_diff1
            value: 13.936251711556865
          - type: nauc_recall_at_20_max
            value: 50.63499099834594
          - type: nauc_recall_at_20_std
            value: -3.4568556682576865
          - type: nauc_recall_at_3_diff1
            value: 35.34279224772573
          - type: nauc_recall_at_3_max
            value: 37.86162276112829
          - type: nauc_recall_at_3_std
            value: -17.607999329247097
          - type: nauc_recall_at_5_diff1
            value: 28.953300377672097
          - type: nauc_recall_at_5_max
            value: 37.98848038772265
          - type: nauc_recall_at_5_std
            value: -19.469906210936013
          - type: ndcg_at_1
            value: 51.025
          - type: ndcg_at_10
            value: 68.65599999999999
          - type: ndcg_at_100
            value: 70.78
          - type: ndcg_at_1000
            value: 71.207
          - type: ndcg_at_20
            value: 69.771
          - type: ndcg_at_3
            value: 63.202999999999996
          - type: ndcg_at_5
            value: 66.425
          - type: precision_at_1
            value: 51.025
          - type: precision_at_10
            value: 8.647
          - type: precision_at_100
            value: 0.963
          - type: precision_at_1000
            value: 0.1
          - type: precision_at_20
            value: 4.543
          - type: precision_at_3
            value: 23.942
          - type: precision_at_5
            value: 15.918
          - type: recall_at_1
            value: 51.025
          - type: recall_at_10
            value: 86.47500000000001
          - type: recall_at_100
            value: 96.33800000000001
          - type: recall_at_1000
            value: 99.70700000000001
          - type: recall_at_20
            value: 90.869
          - type: recall_at_3
            value: 71.82600000000001
          - type: recall_at_5
            value: 79.59
        task:
          type: Retrieval
      - dataset:
          config: default
          name: MTEB TRECCOVID (default)
          revision: bb9466bac8153a0349341eb1b22e06409e78ef4e
          split: test
          type: mteb/trec-covid
        metrics:
          - type: main_score
            value: 67.33
          - type: map_at_1
            value: 0.209
          - type: map_at_10
            value: 1.694
          - type: map_at_100
            value: 9.078999999999999
          - type: map_at_1000
            value: 21.590999999999998
          - type: map_at_20
            value: 2.973
          - type: map_at_3
            value: 0.594
          - type: map_at_5
            value: 0.9259999999999999
          - type: mrr_at_1
            value: 76
          - type: mrr_at_10
            value: 86.66666666666666
          - type: mrr_at_100
            value: 86.66666666666666
          - type: mrr_at_1000
            value: 86.66666666666666
          - type: mrr_at_20
            value: 86.66666666666666
          - type: mrr_at_3
            value: 86.33333333333334
          - type: mrr_at_5
            value: 86.33333333333334
          - type: nauc_map_at_1000_diff1
            value: -39.36100175513192
          - type: nauc_map_at_1000_max
            value: 51.929525559618305
          - type: nauc_map_at_1000_std
            value: 88.1016909960378
          - type: nauc_map_at_100_diff1
            value: -34.61735014934496
          - type: nauc_map_at_100_max
            value: 56.47580379027271
          - type: nauc_map_at_100_std
            value: 67.38435515182985
          - type: nauc_map_at_10_diff1
            value: -12.481472547184628
          - type: nauc_map_at_10_max
            value: 33.265754813595265
          - type: nauc_map_at_10_std
            value: 18.22638592961484
          - type: nauc_map_at_1_diff1
            value: 1.2293816647460412
          - type: nauc_map_at_1_max
            value: 23.006245222860937
          - type: nauc_map_at_1_std
            value: 5.082037971661686
          - type: nauc_map_at_20_diff1
            value: -19.628676448681272
          - type: nauc_map_at_20_max
            value: 38.05654465725153
          - type: nauc_map_at_20_std
            value: 26.77958279973914
          - type: nauc_map_at_3_diff1
            value: 1.894562755188629
          - type: nauc_map_at_3_max
            value: 30.450207405528506
          - type: nauc_map_at_3_std
            value: 8.162817124670042
          - type: nauc_map_at_5_diff1
            value: 1.8118041815327512
          - type: nauc_map_at_5_max
            value: 36.07240804057612
          - type: nauc_map_at_5_std
            value: 10.818513452904504
          - type: nauc_mrr_at_1000_diff1
            value: -0.40174560390189173
          - type: nauc_mrr_at_1000_max
            value: 43.71069182389944
          - type: nauc_mrr_at_1000_std
            value: 37.65627005519199
          - type: nauc_mrr_at_100_diff1
            value: -0.40174560390189173
          - type: nauc_mrr_at_100_max
            value: 43.71069182389944
          - type: nauc_mrr_at_100_std
            value: 37.65627005519199
          - type: nauc_mrr_at_10_diff1
            value: -0.40174560390189173
          - type: nauc_mrr_at_10_max
            value: 43.71069182389944
          - type: nauc_mrr_at_10_std
            value: 37.65627005519199
          - type: nauc_mrr_at_1_diff1
            value: -7.977848922730832
          - type: nauc_mrr_at_1_max
            value: 37.61356753482742
          - type: nauc_mrr_at_1_std
            value: 34.039975772259226
          - type: nauc_mrr_at_20_diff1
            value: -0.40174560390189173
          - type: nauc_mrr_at_20_max
            value: 43.71069182389944
          - type: nauc_mrr_at_20_std
            value: 37.65627005519199
          - type: nauc_mrr_at_3_diff1
            value: 1.934992255034202
          - type: nauc_mrr_at_3_max
            value: 45.21685904162296
          - type: nauc_mrr_at_3_std
            value: 39.32443911457557
          - type: nauc_mrr_at_5_diff1
            value: 1.934992255034202
          - type: nauc_mrr_at_5_max
            value: 45.21685904162296
          - type: nauc_mrr_at_5_std
            value: 39.32443911457557
          - type: nauc_ndcg_at_1000_diff1
            value: -40.908094812392115
          - type: nauc_ndcg_at_1000_max
            value: 48.81499531922747
          - type: nauc_ndcg_at_1000_std
            value: 82.86337304804559
          - type: nauc_ndcg_at_100_diff1
            value: -44.56244464473406
          - type: nauc_ndcg_at_100_max
            value: 40.58624018883107
          - type: nauc_ndcg_at_100_std
            value: 80.45918244234892
          - type: nauc_ndcg_at_10_diff1
            value: -29.518074434000603
          - type: nauc_ndcg_at_10_max
            value: 43.48658903105973
          - type: nauc_ndcg_at_10_std
            value: 56.823103247390485
          - type: nauc_ndcg_at_1_diff1
            value: -26.938752512067293
          - type: nauc_ndcg_at_1_max
            value: 28.738050072310013
          - type: nauc_ndcg_at_1_std
            value: 38.51773942114455
          - type: nauc_ndcg_at_20_diff1
            value: -42.09374526480503
          - type: nauc_ndcg_at_20_max
            value: 39.88753819987586
          - type: nauc_ndcg_at_20_std
            value: 64.9188937434715
          - type: nauc_ndcg_at_3_diff1
            value: -18.983747164354074
          - type: nauc_ndcg_at_3_max
            value: 40.07068201143408
          - type: nauc_ndcg_at_3_std
            value: 44.520014381841264
          - type: nauc_ndcg_at_5_diff1
            value: -17.363745221747216
          - type: nauc_ndcg_at_5_max
            value: 45.3133946971861
          - type: nauc_ndcg_at_5_std
            value: 46.14383772600112
          - type: nauc_precision_at_1000_diff1
            value: -33.997592992905865
          - type: nauc_precision_at_1000_max
            value: 21.194962132845994
          - type: nauc_precision_at_1000_std
            value: 66.17573850908786
          - type: nauc_precision_at_100_diff1
            value: -45.17468834913693
          - type: nauc_precision_at_100_max
            value: 38.919854917565715
          - type: nauc_precision_at_100_std
            value: 81.8096172142314
          - type: nauc_precision_at_10_diff1
            value: -29.42371504029257
          - type: nauc_precision_at_10_max
            value: 44.73436091909716
          - type: nauc_precision_at_10_std
            value: 60.6846867495612
          - type: nauc_precision_at_1_diff1
            value: -7.977848922730832
          - type: nauc_precision_at_1_max
            value: 37.61356753482742
          - type: nauc_precision_at_1_std
            value: 34.039975772259226
          - type: nauc_precision_at_20_diff1
            value: -44.20238216430825
          - type: nauc_precision_at_20_max
            value: 39.17789315325717
          - type: nauc_precision_at_20_std
            value: 66.31262230366372
          - type: nauc_precision_at_3_diff1
            value: -4.094499601420467
          - type: nauc_precision_at_3_max
            value: 47.878469454308245
          - type: nauc_precision_at_3_std
            value: 44.82390028262911
          - type: nauc_precision_at_5_diff1
            value: -6.49531959342742
          - type: nauc_precision_at_5_max
            value: 50.725161705033905
          - type: nauc_precision_at_5_std
            value: 44.29713551082716
          - type: nauc_recall_at_1000_diff1
            value: -31.512611908780038
          - type: nauc_recall_at_1000_max
            value: 46.63381390751818
          - type: nauc_recall_at_1000_std
            value: 75.17070088378604
          - type: nauc_recall_at_100_diff1
            value: -28.875944584904158
          - type: nauc_recall_at_100_max
            value: 54.171746378883675
          - type: nauc_recall_at_100_std
            value: 56.1907274521453
          - type: nauc_recall_at_10_diff1
            value: -10.85790669811538
          - type: nauc_recall_at_10_max
            value: 27.66643799038461
          - type: nauc_recall_at_10_std
            value: 11.509747430055707
          - type: nauc_recall_at_1_diff1
            value: 1.2293816647460412
          - type: nauc_recall_at_1_max
            value: 23.006245222860937
          - type: nauc_recall_at_1_std
            value: 5.082037971661686
          - type: nauc_recall_at_20_diff1
            value: -15.447672155452471
          - type: nauc_recall_at_20_max
            value: 31.74240316416819
          - type: nauc_recall_at_20_std
            value: 15.003324764456485
          - type: nauc_recall_at_3_diff1
            value: 4.254127681493859
          - type: nauc_recall_at_3_max
            value: 30.840766518682173
          - type: nauc_recall_at_3_std
            value: 5.690923550364749
          - type: nauc_recall_at_5_diff1
            value: 4.089262491662224
          - type: nauc_recall_at_5_max
            value: 33.53882138961467
          - type: nauc_recall_at_5_std
            value: 6.213191881998446
          - type: ndcg_at_1
            value: 71
          - type: ndcg_at_10
            value: 67.33
          - type: ndcg_at_100
            value: 50.70099999999999
          - type: ndcg_at_1000
            value: 45.678999999999995
          - type: ndcg_at_20
            value: 64.181
          - type: ndcg_at_3
            value: 71.961
          - type: ndcg_at_5
            value: 69.14200000000001
          - type: precision_at_1
            value: 76
          - type: precision_at_10
            value: 72
          - type: precision_at_100
            value: 51.72
          - type: precision_at_1000
            value: 19.888
          - type: precision_at_20
            value: 68.2
          - type: precision_at_3
            value: 78
          - type: precision_at_5
            value: 74.4
          - type: recall_at_1
            value: 0.209
          - type: recall_at_10
            value: 1.9369999999999998
          - type: recall_at_100
            value: 12.466000000000001
          - type: recall_at_1000
            value: 42.535000000000004
          - type: recall_at_20
            value: 3.6189999999999998
          - type: recall_at_3
            value: 0.632
          - type: recall_at_5
            value: 1.01
        task:
          type: Retrieval
      - dataset:
          config: default
          name: MTEB TRECCOVID-PL (default)
          revision: 81bcb408f33366c2a20ac54adafad1ae7e877fdd
          split: test
          type: clarin-knext/trec-covid-pl
        metrics:
          - type: main_score
            value: 24.014
          - type: map_at_1
            value: 0.079
          - type: map_at_10
            value: 0.437
          - type: map_at_100
            value: 1.4160000000000001
          - type: map_at_1000
            value: 3.2960000000000003
          - type: map_at_20
            value: 0.6930000000000001
          - type: map_at_3
            value: 0.209
          - type: map_at_5
            value: 0.27599999999999997
          - type: mrr_at_1
            value: 32
          - type: mrr_at_10
            value: 40.766666666666666
          - type: mrr_at_100
            value: 42.065712760557446
          - type: mrr_at_1000
            value: 42.070147350357885
          - type: mrr_at_20
            value: 41.34215024741341
          - type: mrr_at_3
            value: 38
          - type: mrr_at_5
            value: 39.9
          - type: nauc_map_at_1000_diff1
            value: 2.763744410970452
          - type: nauc_map_at_1000_max
            value: 44.08217037883956
          - type: nauc_map_at_1000_std
            value: 45.726618026799784
          - type: nauc_map_at_100_diff1
            value: -1.1063775194811718
          - type: nauc_map_at_100_max
            value: 34.96148903048783
          - type: nauc_map_at_100_std
            value: 19.05911651363395
          - type: nauc_map_at_10_diff1
            value: 8.328485941895645
          - type: nauc_map_at_10_max
            value: 29.05954535135173
          - type: nauc_map_at_10_std
            value: -0.5663434331732777
          - type: nauc_map_at_1_diff1
            value: 15.648422889306588
          - type: nauc_map_at_1_max
            value: 33.43814058583505
          - type: nauc_map_at_1_std
            value: 0.38842622009531697
          - type: nauc_map_at_20_diff1
            value: 1.6239706005789203
          - type: nauc_map_at_20_max
            value: 30.780719931285766
          - type: nauc_map_at_20_std
            value: 5.226612687670864
          - type: nauc_map_at_3_diff1
            value: 14.691780792096715
          - type: nauc_map_at_3_max
            value: 29.72244349623907
          - type: nauc_map_at_3_std
            value: -3.894547135006907
          - type: nauc_map_at_5_diff1
            value: 7.625413465278537
          - type: nauc_map_at_5_max
            value: 28.061369021765792
          - type: nauc_map_at_5_std
            value: -1.6405791401954202
          - type: nauc_mrr_at_1000_diff1
            value: 9.903128384417203
          - type: nauc_mrr_at_1000_max
            value: 36.58476712156808
          - type: nauc_mrr_at_1000_std
            value: 11.948350197817941
          - type: nauc_mrr_at_100_diff1
            value: 9.911332457182505
          - type: nauc_mrr_at_100_max
            value: 36.5931154968881
          - type: nauc_mrr_at_100_std
            value: 11.923871293185774
          - type: nauc_mrr_at_10_diff1
            value: 10.061614580122114
          - type: nauc_mrr_at_10_max
            value: 35.90157508486573
          - type: nauc_mrr_at_10_std
            value: 11.683458301136916
          - type: nauc_mrr_at_1_diff1
            value: 13.560954725462931
          - type: nauc_mrr_at_1_max
            value: 41.29294659103072
          - type: nauc_mrr_at_1_std
            value: 23.92967997431232
          - type: nauc_mrr_at_20_diff1
            value: 9.754021662833347
          - type: nauc_mrr_at_20_max
            value: 35.765859492846374
          - type: nauc_mrr_at_20_std
            value: 11.848463872530932
          - type: nauc_mrr_at_3_diff1
            value: 10.028274713949735
          - type: nauc_mrr_at_3_max
            value: 38.88433792246913
          - type: nauc_mrr_at_3_std
            value: 11.48473077363494
          - type: nauc_mrr_at_5_diff1
            value: 9.335538525973245
          - type: nauc_mrr_at_5_max
            value: 37.515438355632824
          - type: nauc_mrr_at_5_std
            value: 11.070968041209992
          - type: nauc_ndcg_at_1000_diff1
            value: 5.584802003379831
          - type: nauc_ndcg_at_1000_max
            value: 33.314515131527834
          - type: nauc_ndcg_at_1000_std
            value: 33.6276642722341
          - type: nauc_ndcg_at_100_diff1
            value: 3.305268021968464
          - type: nauc_ndcg_at_100_max
            value: 31.908715963813822
          - type: nauc_ndcg_at_100_std
            value: 21.516527921316257
          - type: nauc_ndcg_at_10_diff1
            value: 7.032873935225729
          - type: nauc_ndcg_at_10_max
            value: 34.44205777276943
          - type: nauc_ndcg_at_10_std
            value: 7.196543440601352
          - type: nauc_ndcg_at_1_diff1
            value: 10.629073273765686
          - type: nauc_ndcg_at_1_max
            value: 50.10213581979629
          - type: nauc_ndcg_at_1_std
            value: 19.5308700304323
          - type: nauc_ndcg_at_20_diff1
            value: 4.1467973799104
          - type: nauc_ndcg_at_20_max
            value: 30.848940009171866
          - type: nauc_ndcg_at_20_std
            value: 10.699007932982358
          - type: nauc_ndcg_at_3_diff1
            value: 8.388484432353273
          - type: nauc_ndcg_at_3_max
            value: 40.198931190038564
          - type: nauc_ndcg_at_3_std
            value: 9.0164907464323
          - type: nauc_ndcg_at_5_diff1
            value: 4.519480833729923
          - type: nauc_ndcg_at_5_max
            value: 35.50336397828203
          - type: nauc_ndcg_at_5_std
            value: 8.89149616108275
          - type: nauc_precision_at_1000_diff1
            value: 10.810355967699419
          - type: nauc_precision_at_1000_max
            value: 36.198978350147165
          - type: nauc_precision_at_1000_std
            value: 50.02406670342153
          - type: nauc_precision_at_100_diff1
            value: 3.575117640307563
          - type: nauc_precision_at_100_max
            value: 32.374410110652754
          - type: nauc_precision_at_100_std
            value: 27.11269840198216
          - type: nauc_precision_at_10_diff1
            value: 8.163568076583704
          - type: nauc_precision_at_10_max
            value: 30.43788895048762
          - type: nauc_precision_at_10_std
            value: 7.492835893151349
          - type: nauc_precision_at_1_diff1
            value: 13.560954725462931
          - type: nauc_precision_at_1_max
            value: 41.29294659103072
          - type: nauc_precision_at_1_std
            value: 23.92967997431232
          - type: nauc_precision_at_20_diff1
            value: 3.9343182675494726
          - type: nauc_precision_at_20_max
            value: 27.92494826281862
          - type: nauc_precision_at_20_std
            value: 13.512522169356163
          - type: nauc_precision_at_3_diff1
            value: 7.500566731389045
          - type: nauc_precision_at_3_max
            value: 32.78072916153323
          - type: nauc_precision_at_3_std
            value: 5.442592574340379
          - type: nauc_precision_at_5_diff1
            value: 0.8368868088056831
          - type: nauc_precision_at_5_max
            value: 28.50268677413763
          - type: nauc_precision_at_5_std
            value: 6.262437164153216
          - type: nauc_recall_at_1000_diff1
            value: 6.466492011253024
          - type: nauc_recall_at_1000_max
            value: 33.29659361447259
          - type: nauc_recall_at_1000_std
            value: 38.33133164748577
          - type: nauc_recall_at_100_diff1
            value: -4.710612377509895
          - type: nauc_recall_at_100_max
            value: 33.36640370485505
          - type: nauc_recall_at_100_std
            value: 15.00124272531718
          - type: nauc_recall_at_10_diff1
            value: 8.383464512319398
          - type: nauc_recall_at_10_max
            value: 27.753627188606263
          - type: nauc_recall_at_10_std
            value: -6.430500905036131
          - type: nauc_recall_at_1_diff1
            value: 15.648422889306588
          - type: nauc_recall_at_1_max
            value: 33.43814058583505
          - type: nauc_recall_at_1_std
            value: 0.38842622009531697
          - type: nauc_recall_at_20_diff1
            value: 0.6220898729888212
          - type: nauc_recall_at_20_max
            value: 26.706071721975373
          - type: nauc_recall_at_20_std
            value: -0.07228409632534197
          - type: nauc_recall_at_3_diff1
            value: 14.399942747468852
          - type: nauc_recall_at_3_max
            value: 29.839734495614795
          - type: nauc_recall_at_3_std
            value: -7.342370795101286
          - type: nauc_recall_at_5_diff1
            value: 4.307738089595829
          - type: nauc_recall_at_5_max
            value: 26.81904935896474
          - type: nauc_recall_at_5_std
            value: -7.309908018949135
          - type: ndcg_at_1
            value: 28.999999999999996
          - type: ndcg_at_10
            value: 24.014
          - type: ndcg_at_100
            value: 14.601
          - type: ndcg_at_1000
            value: 12.147
          - type: ndcg_at_20
            value: 22.093
          - type: ndcg_at_3
            value: 28.531000000000002
          - type: ndcg_at_5
            value: 26.311
          - type: precision_at_1
            value: 32
          - type: precision_at_10
            value: 24.8
          - type: precision_at_100
            value: 14.06
          - type: precision_at_1000
            value: 5.814
          - type: precision_at_20
            value: 23.1
          - type: precision_at_3
            value: 31.333
          - type: precision_at_5
            value: 27.6
          - type: recall_at_1
            value: 0.079
          - type: recall_at_10
            value: 0.584
          - type: recall_at_100
            value: 2.843
          - type: recall_at_1000
            value: 11.053
          - type: recall_at_20
            value: 1.031
          - type: recall_at_3
            value: 0.22699999999999998
          - type: recall_at_5
            value: 0.327
        task:
          type: Retrieval

base_model: BAAI/bge-base-en-v1.5 datasets: [] metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 widget: - source_sentence: The Opa1 protein localizes to the mitochondria.Opa1 is found normally in the mitochondrial intermembrane space. sentences: - Which is the cellular localization of the protein Opa1? - Which are the genes responsible for Dyskeratosis Congenita? - List blood marker for Non-Hodgkin lymphoma. - source_sentence: CorrSite identifies potential allosteric ligand-binding sites based on motion correlation analyses between cavities.We find that CARDS captures allosteric communication between the two cAMP-Binding Domains (CBDs)Overall, it is demonstrated that the communication pathways could be multiple and intrinsically disposed, and the MC path generation approach provides an effective tool for the prediction of key residues that mediate the allosteric communication in an ensemble of pathways and functionally plausible residuesWe utilized a data set of 24 known allosteric sites from 23 monomer proteins to calculate the correlations between potential ligand-binding sites and corresponding orthosteric sites using a Gaussian network model (GNM)Here, we introduce the Correlation of All Rotameric and Dynamical States (CARDS) framework for quantifying correlations between both the structure and disorder of different regions of a proteinWe present a novel method, "MutInf", to identify statistically significant correlated motions from equilibrium molecular dynamics simulationsCorrSite identifies potential allosteric ligand-binding sites based on motion correlation analyses between cavities.Here, a Monte Carlo (MC) path generation approach is proposed and implemented to define likely allosteric pathways through generating an ensemble of maximum probability paths.Here, a Monte Carlo (MC) path generation approach is proposed and implemented to define likely allosteric pathways through generating an ensemble of maximum probability paths. Overall, it is demonstrated that the communication pathways could be multiple and intrinsically disposed, and the MC path generation approach provides an effective tool for the prediction of key residues that mediate the allosteric communication in an ensemble of pathways and functionally plausible residues We utilized a data set of 24 known allosteric sites from 23 monomer proteins to calculate the correlations between potential ligand-binding sites and corresponding orthosteric sites using a Gaussian network model (GNM)A Monte Carlo (MC) path generation approach is proposed and implemented to define likely allosteric pathways through generating an ensemble of maximum probability paths. A novel method, "MutInf", to identify statistically significant correlated motions from equilibrium molecular dynamics simulations. CorrSite identifies potential alloster-binding sites based on motion correlation analyses between cavities. The Correlation of All Rotameric and Dynamical States (CARDS) framework for quantifying correlations between both the structure and disorder of different regions of a proteinComputational tools for predicting allosteric pathways in proteins include MCPath, MutInf, pySCA, CorrSite, and CARDS. sentences: - Computational tools for predicting allosteric pathways in proteins - What is PANTHER-PSEP? - What illness is transmitted by the Lone Star Tick, Amblyomma americanum? - source_sentence: "Dopaminergic drugs should be given in patients with BMS. \nCatuama
\ reduces the symptoms of BMS and may be a novel therapeutic strategy for the
\ treatment of this disease.\nCapsaicin, alpha-lipoic acid (ALA), and clonazepam
\ were those that showed more reduction in symptoms of BMS.\nTreatment with placebos
\ produced a response that was 72% as large as the response to active drugs" sentences: - What is the cyberknife used for? - Which compounds exist that are thyroid hormone analogs? - Which are the drugs utilized for the burning mouth syndrome? - source_sentence: Tinea is a superficial fungal infections of the skin. sentences: - Which molecule is targeted by a monoclonal antibody Mepolizumab? - What disease is tinea ? - Which algorithm is used for detection of long repeat expansions? - source_sentence: Basset is an open source package which applies CNNs to learn the functional activity of DNA sequences from genomics data. Basset was trained on a compendium of accessible genomic sites mapped in 164 cell types by DNase-seq, and demonstrated greater predictive accuracy than previous methods. Basset predictions for the change in accessibility between variant alleles were far greater for Genome-wide association study (GWAS) SNPs that are likely to be causal relative to nearby SNPs in linkage disequilibrium with them. With Basset, a researcher can perform a single sequencing assay in their cell type of interest and simultaneously learn that cell's chromatin accessibility code and annotate every mutation in the genome with its influence on present accessibility and latent potential for accessibility. Thus, Basset offers a powerful computational approach to annotate and interpret the noncoding genome. sentences: - Givosiran is used for treatment of which disease? - Describe the applicability of Basset in the context of deep learning - What is the causative agent of the "Panama disease" affecting bananas? pipeline_tag: sentence-similarity model-index: - name: BGE base BioASQ Matryoshka results: - task: type: information-retrieval name: Information Retrieval dataset: name: dim 768 type: dim_768 metrics: - type: cosine_accuracy@1 value: 0.8432203389830508 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.9427966101694916 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.961864406779661 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9788135593220338 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8432203389830508 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.3142655367231638 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19237288135593222 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.0978813559322034 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.8432203389830508 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.9427966101694916 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.961864406779661 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9788135593220338 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9167805960832026 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.8963327280064567 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.8971987609787653 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 512 type: dim_512 metrics: - type: cosine_accuracy@1 value: 0.8538135593220338 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.9427966101694916 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.961864406779661 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9745762711864406 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8538135593220338 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.3142655367231638 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19237288135593222 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09745762711864407 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.8538135593220338 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.9427966101694916 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.961864406779661 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9745762711864406 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9198462326957965 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.9016772598870054 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.9026755533837086 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 256 type: dim_256 metrics: - type: cosine_accuracy@1 value: 0.8453389830508474 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.9385593220338984 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9555084745762712 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9745762711864406 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8453389830508474 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.3128531073446327 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19110169491525425 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09745762711864407 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.8453389830508474 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.9385593220338984 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9555084745762712 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9745762711864406 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.914207272128957 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.8944528517621736 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.8952712251263324 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 128 type: dim_128 metrics: - type: cosine_accuracy@1 value: 0.8220338983050848 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.9279661016949152 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9449152542372882 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9703389830508474 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8220338983050848 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.3093220338983051 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.18898305084745767 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09703389830508474 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.8220338983050848 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.9279661016949152 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9449152542372882 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9703389830508474 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.901534580728345 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.8789800242130752 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.8801051507894794 name: Cosine Map@100

BGE base BioASQ Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: BAAI/bge-base-en-v1.5
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 768 tokens
  • Similarity Function: Cosine Similarity
  • Language: en
  • License: apache-2.0

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("pavanmantha/bge-base-en-bioembed768")
# Run inference
sentences = [
    "Basset is an open source package which applies CNNs to learn the functional activity of DNA sequences from genomics data. Basset was trained on a compendium of accessible genomic sites mapped in 164 cell types by DNase-seq, and demonstrated greater predictive accuracy than previous methods. Basset predictions for the change in accessibility between variant alleles were far greater for Genome-wide association study (GWAS) SNPs that are likely to be causal relative to nearby SNPs in linkage disequilibrium with them. With Basset, a researcher can perform a single sequencing assay in their cell type of interest and simultaneously learn that cell's chromatin accessibility code and annotate every mutation in the genome with its influence on present accessibility and latent potential for accessibility. Thus, Basset offers a powerful computational approach to annotate and interpret the noncoding genome.",
    'Describe the applicability of Basset in the context of deep learning',
    'What is the causative agent of the "Panama disease" affecting bananas?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Metric Value
cosine_accuracy@1 0.8432
cosine_accuracy@3 0.9428
cosine_accuracy@5 0.9619
cosine_accuracy@10 0.9788
cosine_precision@1 0.8432
cosine_precision@3 0.3143
cosine_precision@5 0.1924
cosine_precision@10 0.0979
cosine_recall@1 0.8432
cosine_recall@3 0.9428
cosine_recall@5 0.9619
cosine_recall@10 0.9788
cosine_ndcg@10 0.9168
cosine_mrr@10 0.8963
cosine_map@100 0.8972

Information Retrieval

Metric Value
cosine_accuracy@1 0.8538
cosine_accuracy@3 0.9428
cosine_accuracy@5 0.9619
cosine_accuracy@10 0.9746
cosine_precision@1 0.8538
cosine_precision@3 0.3143
cosine_precision@5 0.1924
cosine_precision@10 0.0975
cosine_recall@1 0.8538
cosine_recall@3 0.9428
cosine_recall@5 0.9619
cosine_recall@10 0.9746
cosine_ndcg@10 0.9198
cosine_mrr@10 0.9017
cosine_map@100 0.9027

Information Retrieval

Metric Value
cosine_accuracy@1 0.8453
cosine_accuracy@3 0.9386
cosine_accuracy@5 0.9555
cosine_accuracy@10 0.9746
cosine_precision@1 0.8453
cosine_precision@3 0.3129
cosine_precision@5 0.1911
cosine_precision@10 0.0975
cosine_recall@1 0.8453
cosine_recall@3 0.9386
cosine_recall@5 0.9555
cosine_recall@10 0.9746
cosine_ndcg@10 0.9142
cosine_mrr@10 0.8945
cosine_map@100 0.8953

Information Retrieval

Metric Value
cosine_accuracy@1 0.822
cosine_accuracy@3 0.928
cosine_accuracy@5 0.9449
cosine_accuracy@10 0.9703
cosine_precision@1 0.822
cosine_precision@3 0.3093
cosine_precision@5 0.189
cosine_precision@10 0.097
cosine_recall@1 0.822
cosine_recall@3 0.928
cosine_recall@5 0.9449
cosine_recall@10 0.9703
cosine_ndcg@10 0.9015
cosine_mrr@10 0.879
cosine_map@100 0.8801

Training Details

Training Dataset

Unnamed Dataset

  • Size: 4,247 training samples
  • Columns: positive and anchor
  • Approximate statistics based on the first 1000 samples:
    positive anchor
    type string string
    details
    • min: 3 tokens
    • mean: 102.44 tokens
    • max: 512 tokens
    • min: 5 tokens
    • mean: 15.78 tokens
    • max: 44 tokens
  • Samples:
    positive anchor
    Restless legs syndrome (RLS), also known as Willis-Ekbom disease (WED), is a common movement disorder characterized by an uncontrollable urge to move because of uncomfortable, sometimes painful sensations in the legs with a diurnal variation and a release with movement. Willis-Ekbom disease is also known as?
    Report the outcomes of laser in situ keratomileusis (LASIK) for high myopia correction after long-term follow-up['Report the outcomes of laser in situ keratomileusis (LASIK) for high myopia correction after long-term follow-up.']Laser in situ keratomileusis is also known as LASIKLaser in situ keratomileusis (LASIK) What is another name for keratomileusis?
    CellMaps is an HTML5 open-source web tool that allows displaying, editing, exploring and analyzing biological networks as well as integrating metadata into them.CellMaps is an HTML5 open-source web tool that allows displaying, editing, exploring and analyzing biological networks as well as integrating metadata into them. CellMaps can easily be integrated in any web page by using an available JavaScript API. Computations and analyses are remotely executed in high-end servers, and all the functionalities are available through RESTful web services. CellMaps is an HTML5 open-source web tool that allows displaying, editing, exploring and analyzing biological networks as well as integrating metadata into them. Computations and analyses are remotely executed in high-end servers, and all the functionalities are available through RESTful web services. CellMaps can easily be integrated in any web page by using an available JavaScript API. CellMaps is an HTML5 open-source web tool that allows displaying, editing, exploring and analyzing biological networks as well as integrating metadata into them. Computations and analyses are remotely executed in high-end servers, and all the functionalities are available through RESTful web services. CellMaps can easily be integrated in any web page by using an available JavaScript API.CellMaps is an HTML5 open-source web tool that allows displaying, editing, exploring and analyzing biological networks as well as integrating metadata into them. CellMaps is an HTML5 open-source web tool that allows displaying, editing, exploring and analyzing biological networks as well as integrating metadata into them. CellMaps can easily be integrated in any web page by using an available JavaScript API. Computations and analyses are remotely executed in high-end servers, and all the functionalities are available through RESTful web services. What is CellMaps?
  • Loss: MatryoshkaLoss with these parameters:
    {
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: epoch
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 16
  • gradient_accumulation_steps: 16
  • learning_rate: 2e-05
  • num_train_epochs: 10
  • lr_scheduler_type: cosine
  • warmup_ratio: 0.1
  • fp16: True
  • tf32: False
  • load_best_model_at_end: True
  • optim: adamw_torch_fused
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: epoch
  • prediction_loss_only: True
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 16
  • eval_accumulation_steps: None
  • learning_rate: 2e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 10
  • max_steps: -1
  • lr_scheduler_type: cosine
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: True
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: False
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss dim_128_cosine_map@100 dim_256_cosine_map@100 dim_512_cosine_map@100 dim_768_cosine_map@100
0.9624 8 - 0.8560 0.8821 0.8904 0.8876
1.2030 10 1.2833 - - - -
1.9248 16 - 0.8655 0.8808 0.8909 0.8889
2.4060 20 0.4785 - - - -
2.8872 24 - 0.8720 0.8875 0.8893 0.8921
3.6090 30 0.2417 - - - -
3.9699 33 - 0.8751 0.8924 0.8955 0.8960
4.8120 40 0.1607 - - - -
4.9323 41 - 0.8799 0.8932 0.8964 0.8952
5.8947 49 - 0.8785 0.8944 0.9009 0.8982
6.0150 50 0.1152 - - - -
6.9774 58 - 0.8803 0.8947 0.9018 0.8975
7.2180 60 0.0924 - - - -
7.9398 66 - 0.8802 0.8956 0.9016 0.8973
8.4211 70 0.0832 - - - -
8.9023 74 - 0.8801 0.8956 0.9027 0.8972
9.6241 80 0.074 0.8801 0.8953 0.9027 0.8972
  • The bold row denotes the saved checkpoint.

Framework Versions

  • Python: 3.10.13
  • Sentence Transformers: 3.0.1
  • Transformers: 4.41.2
  • PyTorch: 2.1.2
  • Accelerate: 0.31.0
  • Datasets: 2.19.2
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss 

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss 

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}