--- language: - en license: apache-2.0 library_name: sentence-transformers tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:4247 - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss model-index: - name: pavanmantha/bge-base-en-bioembed768 results: - dataset: config: default name: MTEB MedicalQARetrieval (default) revision: ae763399273d8b20506b80cf6f6f9a31a6a2b238 split: test type: mteb/medical_qa metrics: - type: main_score value: 68.65599999999999 - type: map_at_1 value: 51.025 - type: map_at_10 value: 62.946000000000005 - type: map_at_100 value: 63.392 - type: map_at_1000 value: 63.408 - type: map_at_20 value: 63.254 - type: map_at_3 value: 60.221000000000004 - type: map_at_5 value: 62.022999999999996 - type: mrr_at_1 value: 51.025390625 - type: mrr_at_10 value: 62.94629293774791 - type: mrr_at_100 value: 63.391942840845125 - type: mrr_at_1000 value: 63.40789553144974 - type: mrr_at_20 value: 63.25463963215003 - type: mrr_at_3 value: 60.22135416666657 - type: mrr_at_5 value: 62.02311197916664 - type: nauc_map_at_1000_diff1 value: 44.0289077012448 - type: nauc_map_at_1000_max value: 40.196044082980045 - type: nauc_map_at_1000_std value: -13.087939597961418 - type: nauc_map_at_100_diff1 value: 44.02100215716583 - type: nauc_map_at_100_max value: 40.20884959562626 - type: nauc_map_at_100_std value: -13.074614347446076 - type: nauc_map_at_10_diff1 value: 43.78028009647677 - type: nauc_map_at_10_max value: 40.01480272155306 - type: nauc_map_at_10_std value: -13.377550749542138 - type: nauc_map_at_1_diff1 value: 49.57988513879831 - type: nauc_map_at_1_max value: 40.609452025451915 - type: nauc_map_at_1_std value: -11.541103718714977 - type: nauc_map_at_20_diff1 value: 43.87022043596322 - type: nauc_map_at_20_max value: 40.22321319847734 - type: nauc_map_at_20_std value: -13.164262713651956 - type: nauc_map_at_3_diff1 value: 44.38171551545852 - type: nauc_map_at_3_max value: 39.6459216129053 - type: nauc_map_at_3_std value: -13.770104903701466 - type: nauc_map_at_5_diff1 value: 43.8534318488618 - type: nauc_map_at_5_max value: 39.68004513144695 - type: nauc_map_at_5_std value: -13.877556205848146 - type: nauc_mrr_at_1000_diff1 value: 44.028920021140976 - type: nauc_mrr_at_1000_max value: 40.23569448794487 - type: nauc_mrr_at_1000_std value: -12.956236343591568 - type: nauc_mrr_at_100_diff1 value: 44.02100738083268 - type: nauc_mrr_at_100_max value: 40.24846931727988 - type: nauc_mrr_at_100_std value: -12.942990526259535 - type: nauc_mrr_at_10_diff1 value: 43.78028009647677 - type: nauc_mrr_at_10_max value: 40.05165903518751 - type: nauc_mrr_at_10_std value: -13.249179890670767 - type: nauc_mrr_at_1_diff1 value: 49.57988513879831 - type: nauc_mrr_at_1_max value: 40.64858186876722 - type: nauc_mrr_at_1_std value: -11.320856258088295 - type: nauc_mrr_at_20_diff1 value: 43.87022486033709 - type: nauc_mrr_at_20_max value: 40.26263859989682 - type: nauc_mrr_at_20_std value: -13.033285163966218 - type: nauc_mrr_at_3_diff1 value: 44.38171551545852 - type: nauc_mrr_at_3_max value: 39.69840946271301 - type: nauc_mrr_at_3_std value: -13.633807447557226 - type: nauc_mrr_at_5_diff1 value: 43.8534318488618 - type: nauc_mrr_at_5_max value: 39.72797855901197 - type: nauc_mrr_at_5_std value: -13.747197261890934 - type: nauc_ndcg_at_1000_diff1 value: 42.6428514635724 - type: nauc_ndcg_at_1000_max value: 40.48576100703463 - type: nauc_ndcg_at_1000_std value: -12.705673993292077 - type: nauc_ndcg_at_100_diff1 value: 42.17588553015554 - type: nauc_ndcg_at_100_max value: 40.80768178826183 - type: nauc_ndcg_at_100_std value: -12.05166428600645 - type: nauc_ndcg_at_10_diff1 value: 40.65597689427392 - type: nauc_ndcg_at_10_max value: 40.175863350340215 - type: nauc_ndcg_at_10_std value: -13.557398725708799 - type: nauc_ndcg_at_1_diff1 value: 49.57988513879831 - type: nauc_ndcg_at_1_max value: 40.609452025451915 - type: nauc_ndcg_at_1_std value: -11.541103718714977 - type: nauc_ndcg_at_20_diff1 value: 40.913600886170634 - type: nauc_ndcg_at_20_max value: 41.00724137368964 - type: nauc_ndcg_at_20_std value: -12.593947509615822 - type: nauc_ndcg_at_3_diff1 value: 42.40019474173081 - type: nauc_ndcg_at_3_max value: 39.2565483832548 - type: nauc_ndcg_at_3_std value: -14.612074432976202 - type: nauc_ndcg_at_5_diff1 value: 41.15688939938367 - type: nauc_ndcg_at_5_max value: 39.35405468594706 - type: nauc_ndcg_at_5_std value: -14.899193680376587 - type: nauc_precision_at_1000_diff1 value: 29.214715925497835 - type: nauc_precision_at_1000_max value: 66.23687386181616 - type: nauc_precision_at_1000_std value: 20.38998945371634 - type: nauc_precision_at_100_diff1 value: 16.184818683990663 - type: nauc_precision_at_100_max value: 57.004633019507764 - type: nauc_precision_at_100_std value: 24.749447988511026 - type: nauc_precision_at_10_diff1 value: 20.14231168000379 - type: nauc_precision_at_10_max value: 41.81505494607868 - type: nauc_precision_at_10_std value: -13.588327036007039 - type: nauc_precision_at_1_diff1 value: 49.57988513879831 - type: nauc_precision_at_1_max value: 40.609452025451915 - type: nauc_precision_at_1_std value: -11.541103718714977 - type: nauc_precision_at_20_diff1 value: 13.936251711556528 - type: nauc_precision_at_20_max value: 50.63499099834573 - type: nauc_precision_at_20_std value: -3.4568556682579055 - type: nauc_precision_at_3_diff1 value: 35.342792247725725 - type: nauc_precision_at_3_max value: 37.86162276112827 - type: nauc_precision_at_3_std value: -17.607999329247022 - type: nauc_precision_at_5_diff1 value: 28.95330037767198 - type: nauc_precision_at_5_max value: 37.98848038772262 - type: nauc_precision_at_5_std value: -19.46990621093604 - type: nauc_recall_at_1000_diff1 value: 29.214715925499096 - type: nauc_recall_at_1000_max value: 66.23687386181345 - type: nauc_recall_at_1000_std value: 20.389989453717625 - type: nauc_recall_at_100_diff1 value: 16.184818683991136 - type: nauc_recall_at_100_max value: 57.004633019507104 - type: nauc_recall_at_100_std value: 24.74944798851128 - type: nauc_recall_at_10_diff1 value: 20.14231168000389 - type: nauc_recall_at_10_max value: 41.81505494607885 - type: nauc_recall_at_10_std value: -13.588327036006865 - type: nauc_recall_at_1_diff1 value: 49.57988513879831 - type: nauc_recall_at_1_max value: 40.609452025451915 - type: nauc_recall_at_1_std value: -11.541103718714977 - type: nauc_recall_at_20_diff1 value: 13.936251711556865 - type: nauc_recall_at_20_max value: 50.63499099834594 - type: nauc_recall_at_20_std value: -3.4568556682576865 - type: nauc_recall_at_3_diff1 value: 35.34279224772573 - type: nauc_recall_at_3_max value: 37.86162276112829 - type: nauc_recall_at_3_std value: -17.607999329247097 - type: nauc_recall_at_5_diff1 value: 28.953300377672097 - type: nauc_recall_at_5_max value: 37.98848038772265 - type: nauc_recall_at_5_std value: -19.469906210936013 - type: ndcg_at_1 value: 51.025 - type: ndcg_at_10 value: 68.65599999999999 - type: ndcg_at_100 value: 70.78 - type: ndcg_at_1000 value: 71.207 - type: ndcg_at_20 value: 69.771 - type: ndcg_at_3 value: 63.202999999999996 - type: ndcg_at_5 value: 66.425 - type: precision_at_1 value: 51.025 - type: precision_at_10 value: 8.647 - type: precision_at_100 value: 0.963 - type: precision_at_1000 value: 0.1 - type: precision_at_20 value: 4.543 - type: precision_at_3 value: 23.942 - type: precision_at_5 value: 15.918 - type: recall_at_1 value: 51.025 - type: recall_at_10 value: 86.47500000000001 - type: recall_at_100 value: 96.33800000000001 - type: recall_at_1000 value: 99.70700000000001 - type: recall_at_20 value: 90.869 - type: recall_at_3 value: 71.82600000000001 - type: recall_at_5 value: 79.59 task: type: Retrieval - dataset: config: default name: MTEB TRECCOVID (default) revision: bb9466bac8153a0349341eb1b22e06409e78ef4e split: test type: mteb/trec-covid metrics: - type: main_score value: 67.33 - type: map_at_1 value: 0.209 - type: map_at_10 value: 1.694 - type: map_at_100 value: 9.078999999999999 - type: map_at_1000 value: 21.590999999999998 - type: map_at_20 value: 2.973 - type: map_at_3 value: 0.594 - type: map_at_5 value: 0.9259999999999999 - type: mrr_at_1 value: 76.0 - type: mrr_at_10 value: 86.66666666666666 - type: mrr_at_100 value: 86.66666666666666 - type: mrr_at_1000 value: 86.66666666666666 - type: mrr_at_20 value: 86.66666666666666 - type: mrr_at_3 value: 86.33333333333334 - type: mrr_at_5 value: 86.33333333333334 - type: nauc_map_at_1000_diff1 value: -39.36100175513192 - type: nauc_map_at_1000_max value: 51.929525559618305 - type: nauc_map_at_1000_std value: 88.1016909960378 - type: nauc_map_at_100_diff1 value: -34.61735014934496 - type: nauc_map_at_100_max value: 56.47580379027271 - type: nauc_map_at_100_std value: 67.38435515182985 - type: nauc_map_at_10_diff1 value: -12.481472547184628 - type: nauc_map_at_10_max value: 33.265754813595265 - type: nauc_map_at_10_std value: 18.22638592961484 - type: nauc_map_at_1_diff1 value: 1.2293816647460412 - type: nauc_map_at_1_max value: 23.006245222860937 - type: nauc_map_at_1_std value: 5.082037971661686 - type: nauc_map_at_20_diff1 value: -19.628676448681272 - type: nauc_map_at_20_max value: 38.05654465725153 - type: nauc_map_at_20_std value: 26.77958279973914 - type: nauc_map_at_3_diff1 value: 1.894562755188629 - type: nauc_map_at_3_max value: 30.450207405528506 - type: nauc_map_at_3_std value: 8.162817124670042 - type: nauc_map_at_5_diff1 value: 1.8118041815327512 - type: nauc_map_at_5_max value: 36.07240804057612 - type: nauc_map_at_5_std value: 10.818513452904504 - type: nauc_mrr_at_1000_diff1 value: -0.40174560390189173 - type: nauc_mrr_at_1000_max value: 43.71069182389944 - type: nauc_mrr_at_1000_std value: 37.65627005519199 - type: nauc_mrr_at_100_diff1 value: -0.40174560390189173 - type: nauc_mrr_at_100_max value: 43.71069182389944 - type: nauc_mrr_at_100_std value: 37.65627005519199 - type: nauc_mrr_at_10_diff1 value: -0.40174560390189173 - type: nauc_mrr_at_10_max value: 43.71069182389944 - type: nauc_mrr_at_10_std value: 37.65627005519199 - type: nauc_mrr_at_1_diff1 value: -7.977848922730832 - type: nauc_mrr_at_1_max value: 37.61356753482742 - type: nauc_mrr_at_1_std value: 34.039975772259226 - type: nauc_mrr_at_20_diff1 value: -0.40174560390189173 - type: nauc_mrr_at_20_max value: 43.71069182389944 - type: nauc_mrr_at_20_std value: 37.65627005519199 - type: nauc_mrr_at_3_diff1 value: 1.934992255034202 - type: nauc_mrr_at_3_max value: 45.21685904162296 - type: nauc_mrr_at_3_std value: 39.32443911457557 - type: nauc_mrr_at_5_diff1 value: 1.934992255034202 - type: nauc_mrr_at_5_max value: 45.21685904162296 - type: nauc_mrr_at_5_std value: 39.32443911457557 - type: nauc_ndcg_at_1000_diff1 value: -40.908094812392115 - type: nauc_ndcg_at_1000_max value: 48.81499531922747 - type: nauc_ndcg_at_1000_std value: 82.86337304804559 - type: nauc_ndcg_at_100_diff1 value: -44.56244464473406 - type: nauc_ndcg_at_100_max value: 40.58624018883107 - type: nauc_ndcg_at_100_std value: 80.45918244234892 - type: nauc_ndcg_at_10_diff1 value: -29.518074434000603 - type: nauc_ndcg_at_10_max value: 43.48658903105973 - type: nauc_ndcg_at_10_std value: 56.823103247390485 - type: nauc_ndcg_at_1_diff1 value: -26.938752512067293 - type: nauc_ndcg_at_1_max value: 28.738050072310013 - type: nauc_ndcg_at_1_std value: 38.51773942114455 - type: nauc_ndcg_at_20_diff1 value: -42.09374526480503 - type: nauc_ndcg_at_20_max value: 39.88753819987586 - type: nauc_ndcg_at_20_std value: 64.9188937434715 - type: nauc_ndcg_at_3_diff1 value: -18.983747164354074 - type: nauc_ndcg_at_3_max value: 40.07068201143408 - type: nauc_ndcg_at_3_std value: 44.520014381841264 - type: nauc_ndcg_at_5_diff1 value: -17.363745221747216 - type: nauc_ndcg_at_5_max value: 45.3133946971861 - type: nauc_ndcg_at_5_std value: 46.14383772600112 - type: nauc_precision_at_1000_diff1 value: -33.997592992905865 - type: nauc_precision_at_1000_max value: 21.194962132845994 - type: nauc_precision_at_1000_std value: 66.17573850908786 - type: nauc_precision_at_100_diff1 value: -45.17468834913693 - type: nauc_precision_at_100_max value: 38.919854917565715 - type: nauc_precision_at_100_std value: 81.8096172142314 - type: nauc_precision_at_10_diff1 value: -29.42371504029257 - type: nauc_precision_at_10_max value: 44.73436091909716 - type: nauc_precision_at_10_std value: 60.6846867495612 - type: nauc_precision_at_1_diff1 value: -7.977848922730832 - type: nauc_precision_at_1_max value: 37.61356753482742 - type: nauc_precision_at_1_std value: 34.039975772259226 - type: nauc_precision_at_20_diff1 value: -44.20238216430825 - type: nauc_precision_at_20_max value: 39.17789315325717 - type: nauc_precision_at_20_std value: 66.31262230366372 - type: nauc_precision_at_3_diff1 value: -4.094499601420467 - type: nauc_precision_at_3_max value: 47.878469454308245 - type: nauc_precision_at_3_std value: 44.82390028262911 - type: nauc_precision_at_5_diff1 value: -6.49531959342742 - type: nauc_precision_at_5_max value: 50.725161705033905 - type: nauc_precision_at_5_std value: 44.29713551082716 - type: nauc_recall_at_1000_diff1 value: -31.512611908780038 - type: nauc_recall_at_1000_max value: 46.63381390751818 - type: nauc_recall_at_1000_std value: 75.17070088378604 - type: nauc_recall_at_100_diff1 value: -28.875944584904158 - type: nauc_recall_at_100_max value: 54.171746378883675 - type: nauc_recall_at_100_std value: 56.1907274521453 - type: nauc_recall_at_10_diff1 value: -10.85790669811538 - type: nauc_recall_at_10_max value: 27.66643799038461 - type: nauc_recall_at_10_std value: 11.509747430055707 - type: nauc_recall_at_1_diff1 value: 1.2293816647460412 - type: nauc_recall_at_1_max value: 23.006245222860937 - type: nauc_recall_at_1_std value: 5.082037971661686 - type: nauc_recall_at_20_diff1 value: -15.447672155452471 - type: nauc_recall_at_20_max value: 31.74240316416819 - type: nauc_recall_at_20_std value: 15.003324764456485 - type: nauc_recall_at_3_diff1 value: 4.254127681493859 - type: nauc_recall_at_3_max value: 30.840766518682173 - type: nauc_recall_at_3_std value: 5.690923550364749 - type: nauc_recall_at_5_diff1 value: 4.089262491662224 - type: nauc_recall_at_5_max value: 33.53882138961467 - type: nauc_recall_at_5_std value: 6.213191881998446 - type: ndcg_at_1 value: 71.0 - type: ndcg_at_10 value: 67.33 - type: ndcg_at_100 value: 50.70099999999999 - type: ndcg_at_1000 value: 45.678999999999995 - type: ndcg_at_20 value: 64.181 - type: ndcg_at_3 value: 71.961 - type: ndcg_at_5 value: 69.14200000000001 - type: precision_at_1 value: 76.0 - type: precision_at_10 value: 72.0 - type: precision_at_100 value: 51.72 - type: precision_at_1000 value: 19.888 - type: precision_at_20 value: 68.2 - type: precision_at_3 value: 78.0 - type: precision_at_5 value: 74.4 - type: recall_at_1 value: 0.209 - type: recall_at_10 value: 1.9369999999999998 - type: recall_at_100 value: 12.466000000000001 - type: recall_at_1000 value: 42.535000000000004 - type: recall_at_20 value: 3.6189999999999998 - type: recall_at_3 value: 0.632 - type: recall_at_5 value: 1.01 task: type: Retrieval - dataset: config: default name: MTEB TRECCOVID-PL (default) revision: 81bcb408f33366c2a20ac54adafad1ae7e877fdd split: test type: clarin-knext/trec-covid-pl metrics: - type: main_score value: 24.014 - type: map_at_1 value: 0.079 - type: map_at_10 value: 0.437 - type: map_at_100 value: 1.4160000000000001 - type: map_at_1000 value: 3.2960000000000003 - type: map_at_20 value: 0.6930000000000001 - type: map_at_3 value: 0.209 - type: map_at_5 value: 0.27599999999999997 - type: mrr_at_1 value: 32.0 - type: mrr_at_10 value: 40.766666666666666 - type: mrr_at_100 value: 42.065712760557446 - type: mrr_at_1000 value: 42.070147350357885 - type: mrr_at_20 value: 41.34215024741341 - type: mrr_at_3 value: 38.0 - type: mrr_at_5 value: 39.9 - type: nauc_map_at_1000_diff1 value: 2.763744410970452 - type: nauc_map_at_1000_max value: 44.08217037883956 - type: nauc_map_at_1000_std value: 45.726618026799784 - type: nauc_map_at_100_diff1 value: -1.1063775194811718 - type: nauc_map_at_100_max value: 34.96148903048783 - type: nauc_map_at_100_std value: 19.05911651363395 - type: nauc_map_at_10_diff1 value: 8.328485941895645 - type: nauc_map_at_10_max value: 29.05954535135173 - type: nauc_map_at_10_std value: -0.5663434331732777 - type: nauc_map_at_1_diff1 value: 15.648422889306588 - type: nauc_map_at_1_max value: 33.43814058583505 - type: nauc_map_at_1_std value: 0.38842622009531697 - type: nauc_map_at_20_diff1 value: 1.6239706005789203 - type: nauc_map_at_20_max value: 30.780719931285766 - type: nauc_map_at_20_std value: 5.226612687670864 - type: nauc_map_at_3_diff1 value: 14.691780792096715 - type: nauc_map_at_3_max value: 29.72244349623907 - type: nauc_map_at_3_std value: -3.894547135006907 - type: nauc_map_at_5_diff1 value: 7.625413465278537 - type: nauc_map_at_5_max value: 28.061369021765792 - type: nauc_map_at_5_std value: -1.6405791401954202 - type: nauc_mrr_at_1000_diff1 value: 9.903128384417203 - type: nauc_mrr_at_1000_max value: 36.58476712156808 - type: nauc_mrr_at_1000_std value: 11.948350197817941 - type: nauc_mrr_at_100_diff1 value: 9.911332457182505 - type: nauc_mrr_at_100_max value: 36.5931154968881 - type: nauc_mrr_at_100_std value: 11.923871293185774 - type: nauc_mrr_at_10_diff1 value: 10.061614580122114 - type: nauc_mrr_at_10_max value: 35.90157508486573 - type: nauc_mrr_at_10_std value: 11.683458301136916 - type: nauc_mrr_at_1_diff1 value: 13.560954725462931 - type: nauc_mrr_at_1_max value: 41.29294659103072 - type: nauc_mrr_at_1_std value: 23.92967997431232 - type: nauc_mrr_at_20_diff1 value: 9.754021662833347 - type: nauc_mrr_at_20_max value: 35.765859492846374 - type: nauc_mrr_at_20_std value: 11.848463872530932 - type: nauc_mrr_at_3_diff1 value: 10.028274713949735 - type: nauc_mrr_at_3_max value: 38.88433792246913 - type: nauc_mrr_at_3_std value: 11.48473077363494 - type: nauc_mrr_at_5_diff1 value: 9.335538525973245 - type: nauc_mrr_at_5_max value: 37.515438355632824 - type: nauc_mrr_at_5_std value: 11.070968041209992 - type: nauc_ndcg_at_1000_diff1 value: 5.584802003379831 - type: nauc_ndcg_at_1000_max value: 33.314515131527834 - type: nauc_ndcg_at_1000_std value: 33.6276642722341 - type: nauc_ndcg_at_100_diff1 value: 3.305268021968464 - type: nauc_ndcg_at_100_max value: 31.908715963813822 - type: nauc_ndcg_at_100_std value: 21.516527921316257 - type: nauc_ndcg_at_10_diff1 value: 7.032873935225729 - type: nauc_ndcg_at_10_max value: 34.44205777276943 - type: nauc_ndcg_at_10_std value: 7.196543440601352 - type: nauc_ndcg_at_1_diff1 value: 10.629073273765686 - type: nauc_ndcg_at_1_max value: 50.10213581979629 - type: nauc_ndcg_at_1_std value: 19.5308700304323 - type: nauc_ndcg_at_20_diff1 value: 4.1467973799104 - type: nauc_ndcg_at_20_max value: 30.848940009171866 - type: nauc_ndcg_at_20_std value: 10.699007932982358 - type: nauc_ndcg_at_3_diff1 value: 8.388484432353273 - type: nauc_ndcg_at_3_max value: 40.198931190038564 - type: nauc_ndcg_at_3_std value: 9.0164907464323 - type: nauc_ndcg_at_5_diff1 value: 4.519480833729923 - type: nauc_ndcg_at_5_max value: 35.50336397828203 - type: nauc_ndcg_at_5_std value: 8.89149616108275 - type: nauc_precision_at_1000_diff1 value: 10.810355967699419 - type: nauc_precision_at_1000_max value: 36.198978350147165 - type: nauc_precision_at_1000_std value: 50.02406670342153 - type: nauc_precision_at_100_diff1 value: 3.575117640307563 - type: nauc_precision_at_100_max value: 32.374410110652754 - type: nauc_precision_at_100_std value: 27.11269840198216 - type: nauc_precision_at_10_diff1 value: 8.163568076583704 - type: nauc_precision_at_10_max value: 30.43788895048762 - type: nauc_precision_at_10_std value: 7.492835893151349 - type: nauc_precision_at_1_diff1 value: 13.560954725462931 - type: nauc_precision_at_1_max value: 41.29294659103072 - type: nauc_precision_at_1_std value: 23.92967997431232 - type: nauc_precision_at_20_diff1 value: 3.9343182675494726 - type: nauc_precision_at_20_max value: 27.92494826281862 - type: nauc_precision_at_20_std value: 13.512522169356163 - type: nauc_precision_at_3_diff1 value: 7.500566731389045 - type: nauc_precision_at_3_max value: 32.78072916153323 - type: nauc_precision_at_3_std value: 5.442592574340379 - type: nauc_precision_at_5_diff1 value: 0.8368868088056831 - type: nauc_precision_at_5_max value: 28.50268677413763 - type: nauc_precision_at_5_std value: 6.262437164153216 - type: nauc_recall_at_1000_diff1 value: 6.466492011253024 - type: nauc_recall_at_1000_max value: 33.29659361447259 - type: nauc_recall_at_1000_std value: 38.33133164748577 - type: nauc_recall_at_100_diff1 value: -4.710612377509895 - type: nauc_recall_at_100_max value: 33.36640370485505 - type: nauc_recall_at_100_std value: 15.00124272531718 - type: nauc_recall_at_10_diff1 value: 8.383464512319398 - type: nauc_recall_at_10_max value: 27.753627188606263 - type: nauc_recall_at_10_std value: -6.430500905036131 - type: nauc_recall_at_1_diff1 value: 15.648422889306588 - type: nauc_recall_at_1_max value: 33.43814058583505 - type: nauc_recall_at_1_std value: 0.38842622009531697 - type: nauc_recall_at_20_diff1 value: 0.6220898729888212 - type: nauc_recall_at_20_max value: 26.706071721975373 - type: nauc_recall_at_20_std value: -0.07228409632534197 - type: nauc_recall_at_3_diff1 value: 14.399942747468852 - type: nauc_recall_at_3_max value: 29.839734495614795 - type: nauc_recall_at_3_std value: -7.342370795101286 - type: nauc_recall_at_5_diff1 value: 4.307738089595829 - type: nauc_recall_at_5_max value: 26.81904935896474 - type: nauc_recall_at_5_std value: -7.309908018949135 - type: ndcg_at_1 value: 28.999999999999996 - type: ndcg_at_10 value: 24.014 - type: ndcg_at_100 value: 14.601 - type: ndcg_at_1000 value: 12.147 - type: ndcg_at_20 value: 22.093 - type: ndcg_at_3 value: 28.531000000000002 - type: ndcg_at_5 value: 26.311 - type: precision_at_1 value: 32.0 - type: precision_at_10 value: 24.8 - type: precision_at_100 value: 14.06 - type: precision_at_1000 value: 5.814 - type: precision_at_20 value: 23.1 - type: precision_at_3 value: 31.333 - type: precision_at_5 value: 27.6 - type: recall_at_1 value: 0.079 - type: recall_at_10 value: 0.584 - type: recall_at_100 value: 2.843 - type: recall_at_1000 value: 11.053 - type: recall_at_20 value: 1.031 - type: recall_at_3 value: 0.22699999999999998 - type: recall_at_5 value: 0.327 task: type: Retrieval --- base_model: BAAI/bge-base-en-v1.5 datasets: [] metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 widget: - source_sentence: The Opa1 protein localizes to the mitochondria.Opa1 is found normally in the mitochondrial intermembrane space. sentences: - Which is the cellular localization of the protein Opa1? - Which are the genes responsible for Dyskeratosis Congenita? - List blood marker for Non-Hodgkin lymphoma. - source_sentence: CorrSite identifies potential allosteric ligand-binding sites based on motion correlation analyses between cavities.We find that CARDS captures allosteric communication between the two cAMP-Binding Domains (CBDs)Overall, it is demonstrated that the communication pathways could be multiple and intrinsically disposed, and the MC path generation approach provides an effective tool for the prediction of key residues that mediate the allosteric communication in an ensemble of pathways and functionally plausible residuesWe utilized a data set of 24 known allosteric sites from 23 monomer proteins to calculate the correlations between potential ligand-binding sites and corresponding orthosteric sites using a Gaussian network model (GNM)Here, we introduce the Correlation of All Rotameric and Dynamical States (CARDS) framework for quantifying correlations between both the structure and disorder of different regions of a proteinWe present a novel method, "MutInf", to identify statistically significant correlated motions from equilibrium molecular dynamics simulationsCorrSite identifies potential allosteric ligand-binding sites based on motion correlation analyses between cavities.Here, a Monte Carlo (MC) path generation approach is proposed and implemented to define likely allosteric pathways through generating an ensemble of maximum probability paths.Here, a Monte Carlo (MC) path generation approach is proposed and implemented to define likely allosteric pathways through generating an ensemble of maximum probability paths. Overall, it is demonstrated that the communication pathways could be multiple and intrinsically disposed, and the MC path generation approach provides an effective tool for the prediction of key residues that mediate the allosteric communication in an ensemble of pathways and functionally plausible residues We utilized a data set of 24 known allosteric sites from 23 monomer proteins to calculate the correlations between potential ligand-binding sites and corresponding orthosteric sites using a Gaussian network model (GNM)A Monte Carlo (MC) path generation approach is proposed and implemented to define likely allosteric pathways through generating an ensemble of maximum probability paths. A novel method, "MutInf", to identify statistically significant correlated motions from equilibrium molecular dynamics simulations. CorrSite identifies potential alloster-binding sites based on motion correlation analyses between cavities. The Correlation of All Rotameric and Dynamical States (CARDS) framework for quantifying correlations between both the structure and disorder of different regions of a proteinComputational tools for predicting allosteric pathways in proteins include MCPath, MutInf, pySCA, CorrSite, and CARDS. sentences: - Computational tools for predicting allosteric pathways in proteins - What is PANTHER-PSEP? - What illness is transmitted by the Lone Star Tick, Amblyomma americanum? - source_sentence: "Dopaminergic drugs should be given in patients with BMS. \nCatuama\ \ reduces the symptoms of BMS and may be a novel therapeutic strategy for the\ \ treatment of this disease.\nCapsaicin, alpha-lipoic acid (ALA), and clonazepam\ \ were those that showed more reduction in symptoms of BMS.\nTreatment with placebos\ \ produced a response that was 72% as large as the response to active drugs" sentences: - What is the cyberknife used for? - Which compounds exist that are thyroid hormone analogs? - Which are the drugs utilized for the burning mouth syndrome? - source_sentence: Tinea is a superficial fungal infections of the skin. sentences: - Which molecule is targeted by a monoclonal antibody Mepolizumab? - What disease is tinea ? - Which algorithm is used for detection of long repeat expansions? - source_sentence: Basset is an open source package which applies CNNs to learn the functional activity of DNA sequences from genomics data. Basset was trained on a compendium of accessible genomic sites mapped in 164 cell types by DNase-seq, and demonstrated greater predictive accuracy than previous methods. Basset predictions for the change in accessibility between variant alleles were far greater for Genome-wide association study (GWAS) SNPs that are likely to be causal relative to nearby SNPs in linkage disequilibrium with them. With Basset, a researcher can perform a single sequencing assay in their cell type of interest and simultaneously learn that cell's chromatin accessibility code and annotate every mutation in the genome with its influence on present accessibility and latent potential for accessibility. Thus, Basset offers a powerful computational approach to annotate and interpret the noncoding genome. sentences: - Givosiran is used for treatment of which disease? - Describe the applicability of Basset in the context of deep learning - What is the causative agent of the "Panama disease" affecting bananas? pipeline_tag: sentence-similarity model-index: - name: BGE base BioASQ Matryoshka results: - task: type: information-retrieval name: Information Retrieval dataset: name: dim 768 type: dim_768 metrics: - type: cosine_accuracy@1 value: 0.8432203389830508 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.9427966101694916 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.961864406779661 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9788135593220338 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8432203389830508 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.3142655367231638 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19237288135593222 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.0978813559322034 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.8432203389830508 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.9427966101694916 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.961864406779661 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9788135593220338 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9167805960832026 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.8963327280064567 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.8971987609787653 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 512 type: dim_512 metrics: - type: cosine_accuracy@1 value: 0.8538135593220338 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.9427966101694916 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.961864406779661 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9745762711864406 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8538135593220338 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.3142655367231638 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19237288135593222 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09745762711864407 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.8538135593220338 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.9427966101694916 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.961864406779661 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9745762711864406 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9198462326957965 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.9016772598870054 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.9026755533837086 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 256 type: dim_256 metrics: - type: cosine_accuracy@1 value: 0.8453389830508474 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.9385593220338984 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9555084745762712 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9745762711864406 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8453389830508474 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.3128531073446327 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19110169491525425 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09745762711864407 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.8453389830508474 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.9385593220338984 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9555084745762712 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9745762711864406 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.914207272128957 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.8944528517621736 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.8952712251263324 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 128 type: dim_128 metrics: - type: cosine_accuracy@1 value: 0.8220338983050848 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.9279661016949152 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9449152542372882 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9703389830508474 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8220338983050848 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.3093220338983051 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.18898305084745767 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09703389830508474 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.8220338983050848 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.9279661016949152 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9449152542372882 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9703389830508474 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.901534580728345 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.8789800242130752 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.8801051507894794 name: Cosine Map@100 --- # BGE base BioASQ Matryoshka This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-base-en-v1.5](https://huggingface.co./BAAI/bge-base-en-v1.5). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [BAAI/bge-base-en-v1.5](https://huggingface.co./BAAI/bge-base-en-v1.5) - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 768 tokens - **Similarity Function:** Cosine Similarity - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co./models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("pavanmantha/bge-base-en-bioembed768") # Run inference sentences = [ "Basset is an open source package which applies CNNs to learn the functional activity of DNA sequences from genomics data. Basset was trained on a compendium of accessible genomic sites mapped in 164 cell types by DNase-seq, and demonstrated greater predictive accuracy than previous methods. Basset predictions for the change in accessibility between variant alleles were far greater for Genome-wide association study (GWAS) SNPs that are likely to be causal relative to nearby SNPs in linkage disequilibrium with them. With Basset, a researcher can perform a single sequencing assay in their cell type of interest and simultaneously learn that cell's chromatin accessibility code and annotate every mutation in the genome with its influence on present accessibility and latent potential for accessibility. Thus, Basset offers a powerful computational approach to annotate and interpret the noncoding genome.", 'Describe the applicability of Basset in the context of deep learning', 'What is the causative agent of the "Panama disease" affecting bananas?', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Information Retrieval * Dataset: `dim_768` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.8432 | | cosine_accuracy@3 | 0.9428 | | cosine_accuracy@5 | 0.9619 | | cosine_accuracy@10 | 0.9788 | | cosine_precision@1 | 0.8432 | | cosine_precision@3 | 0.3143 | | cosine_precision@5 | 0.1924 | | cosine_precision@10 | 0.0979 | | cosine_recall@1 | 0.8432 | | cosine_recall@3 | 0.9428 | | cosine_recall@5 | 0.9619 | | cosine_recall@10 | 0.9788 | | cosine_ndcg@10 | 0.9168 | | cosine_mrr@10 | 0.8963 | | **cosine_map@100** | **0.8972** | #### Information Retrieval * Dataset: `dim_512` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.8538 | | cosine_accuracy@3 | 0.9428 | | cosine_accuracy@5 | 0.9619 | | cosine_accuracy@10 | 0.9746 | | cosine_precision@1 | 0.8538 | | cosine_precision@3 | 0.3143 | | cosine_precision@5 | 0.1924 | | cosine_precision@10 | 0.0975 | | cosine_recall@1 | 0.8538 | | cosine_recall@3 | 0.9428 | | cosine_recall@5 | 0.9619 | | cosine_recall@10 | 0.9746 | | cosine_ndcg@10 | 0.9198 | | cosine_mrr@10 | 0.9017 | | **cosine_map@100** | **0.9027** | #### Information Retrieval * Dataset: `dim_256` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.8453 | | cosine_accuracy@3 | 0.9386 | | cosine_accuracy@5 | 0.9555 | | cosine_accuracy@10 | 0.9746 | | cosine_precision@1 | 0.8453 | | cosine_precision@3 | 0.3129 | | cosine_precision@5 | 0.1911 | | cosine_precision@10 | 0.0975 | | cosine_recall@1 | 0.8453 | | cosine_recall@3 | 0.9386 | | cosine_recall@5 | 0.9555 | | cosine_recall@10 | 0.9746 | | cosine_ndcg@10 | 0.9142 | | cosine_mrr@10 | 0.8945 | | **cosine_map@100** | **0.8953** | #### Information Retrieval * Dataset: `dim_128` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.822 | | cosine_accuracy@3 | 0.928 | | cosine_accuracy@5 | 0.9449 | | cosine_accuracy@10 | 0.9703 | | cosine_precision@1 | 0.822 | | cosine_precision@3 | 0.3093 | | cosine_precision@5 | 0.189 | | cosine_precision@10 | 0.097 | | cosine_recall@1 | 0.822 | | cosine_recall@3 | 0.928 | | cosine_recall@5 | 0.9449 | | cosine_recall@10 | 0.9703 | | cosine_ndcg@10 | 0.9015 | | cosine_mrr@10 | 0.879 | | **cosine_map@100** | **0.8801** | ## Training Details ### Training Dataset #### Unnamed Dataset * Size: 4,247 training samples * Columns: positive and anchor * Approximate statistics based on the first 1000 samples: | | positive | anchor | |:--------|:------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | positive | anchor | |:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------| | Restless legs syndrome (RLS), also known as Willis-Ekbom disease (WED), is a common movement disorder characterized by an uncontrollable urge to move because of uncomfortable, sometimes painful sensations in the legs with a diurnal variation and a release with movement. | Willis-Ekbom disease is also known as? | | Report the outcomes of laser in situ keratomileusis (LASIK) for high myopia correction after long-term follow-up['Report the outcomes of laser in situ keratomileusis (LASIK) for high myopia correction after long-term follow-up.']Laser in situ keratomileusis is also known as LASIKLaser in situ keratomileusis (LASIK) | What is another name for keratomileusis? | | CellMaps is an HTML5 open-source web tool that allows displaying, editing, exploring and analyzing biological networks as well as integrating metadata into them.CellMaps is an HTML5 open-source web tool that allows displaying, editing, exploring and analyzing biological networks as well as integrating metadata into them. CellMaps can easily be integrated in any web page by using an available JavaScript API. Computations and analyses are remotely executed in high-end servers, and all the functionalities are available through RESTful web services. CellMaps is an HTML5 open-source web tool that allows displaying, editing, exploring and analyzing biological networks as well as integrating metadata into them. Computations and analyses are remotely executed in high-end servers, and all the functionalities are available through RESTful web services. CellMaps can easily be integrated in any web page by using an available JavaScript API. CellMaps is an HTML5 open-source web tool that allows displaying, editing, exploring and analyzing biological networks as well as integrating metadata into them. Computations and analyses are remotely executed in high-end servers, and all the functionalities are available through RESTful web services. CellMaps can easily be integrated in any web page by using an available JavaScript API.CellMaps is an HTML5 open-source web tool that allows displaying, editing, exploring and analyzing biological networks as well as integrating metadata into them. CellMaps is an HTML5 open-source web tool that allows displaying, editing, exploring and analyzing biological networks as well as integrating metadata into them. CellMaps can easily be integrated in any web page by using an available JavaScript API. Computations and analyses are remotely executed in high-end servers, and all the functionalities are available through RESTful web services. | What is CellMaps? | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "MultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128 ], "matryoshka_weights": [ 1, 1, 1, 1 ], "n_dims_per_step": -1 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `gradient_accumulation_steps`: 16 - `learning_rate`: 2e-05 - `num_train_epochs`: 10 - `lr_scheduler_type`: cosine - `warmup_ratio`: 0.1 - `fp16`: True - `tf32`: False - `load_best_model_at_end`: True - `optim`: adamw_torch_fused - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 16 - `eval_accumulation_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 10 - `max_steps`: -1 - `lr_scheduler_type`: cosine - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: False - `fp16`: True - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: False - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch_fused - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | dim_128_cosine_map@100 | dim_256_cosine_map@100 | dim_512_cosine_map@100 | dim_768_cosine_map@100 | |:----------:|:------:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:----------------------:| | 0.9624 | 8 | - | 0.8560 | 0.8821 | 0.8904 | 0.8876 | | 1.2030 | 10 | 1.2833 | - | - | - | - | | 1.9248 | 16 | - | 0.8655 | 0.8808 | 0.8909 | 0.8889 | | 2.4060 | 20 | 0.4785 | - | - | - | - | | 2.8872 | 24 | - | 0.8720 | 0.8875 | 0.8893 | 0.8921 | | 3.6090 | 30 | 0.2417 | - | - | - | - | | 3.9699 | 33 | - | 0.8751 | 0.8924 | 0.8955 | 0.8960 | | 4.8120 | 40 | 0.1607 | - | - | - | - | | 4.9323 | 41 | - | 0.8799 | 0.8932 | 0.8964 | 0.8952 | | 5.8947 | 49 | - | 0.8785 | 0.8944 | 0.9009 | 0.8982 | | 6.0150 | 50 | 0.1152 | - | - | - | - | | **6.9774** | **58** | **-** | **0.8803** | **0.8947** | **0.9018** | **0.8975** | | 7.2180 | 60 | 0.0924 | - | - | - | - | | 7.9398 | 66 | - | 0.8802 | 0.8956 | 0.9016 | 0.8973 | | 8.4211 | 70 | 0.0832 | - | - | - | - | | 8.9023 | 74 | - | 0.8801 | 0.8956 | 0.9027 | 0.8972 | | 9.6241 | 80 | 0.074 | 0.8801 | 0.8953 | 0.9027 | 0.8972 | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.10.13 - Sentence Transformers: 3.0.1 - Transformers: 4.41.2 - PyTorch: 2.1.2 - Accelerate: 0.31.0 - Datasets: 2.19.2 - Tokenizers: 0.19.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MatryoshkaLoss ```bibtex @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```