hanwenzhu/all-distilroberta-v1-lr2e-4-bs1024-nneg3-ml-feb04

SentenceTransformer based on sentence-transformers/all-distilroberta-v1

This is a sentence-transformers model finetuned from sentence-transformers/all-distilroberta-v1. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-distilroberta-v1
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("hanwenzhu/all-distilroberta-v1-lr2e-4-bs1024-nneg3-ml-feb04")
# Run inference
sentences = [
    'Mathlib.AlgebraicGeometry.Noetherian#22',
    'AlgebraicGeometry.of_affine_open_cover',
    'pow_lt_pow_right_of_lt_one₀',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 4,232,571 training samples
• Columns: state_name and premise_name
• Approximate statistics based on the first 1000 samples:

  	state_name	premise_name
  type	string	string
  details	min: 11 tokens mean: 16.91 tokens max: 28 tokens	min: 3 tokens mean: 10.27 tokens max: 27 tokens

• Samples:

  state_name	premise_name
  Mathlib.Algebra.Group.Subgroup.Pointwise#27	Set.mul_subgroupClosure
  Mathlib.Algebra.Group.Subgroup.Pointwise#27	pow_succ
  Mathlib.Algebra.Group.Subgroup.Pointwise#27	mul_assoc

• Loss: loss.MaskedCachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 1,648 evaluation samples
• Columns: state_name and premise_name
• Approximate statistics based on the first 1000 samples:

  	state_name	premise_name
  type	string	string
  details	min: 12 tokens mean: 17.34 tokens max: 26 tokens	min: 3 tokens mean: 10.9 tokens max: 34 tokens

• Samples:

  state_name	premise_name
  Mathlib.Algebra.BigOperators.Associated#0	Prime.dvd_or_dvd
  Mathlib.Algebra.BigOperators.Associated#0	Multiset.induction_on
  Mathlib.Algebra.BigOperators.Associated#0	Multiset.mem_cons_of_mem

• Loss: loss.MaskedCachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 1024
• per_device_eval_batch_size: 64
• learning_rate: 0.0002
• num_train_epochs: 1.0
• lr_scheduler_type: cosine
• warmup_ratio: 0.03
• bf16: True
• dataloader_num_workers: 4
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 1024
• per_device_eval_batch_size: 64
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 0.0002
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1.0
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.03
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 4
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss	loss
0.0024	10	6.4677	-
0.0048	20	5.8525	-
0.0073	30	5.4638	-
0.0097	40	5.1916	-
0.0102	42	-	1.8024
0.0121	50	5.0406	-
0.0145	60	4.9618	-
0.0169	70	4.8628	-
0.0194	80	4.8094	-
0.0203	84	-	1.6661
0.0218	90	4.728	-
0.0242	100	4.6682	-
0.0266	110	4.6147	-
0.0290	120	4.6194	-
0.0305	126	-	1.4863
0.0314	130	4.5386	-
0.0339	140	4.4644	-
0.0363	150	4.4456	-
0.0387	160	4.4112	-
0.0406	168	-	1.4766
0.0411	170	4.3334	-
0.0435	180	4.326	-
0.0460	190	4.2364	-
0.0484	200	4.232	-
0.0508	210	4.211	1.2885
0.0532	220	4.16	-
0.0556	230	4.1569	-
0.0581	240	4.1202	-
0.0605	250	4.0603	-
0.0610	252	-	1.3337
0.0629	260	4.0629	-
0.0653	270	4.0401	-
0.0677	280	4.0171	-
0.0701	290	4.0034	-
0.0711	294	-	1.2049
0.0726	300	3.9911	-
0.0750	310	3.893	-
0.0774	320	3.8999	-
0.0798	330	3.9395	-
0.0813	336	-	1.2090
0.0822	340	3.9087	-
0.0847	350	3.925	-
0.0871	360	3.8888	-
0.0895	370	3.8282	-
0.0914	378	-	1.2345
0.0919	380	3.8787	-
0.0943	390	3.8517	-
0.0968	400	3.7446	-
0.0992	410	3.7888	-
0.1016	420	3.7833	1.2055
0.1040	430	3.7522	-
0.1064	440	3.7601	-
0.1089	450	3.7093	-
0.1113	460	3.713	-
0.1118	462	-	1.2316
0.1137	470	3.6461	-
0.1161	480	3.6153	-
0.1185	490	3.6419	-
0.1209	500	3.5929	-
0.1219	504	-	1.1385
0.1234	510	3.5881	-
0.1258	520	3.6296	-
0.1282	530	3.6302	-
0.1306	540	3.59	-
0.1321	546	-	1.1055
0.1330	550	3.5682	-
0.1355	560	3.5808	-
0.1379	570	3.4821	-
0.1403	580	3.5309	-
0.1422	588	-	1.1142
0.1427	590	3.4714	-
0.1451	600	3.4859	-
0.1476	610	3.4835	-
0.1500	620	3.4335	-
0.1524	630	3.459	1.0673
0.1548	640	3.4735	-
0.1572	650	3.3819	-
0.1597	660	3.3618	-
0.1621	670	3.317	-
0.1626	672	-	1.0852
0.1645	680	3.3512	-
0.1669	690	3.3519	-
0.1693	700	3.3672	-
0.1717	710	3.3021	-
0.1727	714	-	1.0624
0.1742	720	3.3557	-
0.1766	730	3.3778	-
0.1790	740	3.2999	-
0.1814	750	3.2691	-
0.1829	756	-	1.0505
0.1838	760	3.268	-
0.1863	770	3.2883	-
0.1887	780	3.2546	-
0.1911	790	3.2313	-
0.1930	798	-	1.0156
0.1935	800	3.2847	-
0.1959	810	3.2852	-
0.1984	820	3.2001	-
0.2008	830	3.2561	-
0.2032	840	3.1764	0.9978
0.2056	850	3.1471	-
0.2080	860	3.1506	-
0.2104	870	3.1225	-
0.2129	880	3.2139	-
0.2134	882	-	0.9681
0.2153	890	3.1746	-
0.2177	900	3.0522	-
0.2201	910	3.1581	-
0.2225	920	3.1535	-
0.2235	924	-	0.9900
0.2250	930	3.1659	-
0.2274	940	3.069	-
0.2298	950	3.1406	-
0.2322	960	3.1171	-
0.2337	966	-	0.9881
0.2346	970	3.0983	-
0.2371	980	3.0748	-
0.2395	990	3.0279	-
0.2419	1000	3.0465	-
0.2438	1008	-	0.9158
0.2443	1010	2.9777	-
0.2467	1020	2.9791	-
0.2492	1030	2.9773	-
0.2516	1040	3.0372	-
0.2540	1050	2.998	0.9662
0.2564	1060	3.0012	-
0.2588	1070	2.9436	-
0.2612	1080	2.9661	-
0.2637	1090	3.0087	-
0.2642	1092	-	0.9232
0.2661	1100	2.9713	-
0.2685	1110	2.9653	-
0.2709	1120	2.9268	-
0.2733	1130	2.9321	-
0.2743	1134	-	0.9229
0.2758	1140	2.884	-
0.2782	1150	2.9282	-
0.2806	1160	2.9197	-
0.2830	1170	2.8831	-
0.2845	1176	-	0.9419
0.2854	1180	2.8617	-
0.2879	1190	2.8926	-
0.2903	1200	2.8259	-
0.2927	1210	2.7944	-
0.2946	1218	-	0.8967
0.2951	1220	2.7952	-
0.2975	1230	2.8103	-
0.3000	1240	2.7346	-
0.3024	1250	2.7674	-
0.3048	1260	2.7845	0.9320
0.3072	1270	2.8123	-
0.3096	1280	2.7938	-
0.3120	1290	2.8232	-
0.3145	1300	2.7676	-
0.3149	1302	-	0.8623
0.3169	1310	2.756	-
0.3193	1320	2.7683	-
0.3217	1330	2.8046	-
0.3241	1340	2.8106	-
0.3251	1344	-	0.8718
0.3266	1350	2.7349	-
0.3290	1360	2.747	-
0.3314	1370	2.7276	-
0.3338	1380	2.764	-
0.3353	1386	-	0.8143
0.3362	1390	2.7233	-
0.3387	1400	2.694	-
0.3411	1410	2.7002	-
0.3435	1420	2.705	-
0.3454	1428	-	0.8237
0.3459	1430	2.6625	-
0.3483	1440	2.6954	-
0.3507	1450	2.6671	-
0.3532	1460	2.6801	-
0.3556	1470	2.6724	0.8144
0.3580	1480	2.6525	-
0.3604	1490	2.6905	-
0.3628	1500	2.62	-
0.3653	1510	2.6563	-
0.3657	1512	-	0.8374
0.3677	1520	2.6774	-
0.3701	1530	2.6183	-
0.3725	1540	2.6164	-
0.3749	1550	2.6187	-
0.3759	1554	-	0.8515
0.3774	1560	2.6186	-
0.3798	1570	2.6555	-
0.3822	1580	2.6541	-
0.3846	1590	2.6595	-
0.3861	1596	-	0.8322
0.3870	1600	2.6699	-
0.3895	1610	2.6404	-
0.3919	1620	2.5847	-
0.3943	1630	2.6133	-
0.3962	1638	-	0.8247
0.3967	1640	2.6379	-
0.3991	1650	2.6635	-
0.4015	1660	2.6122	-
0.4040	1670	2.6352	-
0.4064	1680	2.5765	0.8084
0.4088	1690	2.58	-
0.4112	1700	2.5415	-
0.4136	1710	2.5199	-
0.4161	1720	2.5341	-
0.4165	1722	-	0.8113
0.4185	1730	2.5443	-
0.4209	1740	2.5558	-
0.4233	1750	2.547	-
0.4257	1760	2.5383	-
0.4267	1764	-	0.7759
0.4282	1770	2.5656	-
0.4306	1780	2.5711	-
0.4330	1790	2.4755	-
0.4354	1800	2.4792	-
0.4369	1806	-	0.7821
0.4378	1810	2.489	-
0.4403	1820	2.4752	-
0.4427	1830	2.4708	-
0.4451	1840	2.5073	-
0.4470	1848	-	0.7817
0.4475	1850	2.4888	-
0.4499	1860	2.4498	-
0.4523	1870	2.4089	-
0.4548	1880	2.4877	-
0.4572	1890	2.4617	0.7354
0.4596	1900	2.4715	-
0.4620	1910	2.4175	-
0.4644	1920	2.4554	-
0.4669	1930	2.4406	-
0.4673	1932	-	0.7383
0.4693	1940	2.4692	-
0.4717	1950	2.4418	-
0.4741	1960	2.5003	-
0.4765	1970	2.4467	-
0.4775	1974	-	0.7271
0.4790	1980	2.4256	-
0.4814	1990	2.4249	-
0.4838	2000	2.3878	-
0.4862	2010	2.3596	-
0.4877	2016	-	0.7600
0.4886	2020	2.3905	-
0.4910	2030	2.4118	-
0.4935	2040	2.3785	-
0.4959	2050	2.4141	-
0.4978	2058	-	0.7464
0.4983	2060	2.3821	-
0.5007	2070	2.3513	-
0.5031	2080	2.3928	-
0.5056	2090	2.3406	-
0.5080	2100	2.3967	0.7483
0.5104	2110	2.4006	-
0.5128	2120	2.3474	-
0.5152	2130	2.395	-
0.5177	2140	2.3809	-
0.5181	2142	-	0.7415
0.5201	2150	2.3646	-
0.5225	2160	2.3675	-
0.5249	2170	2.3295	-
0.5273	2180	2.3365	-
0.5283	2184	-	0.7332
0.5298	2190	2.3974	-
0.5322	2200	2.2649	-
0.5346	2210	2.3317	-
0.5370	2220	2.3452	-
0.5385	2226	-	0.7313
0.5394	2230	2.3219	-
0.5418	2240	2.3616	-
0.5443	2250	2.3093	-
0.5467	2260	2.2907	-
0.5486	2268	-	0.7111
0.5491	2270	2.3125	-
0.5515	2280	2.3176	-
0.5539	2290	2.3241	-
0.5564	2300	2.3287	-
0.5588	2310	2.2922	0.6895
0.5612	2320	2.3044	-
0.5636	2330	2.3215	-
0.5660	2340	2.3486	-
0.5685	2350	2.3294	-
0.5689	2352	-	0.7046
0.5709	2360	2.3204	-
0.5733	2370	2.273	-
0.5757	2380	2.2787	-
0.5781	2390	2.2772	-
0.5791	2394	-	0.6885
0.5806	2400	2.3089	-
0.5830	2410	2.3235	-
0.5854	2420	2.251	-
0.5878	2430	2.2445	-
0.5893	2436	-	0.6755
0.5902	2440	2.2201	-
0.5926	2450	2.2468	-
0.5951	2460	2.2094	-
0.5975	2470	2.2455	-
0.5994	2478	-	0.6602
0.5999	2480	2.2538	-
0.6023	2490	2.2125	-
0.6047	2500	2.2468	-
0.6072	2510	2.2107	-
0.6096	2520	2.2291	0.6725
0.6120	2530	2.2051	-
0.6144	2540	2.2408	-
0.6168	2550	2.1987	-
0.6193	2560	2.1743	-
0.6197	2562	-	0.6907
0.6217	2570	2.2191	-
0.6241	2580	2.1297	-
0.6265	2590	2.1588	-
0.6289	2600	2.171	-
0.6299	2604	-	0.6736
0.6313	2610	2.1956	-
0.6338	2620	2.1818	-
0.6362	2630	2.1472	-
0.6386	2640	2.1413	-
0.6401	2646	-	0.6672
0.6410	2650	2.1891	-
0.6434	2660	2.228	-
0.6459	2670	2.1725	-
0.6483	2680	2.1622	-
0.6502	2688	-	0.6465
0.6507	2690	2.1245	-
0.6531	2700	2.1485	-
0.6555	2710	2.1685	-
0.6580	2720	2.1808	-
0.6604	2730	2.1547	0.6508
0.6628	2740	2.1459	-
0.6652	2750	2.1755	-
0.6676	2760	2.1741	-
0.6701	2770	2.1441	-
0.6705	2772	-	0.6543
0.6725	2780	2.1351	-
0.6749	2790	2.1795	-
0.6773	2800	2.1234	-
0.6797	2810	2.1839	-
0.6807	2814	-	0.6338
0.6821	2820	2.172	-
0.6846	2830	2.1623	-
0.6870	2840	2.1384	-
0.6894	2850	2.1192	-
0.6909	2856	-	0.6365
0.6918	2860	2.1299	-
0.6942	2870	2.0923	-
0.6967	2880	2.1235	-
0.6991	2890	2.1251	-
0.7010	2898	-	0.6314
0.7015	2900	2.0977	-
0.7039	2910	2.1185	-
0.7063	2920	2.1247	-
0.7088	2930	2.1402	-
0.7112	2940	2.147	0.6360
0.7136	2950	2.0964	-
0.7160	2960	2.0909	-
0.7184	2970	2.1207	-
0.7209	2980	2.168	-
0.7213	2982	-	0.6326
0.7233	2990	2.0948	-
0.7257	3000	2.0667	-
0.7281	3010	2.1356	-
0.7305	3020	2.1777	-
0.7315	3024	-	0.6356
0.7329	3030	2.1407	-
0.7354	3040	2.1987	-
0.7378	3050	2.1879	-
0.7402	3060	2.1648	-
0.7417	3066	-	0.6207
0.7426	3070	2.16	-
0.7450	3080	2.106	-
0.7475	3090	2.1453	-
0.7499	3100	2.1011	-
0.7518	3108	-	0.6296
0.7523	3110	2.104	-
0.7547	3120	2.0936	-
0.7571	3130	2.0504	-
0.7596	3140	2.0462	-
0.7620	3150	2.0193	0.6334
0.7644	3160	2.0453	-
0.7668	3170	2.0954	-
0.7692	3180	2.1092	-
0.7716	3190	2.1209	-
0.7721	3192	-	0.6383
0.7741	3200	2.1385	-
0.7765	3210	2.0488	-
0.7789	3220	2.0748	-
0.7813	3230	2.0723	-
0.7823	3234	-	0.6196
0.7837	3240	2.0578	-
0.7862	3250	2.0708	-
0.7886	3260	2.086	-
0.7910	3270	2.0504	-
0.7925	3276	-	0.6171
0.7934	3280	2.0338	-
0.7958	3290	2.0754	-
0.7983	3300	2.0353	-
0.8007	3310	2.0709	-
0.8026	3318	-	0.6091
0.8031	3320	2.0502	-
0.8055	3330	2.0541	-
0.8079	3340	2.0509	-
0.8104	3350	1.9828	-
0.8128	3360	2.0352	0.6119
0.8152	3370	2.0217	-
0.8176	3380	2.0101	-
0.8200	3390	2.0851	-
0.8224	3400	2.0473	-
0.8229	3402	-	0.6128
0.8249	3410	2.0676	-
0.8273	3420	2.0666	-
0.8297	3430	2.0612	-
0.8321	3440	2.0778	-
0.8331	3444	-	0.6213
0.8345	3450	2.0755	-
0.8370	3460	2.0443	-
0.8394	3470	2.039	-
0.8418	3480	1.9625	-
0.8433	3486	-	0.6093
0.8442	3490	2.0408	-
0.8466	3500	2.0795	-
0.8491	3510	2.0077	-
0.8515	3520	2.0181	-
0.8534	3528	-	0.6073
0.8539	3530	2.0182	-
0.8563	3540	2.0174	-
0.8587	3550	2.0551	-
0.8612	3560	2.0427	-
0.8636	3570	1.9981	0.6042
0.8660	3580	2.0657	-
0.8684	3590	2.0465	-
0.8708	3600	2.0011	-
0.8732	3610	2.0421	-
0.8737	3612	-	0.6024
0.8757	3620	1.9901	-
0.8781	3630	1.6599	-
0.8805	3640	1.5043	-
0.8829	3650	1.593	-
0.8839	3654	-	0.5947
0.8853	3660	1.7022	-
0.8878	3670	1.8429	-
0.8902	3680	1.8807	-
0.8926	3690	1.9801	-
0.8940	3696	-	0.6562
0.8950	3700	1.9469	-
0.8974	3710	1.9986	-
0.8999	3720	1.9962	-
0.9023	3730	1.9762	-
0.9042	3738	-	0.7028
0.9047	3740	1.9579	-
0.9071	3750	1.9973	-
0.9095	3760	1.9966	-
0.9119	3770	1.8911	-
0.9144	3780	1.967	0.7325
0.9168	3790	1.9749	-
0.9192	3800	2.0358	-
0.9216	3810	1.9572	-
0.9240	3820	1.9778	-
0.9245	3822	-	0.7554
0.9265	3830	1.9152	-
0.9289	3840	1.9637	-
0.9313	3850	1.9514	-
0.9337	3860	1.986	-
0.9347	3864	-	0.7732
0.9361	3870	2.0145	-
0.9386	3880	1.8994	-
0.9410	3890	1.9492	-
0.9434	3900	1.9374	-
0.9448	3906	-	0.7844
0.9458	3910	1.9583	-
0.9482	3920	1.9123	-
0.9507	3930	1.9629	-
0.9531	3940	1.92	-
0.9550	3948	-	0.7910
0.9555	3950	2.0428	-
0.9579	3960	1.9489	-
0.9603	3970	1.9754	-
0.9627	3980	1.9748	-
0.9652	3990	1.8946	0.7957
0.9676	4000	1.9238	-
0.9700	4010	1.9004	-
0.9724	4020	1.8867	-
0.9748	4030	1.8482	-
0.9753	4032	-	0.8006
0.9773	4040	1.919	-
0.9797	4050	2.0569	-
0.9821	4060	1.986	-
0.9845	4070	1.9729	-
0.9855	4074	-	0.8030
0.9869	4080	1.8762	-
0.9894	4090	1.9108	-
0.9918	4100	1.9484	-
0.9942	4110	1.9913	-
0.9956	4116	-	0.8033
0.9966	4120	1.8677	-
0.9990	4130	1.963	-

Framework Versions

• Python: 3.11.8
• Sentence Transformers: 3.1.1
• Transformers: 4.45.1
• PyTorch: 2.4.0+cu121
• Accelerate: 0.34.2
• Datasets: 3.0.0
• Tokenizers: 0.20.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MaskedCachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}