--- base_model: BAAI/bge-base-en-v1.5 datasets: [] language: - en library_name: sentence-transformers license: apache-2.0 metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 pipeline_tag: sentence-similarity tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:1000 - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss widget: - source_sentence: 'Revision stage: Edit the output to correct content unsupported by evidence while preserving the original content as much as possible. Initialize the revised text $y=x$. (1) Per $(q_i, e_{ij})$, an agreement model (via few-shot prompting + CoT, $(y, q, e) \to {0,1}$) checks whether the evidence $e_i$ disagrees with the current revised text $y$. (2) Only if a disagreement is detect, the edit model (via few-shot prompting + CoT, $(y, q, e) \to \text{ new }y$) outputs a new version of $y$ that aims to agree with evidence $e_{ij}$ while otherwise minimally altering $y$. (3) Finally only a limited number $M=5$ of evidence goes into the attribution report $A$. Fig. 12. Illustration of RARR (Retrofit Attribution using Research and Revision). (Image source: Gao et al. 2022) When evaluating the revised text $y$, both attribution and preservation metrics matter.' sentences: - What is the impact of claim extraction on the efficiency of query generation within various tool querying methodologies? - What are the implications of integrating both attribution and preservation metrics in the assessment of a revised text for an attribution report? - What impact does the calibration of large language models, as discussed in the research by Kadavath et al. (2022), have on the consistency and accuracy of their responses, particularly in the context of multiple choice questions? - source_sentence: 'Fig. 1. Knowledge categorization of close-book QA examples based on how likely the model outputs correct answers. (Image source: Gekhman et al. 2024) Some interesting observations of the experiments, where dev set accuracy is considered a proxy for hallucinations. Unknown examples are fitted substantially slower than Known. The best dev performance is obtained when the LLM fits the majority of the Known training examples but only a few of the Unknown ones. The model starts to hallucinate when it learns most of the Unknown examples. Among Known examples, MaybeKnown cases result in better overall performance, more essential than HighlyKnown ones.' sentences: - What are the implications of a language model's performance when it is primarily trained on familiar examples compared to a diverse set of unfamiliar examples, and how does this relate to the phenomenon of hallucinations in language models? - How can the insights gained from the evaluation framework inform the future enhancements of AI models, particularly in terms of improving factual accuracy and entity recognition? - What role does the MPNet model play in evaluating the faithfulness of reasoning paths, particularly in relation to scores of entailment and contradiction? - source_sentence: 'Non-context LLM: Prompt LLM directly with True or False? without additional context. Retrieval→LLM: Prompt with $k$ related passages retrieved from the knowledge source as context. Nonparametric probability (NP)): Compute the average likelihood of tokens in the atomic fact by a masked LM and use that to make a prediction. Retrieval→LLM + NP: Ensemble of two methods. Some interesting observations on model hallucination behavior: Error rates are higher for rarer entities in the task of biography generation. Error rates are higher for facts mentioned later in the generation. Using retrieval to ground the model generation significantly helps reduce hallucination.' sentences: - What methods does the model employ to generate impactful, non-standard verification questions that enhance the fact-checking process? - What impact does the timing of fact presentation in AI outputs have on the likelihood of generating inaccuracies? - What are the benefits of using the 'Factor+revise' strategy in enhancing the reliability of verification processes in few-shot learning, particularly when it comes to identifying inconsistencies? - source_sentence: 'Research stage: Find related documents as evidence. (1) First use a query generation model (via few-shot prompting, $x \to {q_1, \dots, q_N}$) to construct a set of search queries ${q_1, \dots, q_N}$ to verify all aspects of each sentence. (2) Run Google search, $K=5$ results per query $q_i$. (3) Utilize a pretrained query-document relevance model to assign relevance scores and only retain one most relevant $J=1$ document $e_{i1}, \dots, e_{iJ}$ per query $q_i$. Revision stage: Edit the output to correct content unsupported by evidence while preserving the original content as much as possible. Initialize the revised text $y=x$.' sentences: - In what ways does the process of generating queries facilitate the verification of content accuracy, particularly through the lens of evidence-based editing methodologies? - What role do attribution and preservation metrics play in assessing the quality of revised texts, and how might these factors influence the success of the Evidence Disagreement Detection process? - What are the practical ways to utilize the F1 @ K metric for assessing how well FacTool identifies factual inaccuracies in various fields? - source_sentence: '(1) Joint: join with step 2, where the few-shot examples are structured as (response, verification questions, verification answers); The drawback is that the original response is in the context, so the model may repeat similar hallucination. (2) 2-step: separate the verification planning and execution steps, such as the original response doesn’t impact (3) Factored: each verification question is answered separately. Say, if a long-form base generation results in multiple verification questions, we would answer each question one-by-one. (4) Factor+revise: adding a “cross-checking” step after factored verification execution, conditioned on both the baseline response and the verification question and answer. It detects inconsistency. Final output: Generate the final, refined output. The output gets revised at this step if any inconsistency is discovered.' sentences: - What are the key challenges associated with using a pre-training dataset for world knowledge, particularly in maintaining the factual accuracy of the outputs generated by the model? - What obstacles arise when depending on the pre-training dataset in the context of extrinsic hallucination affecting model outputs? - In what ways does the 'Factor+revise' method enhance the reliability of responses when compared to the 'Joint' and '2-step' methods used for cross-checking? model-index: - name: BGE base Financial Matryoshka results: - task: type: information-retrieval name: Information Retrieval dataset: name: dim 768 type: dim_768 metrics: - type: cosine_accuracy@1 value: 0.8802083333333334 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.984375 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9947916666666666 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9947916666666666 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8802083333333334 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.328125 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19895833333333335 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09947916666666667 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.8802083333333334 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.984375 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9947916666666666 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9947916666666666 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9495062223081544 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.9337673611111109 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.934240845959596 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 512 type: dim_512 metrics: - type: cosine_accuracy@1 value: 0.8854166666666666 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.984375 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9947916666666666 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 1.0 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8854166666666666 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.328125 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19895833333333335 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09999999999999999 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.8854166666666666 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.984375 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9947916666666666 name: Cosine Recall@5 - type: cosine_recall@10 value: 1.0 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9536782535355709 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.937818287037037 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.937818287037037 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 256 type: dim_256 metrics: - type: cosine_accuracy@1 value: 0.9010416666666666 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.984375 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 1.0 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 1.0 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.9010416666666666 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.328125 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19999999999999998 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09999999999999999 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.9010416666666666 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.984375 name: Cosine Recall@3 - type: cosine_recall@5 value: 1.0 name: Cosine Recall@5 - type: cosine_recall@10 value: 1.0 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9587563670488631 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.9446180555555554 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.9446180555555556 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 128 type: dim_128 metrics: - type: cosine_accuracy@1 value: 0.90625 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.984375 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 1.0 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 1.0 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.90625 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.328125 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19999999999999998 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09999999999999999 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.90625 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.984375 name: Cosine Recall@3 - type: cosine_recall@5 value: 1.0 name: Cosine Recall@5 - type: cosine_recall@10 value: 1.0 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9609068566179642 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.9474826388888888 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.947482638888889 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 64 type: dim_64 metrics: - type: cosine_accuracy@1 value: 0.890625 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.984375 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 1.0 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 1.0 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.890625 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.328125 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.19999999999999998 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09999999999999999 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.890625 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.984375 name: Cosine Recall@3 - type: cosine_recall@5 value: 1.0 name: Cosine Recall@5 - type: cosine_recall@10 value: 1.0 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9551401340175182 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.9396701388888888 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.939670138888889 name: Cosine Map@100 --- # BGE base Financial Matryoshka This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-base-en-v1.5](https://huggingface.co./BAAI/bge-base-en-v1.5). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [BAAI/bge-base-en-v1.5](https://huggingface.co./BAAI/bge-base-en-v1.5) - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 768 tokens - **Similarity Function:** Cosine Similarity - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co./models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("joshuapb/fine-tuned-matryoshka-1000") # Run inference sentences = [ '(1) Joint: join with step 2, where the few-shot examples are structured as (response, verification questions, verification answers); The drawback is that the original response is in the context, so the model may repeat similar hallucination.\n(2) 2-step: separate the verification planning and execution steps, such as the original response doesn’t impact\n(3) Factored: each verification question is answered separately. Say, if a long-form base generation results in multiple verification questions, we would answer each question one-by-one.\n(4) Factor+revise: adding a “cross-checking” step after factored verification execution, conditioned on both the baseline response and the verification question and answer. It detects inconsistency.\n\n\nFinal output: Generate the final, refined output. The output gets revised at this step if any inconsistency is discovered.', "In what ways does the 'Factor+revise' method enhance the reliability of responses when compared to the 'Joint' and '2-step' methods used for cross-checking?", 'What obstacles arise when depending on the pre-training dataset in the context of extrinsic hallucination affecting model outputs?', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Information Retrieval * Dataset: `dim_768` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.8802 | | cosine_accuracy@3 | 0.9844 | | cosine_accuracy@5 | 0.9948 | | cosine_accuracy@10 | 0.9948 | | cosine_precision@1 | 0.8802 | | cosine_precision@3 | 0.3281 | | cosine_precision@5 | 0.199 | | cosine_precision@10 | 0.0995 | | cosine_recall@1 | 0.8802 | | cosine_recall@3 | 0.9844 | | cosine_recall@5 | 0.9948 | | cosine_recall@10 | 0.9948 | | cosine_ndcg@10 | 0.9495 | | cosine_mrr@10 | 0.9338 | | **cosine_map@100** | **0.9342** | #### Information Retrieval * Dataset: `dim_512` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.8854 | | cosine_accuracy@3 | 0.9844 | | cosine_accuracy@5 | 0.9948 | | cosine_accuracy@10 | 1.0 | | cosine_precision@1 | 0.8854 | | cosine_precision@3 | 0.3281 | | cosine_precision@5 | 0.199 | | cosine_precision@10 | 0.1 | | cosine_recall@1 | 0.8854 | | cosine_recall@3 | 0.9844 | | cosine_recall@5 | 0.9948 | | cosine_recall@10 | 1.0 | | cosine_ndcg@10 | 0.9537 | | cosine_mrr@10 | 0.9378 | | **cosine_map@100** | **0.9378** | #### Information Retrieval * Dataset: `dim_256` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.901 | | cosine_accuracy@3 | 0.9844 | | cosine_accuracy@5 | 1.0 | | cosine_accuracy@10 | 1.0 | | cosine_precision@1 | 0.901 | | cosine_precision@3 | 0.3281 | | cosine_precision@5 | 0.2 | | cosine_precision@10 | 0.1 | | cosine_recall@1 | 0.901 | | cosine_recall@3 | 0.9844 | | cosine_recall@5 | 1.0 | | cosine_recall@10 | 1.0 | | cosine_ndcg@10 | 0.9588 | | cosine_mrr@10 | 0.9446 | | **cosine_map@100** | **0.9446** | #### Information Retrieval * Dataset: `dim_128` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.9062 | | cosine_accuracy@3 | 0.9844 | | cosine_accuracy@5 | 1.0 | | cosine_accuracy@10 | 1.0 | | cosine_precision@1 | 0.9062 | | cosine_precision@3 | 0.3281 | | cosine_precision@5 | 0.2 | | cosine_precision@10 | 0.1 | | cosine_recall@1 | 0.9062 | | cosine_recall@3 | 0.9844 | | cosine_recall@5 | 1.0 | | cosine_recall@10 | 1.0 | | cosine_ndcg@10 | 0.9609 | | cosine_mrr@10 | 0.9475 | | **cosine_map@100** | **0.9475** | #### Information Retrieval * Dataset: `dim_64` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.8906 | | cosine_accuracy@3 | 0.9844 | | cosine_accuracy@5 | 1.0 | | cosine_accuracy@10 | 1.0 | | cosine_precision@1 | 0.8906 | | cosine_precision@3 | 0.3281 | | cosine_precision@5 | 0.2 | | cosine_precision@10 | 0.1 | | cosine_recall@1 | 0.8906 | | cosine_recall@3 | 0.9844 | | cosine_recall@5 | 1.0 | | cosine_recall@10 | 1.0 | | cosine_ndcg@10 | 0.9551 | | cosine_mrr@10 | 0.9397 | | **cosine_map@100** | **0.9397** | ## Training Details ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `per_device_eval_batch_size`: 16 - `learning_rate`: 2e-05 - `num_train_epochs`: 5 - `lr_scheduler_type`: cosine - `warmup_ratio`: 0.1 - `load_best_model_at_end`: True #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 8 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 5 - `max_steps`: -1 - `lr_scheduler_type`: cosine - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: False - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: proportional
### Training Logs
Click to expand | Epoch | Step | Training Loss | dim_128_cosine_map@100 | dim_256_cosine_map@100 | dim_512_cosine_map@100 | dim_64_cosine_map@100 | dim_768_cosine_map@100 | |:-------:|:-------:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|:----------------------:| | 0.04 | 5 | 4.9678 | - | - | - | - | - | | 0.08 | 10 | 4.6482 | - | - | - | - | - | | 0.12 | 15 | 5.0735 | - | - | - | - | - | | 0.16 | 20 | 4.0336 | - | - | - | - | - | | 0.2 | 25 | 3.7572 | - | - | - | - | - | | 0.24 | 30 | 4.3054 | - | - | - | - | - | | 0.28 | 35 | 2.6705 | - | - | - | - | - | | 0.32 | 40 | 3.1929 | - | - | - | - | - | | 0.36 | 45 | 3.1139 | - | - | - | - | - | | 0.4 | 50 | 2.5219 | - | - | - | - | - | | 0.44 | 55 | 3.1847 | - | - | - | - | - | | 0.48 | 60 | 2.2306 | - | - | - | - | - | | 0.52 | 65 | 2.251 | - | - | - | - | - | | 0.56 | 70 | 2.2432 | - | - | - | - | - | | 0.6 | 75 | 2.7462 | - | - | - | - | - | | 0.64 | 80 | 2.9992 | - | - | - | - | - | | 0.68 | 85 | 2.338 | - | - | - | - | - | | 0.72 | 90 | 2.0169 | - | - | - | - | - | | 0.76 | 95 | 1.257 | - | - | - | - | - | | 0.8 | 100 | 1.5015 | - | - | - | - | - | | 0.84 | 105 | 1.9198 | - | - | - | - | - | | 0.88 | 110 | 2.2154 | - | - | - | - | - | | 0.92 | 115 | 2.4026 | - | - | - | - | - | | 0.96 | 120 | 1.911 | - | - | - | - | - | | 1.0 | 125 | 2.079 | 0.9151 | 0.9098 | 0.9220 | 0.8788 | 0.9251 | | 1.04 | 130 | 1.4704 | - | - | - | - | - | | 1.08 | 135 | 0.7323 | - | - | - | - | - | | 1.12 | 140 | 0.6308 | - | - | - | - | - | | 1.16 | 145 | 0.4655 | - | - | - | - | - | | 1.2 | 150 | 1.0186 | - | - | - | - | - | | 1.24 | 155 | 1.1408 | - | - | - | - | - | | 1.28 | 160 | 1.965 | - | - | - | - | - | | 1.32 | 165 | 1.5987 | - | - | - | - | - | | 1.3600 | 170 | 3.288 | - | - | - | - | - | | 1.4 | 175 | 1.632 | - | - | - | - | - | | 1.44 | 180 | 1.0376 | - | - | - | - | - | | 1.48 | 185 | 0.9466 | - | - | - | - | - | | 1.52 | 190 | 1.0106 | - | - | - | - | - | | 1.56 | 195 | 1.4875 | - | - | - | - | - | | 1.6 | 200 | 1.314 | - | - | - | - | - | | 1.6400 | 205 | 1.3022 | - | - | - | - | - | | 1.6800 | 210 | 1.5312 | - | - | - | - | - | | 1.72 | 215 | 1.7982 | - | - | - | - | - | | 1.76 | 220 | 1.7962 | - | - | - | - | - | | 1.8 | 225 | 1.5788 | - | - | - | - | - | | 1.8400 | 230 | 1.152 | - | - | - | - | - | | 1.88 | 235 | 2.0556 | - | - | - | - | - | | 1.92 | 240 | 1.3165 | - | - | - | - | - | | 1.96 | 245 | 0.6941 | - | - | - | - | - | | **2.0** | **250** | **1.2239** | **0.9404** | **0.944** | **0.9427** | **0.9327** | **0.9424** | | 2.04 | 255 | 1.0423 | - | - | - | - | - | | 2.08 | 260 | 0.8893 | - | - | - | - | - | | 2.12 | 265 | 1.2859 | - | - | - | - | - | | 2.16 | 270 | 1.4505 | - | - | - | - | - | | 2.2 | 275 | 0.2728 | - | - | - | - | - | | 2.24 | 280 | 0.6588 | - | - | - | - | - | | 2.2800 | 285 | 0.8014 | - | - | - | - | - | | 2.32 | 290 | 0.3053 | - | - | - | - | - | | 2.36 | 295 | 1.4289 | - | - | - | - | - | | 2.4 | 300 | 1.1458 | - | - | - | - | - | | 2.44 | 305 | 0.6987 | - | - | - | - | - | | 2.48 | 310 | 1.3389 | - | - | - | - | - | | 2.52 | 315 | 1.2991 | - | - | - | - | - | | 2.56 | 320 | 1.8088 | - | - | - | - | - | | 2.6 | 325 | 0.4242 | - | - | - | - | - | | 2.64 | 330 | 1.5873 | - | - | - | - | - | | 2.68 | 335 | 1.3873 | - | - | - | - | - | | 2.7200 | 340 | 1.4297 | - | - | - | - | - | | 2.76 | 345 | 2.0637 | - | - | - | - | - | | 2.8 | 350 | 1.1252 | - | - | - | - | - | | 2.84 | 355 | 0.367 | - | - | - | - | - | | 2.88 | 360 | 1.7606 | - | - | - | - | - | | 2.92 | 365 | 1.196 | - | - | - | - | - | | 2.96 | 370 | 1.8827 | - | - | - | - | - | | 3.0 | 375 | 0.6822 | 0.9494 | 0.9479 | 0.9336 | 0.9414 | 0.9405 | | 3.04 | 380 | 0.4954 | - | - | - | - | - | | 3.08 | 385 | 0.1717 | - | - | - | - | - | | 3.12 | 390 | 0.7435 | - | - | - | - | - | | 3.16 | 395 | 1.4323 | - | - | - | - | - | | 3.2 | 400 | 1.1207 | - | - | - | - | - | | 3.24 | 405 | 1.9009 | - | - | - | - | - | | 3.2800 | 410 | 1.6706 | - | - | - | - | - | | 3.32 | 415 | 0.8378 | - | - | - | - | - | | 3.36 | 420 | 1.0911 | - | - | - | - | - | | 3.4 | 425 | 0.6565 | - | - | - | - | - | | 3.44 | 430 | 1.0302 | - | - | - | - | - | | 3.48 | 435 | 0.6425 | - | - | - | - | - | | 3.52 | 440 | 1.1472 | - | - | - | - | - | | 3.56 | 445 | 1.996 | - | - | - | - | - | | 3.6 | 450 | 1.5308 | - | - | - | - | - | | 3.64 | 455 | 0.7427 | - | - | - | - | - | | 3.68 | 460 | 1.4596 | - | - | - | - | - | | 3.7200 | 465 | 1.1984 | - | - | - | - | - | | 3.76 | 470 | 0.7601 | - | - | - | - | - | | 3.8 | 475 | 1.3544 | - | - | - | - | - | | 3.84 | 480 | 1.6655 | - | - | - | - | - | | 3.88 | 485 | 1.2596 | - | - | - | - | - | | 3.92 | 490 | 0.9451 | - | - | - | - | - | | 3.96 | 495 | 0.7079 | - | - | - | - | - | | 4.0 | 500 | 1.3471 | 0.9453 | 0.9446 | 0.9404 | 0.9371 | 0.9335 | | 4.04 | 505 | 0.4583 | - | - | - | - | - | | 4.08 | 510 | 1.288 | - | - | - | - | - | | 4.12 | 515 | 1.6946 | - | - | - | - | - | | 4.16 | 520 | 1.1239 | - | - | - | - | - | | 4.2 | 525 | 1.1026 | - | - | - | - | - | | 4.24 | 530 | 1.4121 | - | - | - | - | - | | 4.28 | 535 | 1.7113 | - | - | - | - | - | | 4.32 | 540 | 0.8389 | - | - | - | - | - | | 4.36 | 545 | 0.3117 | - | - | - | - | - | | 4.4 | 550 | 0.3144 | - | - | - | - | - | | 4.44 | 555 | 1.4694 | - | - | - | - | - | | 4.48 | 560 | 1.3233 | - | - | - | - | - | | 4.52 | 565 | 0.792 | - | - | - | - | - | | 4.5600 | 570 | 0.4881 | - | - | - | - | - | | 4.6 | 575 | 0.5097 | - | - | - | - | - | | 4.64 | 580 | 1.6377 | - | - | - | - | - | | 4.68 | 585 | 0.7273 | - | - | - | - | - | | 4.72 | 590 | 1.5464 | - | - | - | - | - | | 4.76 | 595 | 1.4392 | - | - | - | - | - | | 4.8 | 600 | 1.4384 | - | - | - | - | - | | 4.84 | 605 | 0.6375 | - | - | - | - | - | | 4.88 | 610 | 1.0528 | - | - | - | - | - | | 4.92 | 615 | 0.0276 | - | - | - | - | - | | 4.96 | 620 | 0.9604 | - | - | - | - | - | | 5.0 | 625 | 0.7219 | 0.9475 | 0.9446 | 0.9378 | 0.9397 | 0.9342 | * The bold row denotes the saved checkpoint.
### Framework Versions - Python: 3.10.12 - Sentence Transformers: 3.0.1 - Transformers: 4.42.4 - PyTorch: 2.3.1+cu121 - Accelerate: 0.32.1 - Datasets: 2.21.0 - Tokenizers: 0.19.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MatryoshkaLoss ```bibtex @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```