README.md

---
base_model: BAAI/bge-base-en-v1.5
datasets: []
language:
- en
library_name: sentence-transformers
license: apache-2.0
metrics:
- cosine_accuracy@1
- cosine_accuracy@3
- cosine_accuracy@5
- cosine_accuracy@10
- cosine_precision@1
- cosine_precision@3
- cosine_precision@5
- cosine_precision@10
- cosine_recall@1
- cosine_recall@3
- cosine_recall@5
- cosine_recall@10
- cosine_ndcg@10
- cosine_mrr@10
- cosine_map@100
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:200
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
widget:
- source_sentence: 'Verbalized number or word (e.g. “lowest”, “low”, “medium”, “high”,
    “highest”), such as "Confidence: 60% / Medium".

    Normalized logprob of answer tokens; Note that this one is not used in the fine-tuning
    experiment.

    Logprob of an indirect "True/False" token after the raw answer.

    Their experiments focused on how well calibration generalizes under distribution
    shifts in task difficulty or content. Each fine-tuning datapoint is a question,
    the model’s answer (possibly incorrect), and a calibrated confidence. Verbalized
    probability generalizes well to both cases, while all setups are doing well on
    multiply-divide task shift.  Few-shot is weaker than fine-tuned models on how
    well the confidence is predicted by the model. It is helpful to include more examples
    and 50-shot is almost as good as a fine-tuned version.'
  sentences:
  - What is the relationship between the calibration of AI models and the effectiveness
    of verbalized probabilities when applied to tasks of varying difficulty levels?
  - In what ways does the F1 @ K metric contribute to evaluating the factual accuracy
    and comprehensiveness of outputs generated by long-form language models?
  - What impact does the implementation of a pretrained query-document relevance model
    have on the process of document selection in research methodologies?
- source_sentence: 'Fig. 4. Overview of SAFE for factuality evaluation of long-form
    LLM generation. (Image source: Wei et al. 2024)

    The SAFE evaluation metric is F1 @ K. The motivation is that model response for
    long-form factuality should ideally hit both precision and recall, as the response
    should be both


    factual : measured by precision, the percentage of supported facts among all facts
    in the entire response.

    long : measured by recall, the percentage of provided facts among all relevant
    facts that should appear in the response. Therefore we want to consider the number
    of supported facts up to $K$.


    Given the model response $y$, the metric F1 @ K is defined as:'
  sentences:
  - What methodologies does the agreement model employ to identify discrepancies between
    the original and revised text, and how do these methodologies impact the overall
    editing workflow?
  - In what ways does the SAFE evaluation metric achieve a harmonious equilibrium
    between precision and recall in the context of evaluating the factual accuracy
    of long-form outputs generated by large language models?
  - In what ways does the inherently adversarial structure of TruthfulQA inquiries
    facilitate the detection of prevalent fallacies in human cognitive processes,
    and what implications does this have for understanding the constraints of expansive
    language models?
- source_sentence: 'Non-context LLM: Prompt LLM directly with <atomic-fact> True or
    False? without additional context.

    Retrieval→LLM: Prompt with $k$ related passages retrieved from the knowledge source
    as context.

    Nonparametric probability (NP)): Compute the average likelihood of tokens in the
    atomic fact by a masked LM and use that to make a prediction.

    Retrieval→LLM + NP: Ensemble of two methods.


    Some interesting observations on model hallucination behavior:


    Error rates are higher for rarer entities in the task of biography generation.

    Error rates are higher for facts mentioned later in the generation.

    Using retrieval to ground the model generation significantly helps reduce hallucination.'
  sentences:
  - In what ways does the Rethinking with Retrieval (RR) methodology leverage Chain-of-Thought
    (CoT) prompting to enhance the efficacy of external knowledge retrieval, and what
    implications does this have for the precision of predictive outcomes generated
    by models?
  - In what ways does the retrieval of related passages contribute to minimizing hallucinations
    in large language models, and what specific techniques can be employed to evaluate
    the impact of this approach?
  - What are the benefits of using retrieval methods in biography generation to minimize
    inaccuracies, especially when compared to traditional prompting techniques that
    lack context?
- source_sentence: 'Yin et al. (2023) studies the concept of self-knowledge, referring
    to whether language models know what they know or don’t know.

    SelfAware, containing 1,032 unanswerable questions across five categories and
    2,337 answerable questions. Unanswerable questions are sourced from online forums
    with human annotations while answerable questions are sourced from SQuAD, HotpotQA
    and TriviaQA based on text similarity with unanswerable questions. A question
    may be unanswerable due to various reasons, such as no scientific consensus, imaginations
    of the future, completely subjective, philosophical reasons that may yield multiple
    responses, etc. Considering separating answerable vs unanswerable questions as
    a binary classification task, we can measure F1-score or accuracy and the experiments
    showed that larger models can do better at this task.'
  sentences:
  - What is the relationship between model size and performance metrics, such as F1-score
    and accuracy, in the context of classifying questions into answerable and unanswerable
    categories?
  - How does the introduction of stochastic perturbations in synthetic training data
    contribute to the enhancement of editor model efficacy within LangChain frameworks?
  - How do the various output values linked to reflection tokens in the Self-RAG framework
    impact the generation process, and why are they important?
- source_sentence: 'Fig. 1. Knowledge categorization of close-book QA examples based
    on how likely the model outputs correct answers. (Image source: Gekhman et al.
    2024)

    Some interesting observations of the experiments, where dev set accuracy is considered
    a proxy for hallucinations.


    Unknown examples are fitted substantially slower than Known.

    The best dev performance is obtained when the LLM fits the majority of the Known
    training examples but only a few of the Unknown ones. The model starts to hallucinate
    when it learns most of the Unknown examples.

    Among Known examples, MaybeKnown cases result in better overall performance, more
    essential than HighlyKnown ones.'
  sentences:
  - In what ways does the fitting speed of examples that are not previously encountered
    differ from that of familiar examples, and how does this variation influence the
    overall accuracy of the model on the development set?
  - What role do reflection tokens play in enhancing the efficiency of document retrieval
    and generation within the Self-RAG framework?
  - How do the results presented by Gekhman et al. in their 2024 study inform our
    understanding of the reliability metrics associated with large language models
    (LLMs) when subjected to fine-tuning with novel datasets?
model-index:
- name: BGE base Financial Matryoshka
  results:
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 768
      type: dim_768
    metrics:
    - type: cosine_accuracy@1
      value: 0.8802083333333334
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.96875
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.984375
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9947916666666666
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.8802083333333334
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.3229166666666667
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.196875
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09947916666666667
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.8802083333333334
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.96875
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.984375
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9947916666666666
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9433275174124347
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.9261284722222224
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.9264025950292397
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 512
      type: dim_512
    metrics:
    - type: cosine_accuracy@1
      value: 0.8697916666666666
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.9739583333333334
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.9739583333333334
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9947916666666666
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.8697916666666666
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.3246527777777778
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.1947916666666666
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09947916666666667
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.8697916666666666
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.9739583333333334
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.9739583333333334
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9947916666666666
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.939968526552219
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.9216269841269841
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.9220610119047619
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 256
      type: dim_256
    metrics:
    - type: cosine_accuracy@1
      value: 0.8697916666666666
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.9739583333333334
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.984375
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 1.0
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.8697916666666666
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.3246527777777778
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.196875
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09999999999999999
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.8697916666666666
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.9739583333333334
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.984375
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 1.0
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9419747509776967
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.922676917989418
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.922676917989418
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 128
      type: dim_128
    metrics:
    - type: cosine_accuracy@1
      value: 0.8541666666666666
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.9583333333333334
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.96875
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9947916666666666
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.8541666666666666
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.3194444444444445
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.19374999999999998
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09947916666666667
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.8541666666666666
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.9583333333333334
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.96875
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9947916666666666
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9306358745697197
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.9094328703703702
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.9098668981481483
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 64
      type: dim_64
    metrics:
    - type: cosine_accuracy@1
      value: 0.7916666666666666
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.953125
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.9739583333333334
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9895833333333334
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.7916666666666666
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.3177083333333333
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.1947916666666666
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09895833333333333
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.7916666666666666
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.953125
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.9739583333333334
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9895833333333334
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9003914274568845
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.8705935846560847
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.8713150853775854
      name: Cosine Map@100
---

# BGE base Financial Matryoshka

This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-base-en-v1.5](https://huggingface.co./BAAI/bge-base-en-v1.5). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

## Model Details

### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [BAAI/bge-base-en-v1.5](https://huggingface.co./BAAI/bge-base-en-v1.5) <!-- at revision a5beb1e3e68b9ab74eb54cfd186867f64f240e1a -->
- **Maximum Sequence Length:** 512 tokens
- **Output Dimensionality:** 768 tokens
- **Similarity Function:** Cosine Similarity
<!-- - **Training Dataset:** Unknown -->
- **Language:** en
- **License:** apache-2.0

### Model Sources

- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co./models?library=sentence-transformers)

### Full Model Architecture

```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)
```

## Usage

### Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

```bash
pip install -U sentence-transformers
```

Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("joshuapb/fine-tuned-matryoshka-200")
# Run inference
sentences = [
    'Fig. 1. Knowledge categorization of close-book QA examples based on how likely the model outputs correct answers. (Image source: Gekhman et al. 2024)\nSome interesting observations of the experiments, where dev set accuracy is considered a proxy for hallucinations.\n\nUnknown examples are fitted substantially slower than Known.\nThe best dev performance is obtained when the LLM fits the majority of the Known training examples but only a few of the Unknown ones. The model starts to hallucinate when it learns most of the Unknown examples.\nAmong Known examples, MaybeKnown cases result in better overall performance, more essential than HighlyKnown ones.',
    'In what ways does the fitting speed of examples that are not previously encountered differ from that of familiar examples, and how does this variation influence the overall accuracy of the model on the development set?',
    'How do the results presented by Gekhman et al. in their 2024 study inform our understanding of the reliability metrics associated with large language models (LLMs) when subjected to fine-tuning with novel datasets?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```

<!--
### Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details>
-->

<!--
### Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details>
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

## Evaluation

### Metrics

#### Information Retrieval
* Dataset: `dim_768`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.8802     |
| cosine_accuracy@3   | 0.9688     |
| cosine_accuracy@5   | 0.9844     |
| cosine_accuracy@10  | 0.9948     |
| cosine_precision@1  | 0.8802     |
| cosine_precision@3  | 0.3229     |
| cosine_precision@5  | 0.1969     |
| cosine_precision@10 | 0.0995     |
| cosine_recall@1     | 0.8802     |
| cosine_recall@3     | 0.9688     |
| cosine_recall@5     | 0.9844     |
| cosine_recall@10    | 0.9948     |
| cosine_ndcg@10      | 0.9433     |
| cosine_mrr@10       | 0.9261     |
| **cosine_map@100**  | **0.9264** |

#### Information Retrieval
* Dataset: `dim_512`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.8698     |
| cosine_accuracy@3   | 0.974      |
| cosine_accuracy@5   | 0.974      |
| cosine_accuracy@10  | 0.9948     |
| cosine_precision@1  | 0.8698     |
| cosine_precision@3  | 0.3247     |
| cosine_precision@5  | 0.1948     |
| cosine_precision@10 | 0.0995     |
| cosine_recall@1     | 0.8698     |
| cosine_recall@3     | 0.974      |
| cosine_recall@5     | 0.974      |
| cosine_recall@10    | 0.9948     |
| cosine_ndcg@10      | 0.94       |
| cosine_mrr@10       | 0.9216     |
| **cosine_map@100**  | **0.9221** |

#### Information Retrieval
* Dataset: `dim_256`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.8698     |
| cosine_accuracy@3   | 0.974      |
| cosine_accuracy@5   | 0.9844     |
| cosine_accuracy@10  | 1.0        |
| cosine_precision@1  | 0.8698     |
| cosine_precision@3  | 0.3247     |
| cosine_precision@5  | 0.1969     |
| cosine_precision@10 | 0.1        |
| cosine_recall@1     | 0.8698     |
| cosine_recall@3     | 0.974      |
| cosine_recall@5     | 0.9844     |
| cosine_recall@10    | 1.0        |
| cosine_ndcg@10      | 0.942      |
| cosine_mrr@10       | 0.9227     |
| **cosine_map@100**  | **0.9227** |

#### Information Retrieval
* Dataset: `dim_128`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.8542     |
| cosine_accuracy@3   | 0.9583     |
| cosine_accuracy@5   | 0.9688     |
| cosine_accuracy@10  | 0.9948     |
| cosine_precision@1  | 0.8542     |
| cosine_precision@3  | 0.3194     |
| cosine_precision@5  | 0.1937     |
| cosine_precision@10 | 0.0995     |
| cosine_recall@1     | 0.8542     |
| cosine_recall@3     | 0.9583     |
| cosine_recall@5     | 0.9688     |
| cosine_recall@10    | 0.9948     |
| cosine_ndcg@10      | 0.9306     |
| cosine_mrr@10       | 0.9094     |
| **cosine_map@100**  | **0.9099** |

#### Information Retrieval
* Dataset: `dim_64`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.7917     |
| cosine_accuracy@3   | 0.9531     |
| cosine_accuracy@5   | 0.974      |
| cosine_accuracy@10  | 0.9896     |
| cosine_precision@1  | 0.7917     |
| cosine_precision@3  | 0.3177     |
| cosine_precision@5  | 0.1948     |
| cosine_precision@10 | 0.099      |
| cosine_recall@1     | 0.7917     |
| cosine_recall@3     | 0.9531     |
| cosine_recall@5     | 0.974      |
| cosine_recall@10    | 0.9896     |
| cosine_ndcg@10      | 0.9004     |
| cosine_mrr@10       | 0.8706     |
| **cosine_map@100**  | **0.8713** |

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Hyperparameters
#### Non-Default Hyperparameters

- `eval_strategy`: epoch
- `per_device_eval_batch_size`: 16
- `learning_rate`: 2e-05
- `num_train_epochs`: 5
- `lr_scheduler_type`: cosine
- `warmup_ratio`: 0.1
- `load_best_model_at_end`: True

#### All Hyperparameters
<details><summary>Click to expand</summary>

- `overwrite_output_dir`: False
- `do_predict`: False
- `eval_strategy`: epoch
- `prediction_loss_only`: True
- `per_device_train_batch_size`: 8
- `per_device_eval_batch_size`: 16
- `per_gpu_train_batch_size`: None
- `per_gpu_eval_batch_size`: None
- `gradient_accumulation_steps`: 1
- `eval_accumulation_steps`: None
- `learning_rate`: 2e-05
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `max_grad_norm`: 1.0
- `num_train_epochs`: 5
- `max_steps`: -1
- `lr_scheduler_type`: cosine
- `lr_scheduler_kwargs`: {}
- `warmup_ratio`: 0.1
- `warmup_steps`: 0
- `log_level`: passive
- `log_level_replica`: warning
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `save_safetensors`: True
- `save_on_each_node`: False
- `save_only_model`: False
- `restore_callback_states_from_checkpoint`: False
- `no_cuda`: False
- `use_cpu`: False
- `use_mps_device`: False
- `seed`: 42
- `data_seed`: None
- `jit_mode_eval`: False
- `use_ipex`: False
- `bf16`: False
- `fp16`: False
- `fp16_opt_level`: O1
- `half_precision_backend`: auto
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: None
- `local_rank`: 0
- `ddp_backend`: None
- `tpu_num_cores`: None
- `tpu_metrics_debug`: False
- `debug`: []
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 0
- `dataloader_prefetch_factor`: None
- `past_index`: -1
- `disable_tqdm`: False
- `remove_unused_columns`: True
- `label_names`: None
- `load_best_model_at_end`: True
- `ignore_data_skip`: False
- `fsdp`: []
- `fsdp_min_num_params`: 0
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
- `fsdp_transformer_layer_cls_to_wrap`: None
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `deepspeed`: None
- `label_smoothing_factor`: 0.0
- `optim`: adamw_torch
- `optim_args`: None
- `adafactor`: False
- `group_by_length`: False
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `skip_memory_metrics`: True
- `use_legacy_prediction_loop`: False
- `push_to_hub`: False
- `resume_from_checkpoint`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_private_repo`: False
- `hub_always_push`: False
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `include_inputs_for_metrics`: False
- `eval_do_concat_batches`: True
- `fp16_backend`: auto
- `push_to_hub_model_id`: None
- `push_to_hub_organization`: None
- `mp_parameters`: 
- `auto_find_batch_size`: False
- `full_determinism`: False
- `torchdynamo`: None
- `ray_scope`: last
- `ddp_timeout`: 1800
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `dispatch_batches`: None
- `split_batches`: None
- `include_tokens_per_second`: False
- `include_num_input_tokens_seen`: False
- `neftune_noise_alpha`: None
- `optim_target_modules`: None
- `batch_eval_metrics`: False
- `eval_on_start`: False
- `batch_sampler`: batch_sampler
- `multi_dataset_batch_sampler`: proportional

</details>

### Training Logs
| Epoch   | Step    | Training Loss | dim_128_cosine_map@100 | dim_256_cosine_map@100 | dim_512_cosine_map@100 | dim_64_cosine_map@100 | dim_768_cosine_map@100 |
|:-------:|:-------:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|:----------------------:|
| 0.2     | 5       | 5.2225        | -                      | -                      | -                      | -                     | -                      |
| 0.4     | 10      | 4.956         | -                      | -                      | -                      | -                     | -                      |
| 0.6     | 15      | 3.6388        | -                      | -                      | -                      | -                     | -                      |
| 0.8     | 20      | 3.1957        | -                      | -                      | -                      | -                     | -                      |
| 1.0     | 25      | 2.6928        | 0.8661                 | 0.8770                 | 0.8754                 | 0.8312                | 0.8871                 |
| 1.2     | 30      | 2.5565        | -                      | -                      | -                      | -                     | -                      |
| 1.4     | 35      | 1.5885        | -                      | -                      | -                      | -                     | -                      |
| 1.6     | 40      | 2.1406        | -                      | -                      | -                      | -                     | -                      |
| 1.8     | 45      | 2.193         | -                      | -                      | -                      | -                     | -                      |
| 2.0     | 50      | 1.326         | 0.8944                 | 0.9110                 | 0.9028                 | 0.8615                | 0.9037                 |
| 2.2     | 55      | 2.6832        | -                      | -                      | -                      | -                     | -                      |
| 2.4     | 60      | 1.0584        | -                      | -                      | -                      | -                     | -                      |
| 2.6     | 65      | 0.8853        | -                      | -                      | -                      | -                     | -                      |
| 2.8     | 70      | 1.7129        | -                      | -                      | -                      | -                     | -                      |
| 3.0     | 75      | 2.1856        | 0.9106                 | 0.9293                 | 0.9075                 | 0.8778                | 0.9266                 |
| 3.2     | 80      | 1.7658        | -                      | -                      | -                      | -                     | -                      |
| 3.4     | 85      | 1.9783        | -                      | -                      | -                      | -                     | -                      |
| 3.6     | 90      | 1.9583        | -                      | -                      | -                      | -                     | -                      |
| 3.8     | 95      | 1.2396        | -                      | -                      | -                      | -                     | -                      |
| 4.0     | 100     | 1.1901        | 0.9073                 | 0.9253                 | 0.9151                 | 0.8750                | 0.9312                 |
| 4.2     | 105     | 2.6547        | -                      | -                      | -                      | -                     | -                      |
| 4.4     | 110     | 1.3485        | -                      | -                      | -                      | -                     | -                      |
| 4.6     | 115     | 1.0767        | -                      | -                      | -                      | -                     | -                      |
| 4.8     | 120     | 0.6663        | -                      | -                      | -                      | -                     | -                      |
| **5.0** | **125** | **1.3869**    | **0.9099**             | **0.9227**             | **0.9221**             | **0.8713**            | **0.9264**             |

* The bold row denotes the saved checkpoint.

### Framework Versions
- Python: 3.10.12
- Sentence Transformers: 3.0.1
- Transformers: 4.42.4
- PyTorch: 2.3.1+cu121
- Accelerate: 0.32.1
- Datasets: 2.21.0
- Tokenizers: 0.19.1

## Citation

### BibTeX

#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}
```

#### MatryoshkaLoss
```bibtex
@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
```

#### MultipleNegativesRankingLoss
```bibtex
@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->