---
language:
- en
library_name: sentence-transformers
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- loss:Matryoshka2dLoss
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
base_model: distilbert/distilroberta-base
metrics:
- pearson_cosine
- spearman_cosine
- pearson_manhattan
- spearman_manhattan
- pearson_euclidean
- spearman_euclidean
- pearson_dot
- spearman_dot
- pearson_max
- spearman_max
widget:
- source_sentence: A woman is reading.
sentences:
- A woman is writing something.
- A man helps a boy ride a bike.
- A group wading across a ditch
- source_sentence: A man shoots a man.
sentences:
- A man with a pistol shoots another man.
- Suicide bomber strikes in Syria
- China and Taiwan hold historic talks
- source_sentence: A boy is vacuuming.
sentences:
- A little boy is vacuuming the floor.
- 'Breivik: Jail term ''ridiculous'''
- Glorious triple-gold night for Britain
- source_sentence: A man is spitting.
sentences:
- A man is speaking.
- The boy is jumping into a lake.
- 10 Things to Know for Thursday
- source_sentence: A plane in the sky.
sentences:
- Two airplanes in the sky.
- Nelson Mandela undergoes surgery
- Nelson Mandela undergoes surgery
pipeline_tag: sentence-similarity
co2_eq_emissions:
emissions: 69.2573690422145
energy_consumed: 0.1781760038338226
source: codecarbon
training_type: fine-tuning
on_cloud: false
cpu_model: 13th Gen Intel(R) Core(TM) i7-13700K
ram_total_size: 31.777088165283203
hours_used: 0.626
hardware_used: 1 x NVIDIA GeForce RTX 3090
model-index:
- name: SentenceTransformer based on distilbert/distilroberta-base
results:
- task:
type: semantic-similarity
name: Semantic Similarity
dataset:
name: sts dev
type: sts-dev
metrics:
- type: pearson_cosine
value: 0.8395203447657347
name: Pearson Cosine
- type: spearman_cosine
value: 0.8424556124488326
name: Spearman Cosine
- type: pearson_manhattan
value: 0.8432537220190851
name: Pearson Manhattan
- type: spearman_manhattan
value: 0.8435994230515586
name: Spearman Manhattan
- type: pearson_euclidean
value: 0.8440900768179745
name: Pearson Euclidean
- type: spearman_euclidean
value: 0.8449067313707376
name: Spearman Euclidean
- type: pearson_dot
value: 0.763767029856877
name: Pearson Dot
- type: spearman_dot
value: 0.7569706383510251
name: Spearman Dot
- type: pearson_max
value: 0.8440900768179745
name: Pearson Max
- type: spearman_max
value: 0.8449067313707376
name: Spearman Max
- task:
type: semantic-similarity
name: Semantic Similarity
dataset:
name: sts test
type: sts-test
metrics:
- type: pearson_cosine
value: 0.8186702838538092
name: Pearson Cosine
- type: spearman_cosine
value: 0.8170686920551
name: Spearman Cosine
- type: pearson_manhattan
value: 0.8117192659894803
name: Pearson Manhattan
- type: spearman_manhattan
value: 0.804879002947593
name: Spearman Manhattan
- type: pearson_euclidean
value: 0.8127154744140831
name: Pearson Euclidean
- type: spearman_euclidean
value: 0.8058410028545979
name: Spearman Euclidean
- type: pearson_dot
value: 0.7396245702595934
name: Pearson Dot
- type: spearman_dot
value: 0.7256120569318246
name: Spearman Dot
- type: pearson_max
value: 0.8186702838538092
name: Pearson Max
- type: spearman_max
value: 0.8170686920551
name: Spearman Max
---
# SentenceTransformer based on distilbert/distilroberta-base
This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [distilbert/distilroberta-base](https://huggingface.co./distilbert/distilroberta-base) on the [sentence-transformers/all-nli](https://huggingface.co./datasets/sentence-transformers/all-nli) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
## Model Details
### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [distilbert/distilroberta-base](https://huggingface.co./distilbert/distilroberta-base)
- **Maximum Sequence Length:** 512 tokens
- **Output Dimensionality:** 768 tokens
- **Similarity Function:** Cosine Similarity
- **Training Dataset:**
- [sentence-transformers/all-nli](https://huggingface.co./datasets/sentence-transformers/all-nli)
- **Language:** en
### Model Sources
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co./models?library=sentence-transformers)
### Full Model Architecture
```
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
```
## Usage
### Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/distilroberta-base-nli-2d-matryoshka")
# Run inference
sentences = [
'A plane in the sky.',
'Two airplanes in the sky.',
'Nelson Mandela undergoes surgery',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings)
print(similarities.shape)
# [3, 3]
```
## Evaluation
### Metrics
#### Semantic Similarity
* Dataset: `sts-dev`
* Evaluated with [EmbeddingSimilarityEvaluator
](https://sbert.net/docs/package_reference/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)
| Metric | Value |
|:--------------------|:-----------|
| pearson_cosine | 0.8395 |
| **spearman_cosine** | **0.8425** |
| pearson_manhattan | 0.8433 |
| spearman_manhattan | 0.8436 |
| pearson_euclidean | 0.8441 |
| spearman_euclidean | 0.8449 |
| pearson_dot | 0.7638 |
| spearman_dot | 0.757 |
| pearson_max | 0.8441 |
| spearman_max | 0.8449 |
#### Semantic Similarity
* Dataset: `sts-test`
* Evaluated with [EmbeddingSimilarityEvaluator
](https://sbert.net/docs/package_reference/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)
| Metric | Value |
|:--------------------|:-----------|
| pearson_cosine | 0.8187 |
| **spearman_cosine** | **0.8171** |
| pearson_manhattan | 0.8117 |
| spearman_manhattan | 0.8049 |
| pearson_euclidean | 0.8127 |
| spearman_euclidean | 0.8058 |
| pearson_dot | 0.7396 |
| spearman_dot | 0.7256 |
| pearson_max | 0.8187 |
| spearman_max | 0.8171 |
## Training Details
### Training Dataset
#### sentence-transformers/all-nli
* Dataset: [sentence-transformers/all-nli](https://huggingface.co./datasets/sentence-transformers/all-nli) at [65dd388](https://huggingface.co./datasets/sentence-transformers/all-nli/tree/65dd38867b600f42241d2066ba1a35fbd097fcfe)
* Size: 557,850 training samples
* Columns: anchor
, positive
, and negative
* Approximate statistics based on the first 1000 samples:
| | anchor | positive | negative |
|:--------|:----------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|
| type | string | string | string |
| details |
A person on a horse jumps over a broken down airplane.
| A person is outdoors, on a horse.
| A person is at a diner, ordering an omelette.
|
| Children smiling and waving at camera
| There are children present
| The kids are frowning
|
| A boy is jumping on skateboard in the middle of a red bridge.
| The boy does a skateboarding trick.
| The boy skates down the sidewalk.
|
* Loss: [Matryoshka2dLoss
](https://sbert.net/docs/package_reference/losses.html#matryoshka2dloss) with these parameters:
```json
{
"loss": "MultipleNegativesRankingLoss",
"n_layers_per_step": 1,
"last_layer_weight": 1.0,
"prior_layers_weight": 1.0,
"kl_div_weight": 1.0,
"kl_temperature": 0.3,
"matryoshka_dims": [
768,
512,
256,
128,
64
],
"matryoshka_weights": [
1,
1,
1,
1,
1
],
"n_dims_per_step": 1
}
```
### Evaluation Dataset
#### sentence-transformers/stsb
* Dataset: [sentence-transformers/stsb](https://huggingface.co./datasets/sentence-transformers/stsb) at [ab7a5ac](https://huggingface.co./datasets/sentence-transformers/stsb/tree/ab7a5ac0e35aa22088bdcf23e7fd99b220e53308)
* Size: 1,500 evaluation samples
* Columns: sentence1
, sentence2
, and score
* Approximate statistics based on the first 1000 samples:
| | sentence1 | sentence2 | score |
|:--------|:---------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:---------------------------------------------------------------|
| type | string | string | float |
| details | A man with a hard hat is dancing.
| A man wearing a hard hat is dancing.
| 1.0
|
| A young child is riding a horse.
| A child is riding a horse.
| 0.95
|
| A man is feeding a mouse to a snake.
| The man is feeding a mouse to the snake.
| 1.0
|
* Loss: [Matryoshka2dLoss
](https://sbert.net/docs/package_reference/losses.html#matryoshka2dloss) with these parameters:
```json
{
"loss": "MultipleNegativesRankingLoss",
"n_layers_per_step": 1,
"last_layer_weight": 1.0,
"prior_layers_weight": 1.0,
"kl_div_weight": 1.0,
"kl_temperature": 0.3,
"matryoshka_dims": [
768,
512,
256,
128,
64
],
"matryoshka_weights": [
1,
1,
1,
1,
1
],
"n_dims_per_step": 1
}
```
### Training Hyperparameters
#### Non-Default Hyperparameters
- `eval_strategy`: steps
- `per_device_train_batch_size`: 128
- `per_device_eval_batch_size`: 128
- `num_train_epochs`: 1
- `warmup_ratio`: 0.1
- `fp16`: True
- `batch_sampler`: no_duplicates
#### All Hyperparameters