MMTEB: Massive Multilingual Text Embedding Benchmark
Paper
ā¢
2502.13595
ā¢
Published
ā¢
31
Sentence Transformers - Cross-Encoders
university
AI & ML interests
This repository hosts the cross-encoders from the SentenceTransformers package. More details on https://www.sbert.net/docs/pretrained_cross-encoders.html
Recent Activity
cross-encoder's activity
tomaarsenĀ
authored
a
paper
7 days ago
Post
2046
I just released Sentence Transformers v3.4.0, featuring a memory leak fix, compatibility between the powerful Cached... losses and the Matryoshka loss modifier, and a bunch of fixes & small features.
šŖ Matryoshka & Cached loss compatibility
It is now possible to combine the powerful Cached... losses (which use in-batch negatives & a caching mechanism to allow for endless batch size & negatives) with the Matryoshka loss modifier which modifies a base loss such that it is trained not only on the maximum dimensionality (e.g. 1024 dimensions), but also on many lower dimensions (e.g. 768, 512, 256, 128, 64, 32).
After training, these models' embeddings can be truncated for faster retrieval, etc.
šļø Resolve memory leak when Model and Trainer are reinitialized
Due to a circular dependency between Trainer -> Model -> ModelCardData -> Trainer, deleting both the trainer & model still didn't free up the memory.
This led to a memory leak in scripts where you repeatedly do so.
ā New Features
Many new small features, e.g. multi-GPU support for 'mine_hard_negatives', a 'margin' parameter to TripletEvaluator, and Matthews Correlation Coefficient in the BinaryClassificationEvaluator.
š Bug Fixes
Also a bunch of fixes, for example that subsequent batches were not sorted when using the "no_duplicates" batch sampler. See the release notes for more details.
Full release notes: https://github.com/UKPLab/sentence-transformers/releases/tag/v3.4.0
Big thanks to all community members who assisted in this release. 10 folks with their first contribution this time around!
šŖ Matryoshka & Cached loss compatibility
It is now possible to combine the powerful Cached... losses (which use in-batch negatives & a caching mechanism to allow for endless batch size & negatives) with the Matryoshka loss modifier which modifies a base loss such that it is trained not only on the maximum dimensionality (e.g. 1024 dimensions), but also on many lower dimensions (e.g. 768, 512, 256, 128, 64, 32).
After training, these models' embeddings can be truncated for faster retrieval, etc.
šļø Resolve memory leak when Model and Trainer are reinitialized
Due to a circular dependency between Trainer -> Model -> ModelCardData -> Trainer, deleting both the trainer & model still didn't free up the memory.
This led to a memory leak in scripts where you repeatedly do so.
ā New Features
Many new small features, e.g. multi-GPU support for 'mine_hard_negatives', a 'margin' parameter to TripletEvaluator, and Matthews Correlation Coefficient in the BinaryClassificationEvaluator.
š Bug Fixes
Also a bunch of fixes, for example that subsequent batches were not sorted when using the "no_duplicates" batch sampler. See the release notes for more details.
Full release notes: https://github.com/UKPLab/sentence-transformers/releases/tag/v3.4.0
Big thanks to all community members who assisted in this release. 10 folks with their first contribution this time around!
Post
4630
šļø Today I'm introducing a method to train static embedding models that run 100x to 400x faster on CPU than common embedding models, while retaining 85%+ of the quality! Including 2 fully open models: training scripts, datasets, metrics.
We apply our recipe to train 2 Static Embedding models that we release today! We release:
2ļøā£ an English Retrieval model and a general-purpose Multilingual similarity model (e.g. classification, clustering, etc.), both Apache 2.0
š§ my modern training strategy: ideation -> dataset choice -> implementation -> evaluation
š my training scripts, using the Sentence Transformers library
š my Weights & Biases reports with losses & metrics
š my list of 30 training and 13 evaluation datasets
The 2 Static Embedding models have the following properties:
šļø Extremely fast, e.g. 107500 sentences per second on a consumer CPU, compared to 270 for 'all-mpnet-base-v2' and 56 for 'gte-large-en-v1.5'
0ļøā£ Zero active parameters: No Transformer blocks, no attention, not even a matrix multiplication. Super speed!
š No maximum sequence length! Embed texts at any length (note: longer texts may embed worse)
š Linear instead of exponential complexity: 2x longer text takes 2x longer, instead of 2.5x or more.
šŖ Matryoshka support: allow you to truncate embeddings with minimal performance loss (e.g. 4x smaller with a 0.56% perf. decrease for English Similarity tasks)
Check out the full blogpost if you'd like to 1) use these lightning-fast models or 2) learn how to train them with consumer-level hardware: https://huggingface.co./blog/static-embeddings
The blogpost contains a lengthy list of possible advancements; I'm very confident that our 2 models are only the tip of the iceberg, and we may be able to get even better performance.
Alternatively, check out the models:
* sentence-transformers/static-retrieval-mrl-en-v1
* sentence-transformers/static-similarity-mrl-multilingual-v1
We apply our recipe to train 2 Static Embedding models that we release today! We release:
2ļøā£ an English Retrieval model and a general-purpose Multilingual similarity model (e.g. classification, clustering, etc.), both Apache 2.0
š§ my modern training strategy: ideation -> dataset choice -> implementation -> evaluation
š my training scripts, using the Sentence Transformers library
š my Weights & Biases reports with losses & metrics
š my list of 30 training and 13 evaluation datasets
The 2 Static Embedding models have the following properties:
šļø Extremely fast, e.g. 107500 sentences per second on a consumer CPU, compared to 270 for 'all-mpnet-base-v2' and 56 for 'gte-large-en-v1.5'
0ļøā£ Zero active parameters: No Transformer blocks, no attention, not even a matrix multiplication. Super speed!
š No maximum sequence length! Embed texts at any length (note: longer texts may embed worse)
š Linear instead of exponential complexity: 2x longer text takes 2x longer, instead of 2.5x or more.
šŖ Matryoshka support: allow you to truncate embeddings with minimal performance loss (e.g. 4x smaller with a 0.56% perf. decrease for English Similarity tasks)
Check out the full blogpost if you'd like to 1) use these lightning-fast models or 2) learn how to train them with consumer-level hardware: https://huggingface.co./blog/static-embeddings
The blogpost contains a lengthy list of possible advancements; I'm very confident that our 2 models are only the tip of the iceberg, and we may be able to get even better performance.
Alternatively, check out the models:
* sentence-transformers/static-retrieval-mrl-en-v1
* sentence-transformers/static-similarity-mrl-multilingual-v1
- 1 reply
Post
3019
That didn't take long! Nomic AI has finetuned the new ModernBERT-base encoder model into a strong embedding model for search, classification, clustering and more!
Details:
š¤ Based on ModernBERT-base with 149M parameters.
š Outperforms both nomic-embed-text-v1 and nomic-embed-text-v1.5 on MTEB!
šļø Immediate FA2 and unpacking support for super efficient inference.
šŖ Trained with Matryoshka support, i.e. 2 valid output dimensionalities: 768 and 256.
ā”ļø Maximum sequence length of 8192 tokens!
2ļøā£ Trained in 2 stages: unsupervised contrastive data -> high quality labeled datasets.
ā Integrated in Sentence Transformers, Transformers, LangChain, LlamaIndex, Haystack, etc.
šļø Apache 2.0 licensed: fully commercially permissible
Try it out here: nomic-ai/modernbert-embed-base
Very nice work by Zach Nussbaum and colleagues at Nomic AI.
Details:
š¤ Based on ModernBERT-base with 149M parameters.
š Outperforms both nomic-embed-text-v1 and nomic-embed-text-v1.5 on MTEB!
šļø Immediate FA2 and unpacking support for super efficient inference.
šŖ Trained with Matryoshka support, i.e. 2 valid output dimensionalities: 768 and 256.
ā”ļø Maximum sequence length of 8192 tokens!
2ļøā£ Trained in 2 stages: unsupervised contrastive data -> high quality labeled datasets.
ā Integrated in Sentence Transformers, Transformers, LangChain, LlamaIndex, Haystack, etc.
šļø Apache 2.0 licensed: fully commercially permissible
Try it out here: nomic-ai/modernbert-embed-base
Very nice work by Zach Nussbaum and colleagues at Nomic AI.
tomaarsenĀ
authored
a
paper
2 months ago
Adding `safetensors` variant of this model
#1 opened almost 2 years ago
by
SFconvertbot
Adding `safetensors` variant of this model
#1 opened almost 2 years ago
by
SFconvertbot
Adding `safetensors` variant of this model
#1 opened almost 2 years ago
by
SFconvertbot
Adding `safetensors` variant of this model
#1 opened almost 2 years ago
by
SFconvertbot
Adding `safetensors` variant of this model
#1 opened over 1 year ago
by
SFconvertbot
Adding `safetensors` variant of this model
#1 opened almost 2 years ago
by
SFconvertbot
Adding `safetensors` variant of this model
#1 opened almost 2 years ago
by
SFconvertbot
Adding `safetensors` variant of this model
#1 opened almost 2 years ago
by
SFconvertbot
Adding `safetensors` variant of this model
#1 opened almost 2 years ago
by
SFconvertbot
Adding `safetensors` variant of this model
#1 opened almost 2 years ago
by
SFconvertbot
Adding `safetensors` variant of this model
#3 opened 3 months ago
by
SFconvertbot
Adding `safetensors` variant of this model
#4 opened 3 months ago
by
SFconvertbot
Adding `safetensors` variant of this model
#1 opened almost 2 years ago
by
SFconvertbot
Adding `safetensors` variant of this model
#1 opened almost 2 years ago
by
SFconvertbot
fix with actual model name
1
#4 opened 4 months ago
by
venkyyuvy