README.md

---
license: apache-2.0
---

This model is a merged model, using [bge-small-en-v1.5](https://huggingface.co./BAAI/bge-small-en-v1.5), [GIST-Embedding-v0](https://huggingface.co./avsolatorio/GIST-Embedding-v0) and [gte-base](https://huggingface.co./thenlper/gte-base).
This model focuses on retrieval tasks while also performing well on various tasks (See experiment details below).

## Usage

## For retrieval tasks
```python
from transformers import AutoTokenizer, AutoModel
import torch
# Sentences we want sentence embeddings for
token=""
sentences = ["this is a test sentence", "this is another test sentence"]

# Prefixing for retrieval tasks
instruction = "Represent this sentence for searching relevant passages: "

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('Marqo/marqo-merged-bge-gist-gte-base', token=token)
model = AutoModel.from_pretrained('Marqo/marqo-merged-bge-gist-gte-base', token=token)
model.eval()

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
encoded_input_with_prefixing = tokenizer([instruction + q for q in sentences], padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)
    model_output_with_prefixing = model(**encoded_input_with_prefixing)
    sentence_embeddings = model_output[0][:, 0]
    sentence_embeddings_with_prefixing = model_output_with_prefixing[0][:, 0]
    
    sentence_embeddings_avg = (sentence_embeddings + sentence_embeddings_with_prefixing) / 2
    
# normalize embeddings
sentence_embeddings_avg = torch.nn.functional.normalize(sentence_embeddings_avg, p=2, dim=1)
print("Sentence embeddings:", sentence_embeddings_avg)
```
## Evaluation
<img src="slerp.png" alt="109M models retrieval benchmarks" width="650" style="margin-left:'auto' margin-right:'auto' display:'block'"/>

| Models | Average | ArguAna | ClimateFEVER | CQADupstackRetrieval | DBPedia | FEVER | FiQA2018 | HotpotQA | MSMARCO | NFCorpus | NQ | QuoraRetrieval | SCIDOCS | SciFact | Touche2020 | TRECCOVID |
| ------ | ------- | ------- | ------------ | -------------------- | ------- | ----- | -------- | -------- | ------- | -------- | -- | -------------- | ------- | ------- | ---------- | --------- |
| Slerp (average prefixing) | 53.37 | 63.45 | 31.66 | 42.51 | 42.15 | 86.53 | 43.28 | 73.72 | 39.16 | 37.66 | 54.39 | 88.96 | 22.7 | 76.66 | 21.46 | 76.23 |
| Slerp (prefixing) | 53.18 | 63.58 | 30.67 | 43.23 | 41.52 | 86.54 | 41.28 | 71.43 | 41.16 | 38.01 | 55.28 | 88.72 | 22.75 | 75.05 | 22.15 | 76.27 |
| BGE | 53.25 | 63.61 | 31.17 | 42.35 | 40.77 | 86.29 | 40.65 | 72.6 | 41.35 | 37.39 | 54.15 | 88.9 | 21.73 | 74.04 | 25.7 | 78.07 |
| GIST | 52.31 | 62.62 | 31.49 | 43.2 | 41.7 | 86.65 | 40.64 | 68.92 | 40.64 | 37.64 | 53.43 | 88.81 | 23.47 | 75.29 | 20.58 | 69.6 |
| GTE | 51.14 | 57.12 | 28.1 | 42.91 | 41.19 | 81.52 | 40.76 | 65.75 | 40.21 | 37.9 | 52.84 | 88.15 | 23.13 | 76.18 | 22.55 | 68.78 |