intfloat

/

e5-mistral-7b-instruct

like 469

Hello!

I've quite enjoyed watching this model release. In truth, I did not expect an LLM to be capable of producing such valuable embeddings. To me, this begs the question: could we distill these high quality embeddings into a smaller model (e.g. bge-small) to 1) improve the performance of the smaller student model and 2) create longer sequence length models without requiring long-sequence labeled training data.

Additionally, I'm very interested in implementing first-party support for models such as this one in Sentence Transformers. One key aspect that is currently lacking is prompt template support. I was picturing adding a prompts configuration option like so:

{
    ...
    "prompts": {
        "retrieval": "Retrieve semantically similar text. {}",
        "summarization": "Given a news summary, retrieve other semantically similar summaries. {}",
        ...
    },
    "default_prompt_key": "retrieval",
}

Then, models can be ran like so:

from sentence_transformers import SentenceTransformers

model = SentenceTransformers("intfloat/multilingual-e5-base")
model.encode("how much protein should a female eat", prompt_key="query")
model.encode("As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day.", prompt_key="passage")

The same would work for intfloat/e5-mistral-7b-instruct.
I have this change planned alongside various other improvements of Sentence Transformers, such as a stronger Trainer, more modern objective functions (InfoNCE, AngLE), multi-gpu training, and more. I'd be very curious to hear your thoughts on all of this.

Keep up the great work.

• Tom Aarsen