Finalize Sentence Transformers integration
Hello!
Thanks @Samoed for setting up this work.
Pull Request overview
- Add
modules.jsonto inform Sentence Transformers what modules to use- It points to the new MultiModalTransformer module, but also normal Pooling and Normalize modules
- Configure the Pooling layer to use last token pooling
- Add the Sentence Transformers usage to the README
Details
I've updated the usage compared to
@Samoed
's suggestion: instead of always specifying both a type and a content, users can now specify a dictionary with text, image, or both. Still, as
@intfloat
mentioned, you cannot embed both image-text and pure text at the same time, but I think that's not a big problem. As before, just using "A cat and a dog" instead of {"text": "A cat and a dog"} also works, but in the README I want to inform users of {"text": "A cat and a dog"} so they (hopefully) understand how it can be expanded to also include an image in that embedding.
I want to share that I was unable to get 100% correspondance between Transformers and Sentence Transformers, annoyingly enough. The problem is somewhere in the processing or model forward, not the pooling or normalization, and the difference is quite small. For example, the mean and std of the model last hidden states are identical to at least 4 significant digits. The cosine similarity of the Image + Text -> Text README examples is [0.3965, 0.3105] for transformers and [0.3967, 0.3090] for Sentence Transformers. The Text -> Image examples has [0.4219, 0.3887] for transformers and [0.4250, 0.3896] for Sentence Transformers.
- Tom Aarsen
Great! Can you also update custom_st file, because it seems that it would require to pass <|begin_of_text|> after an image. https://github.com/haon-chen/mmE5/issues/1#issuecomment-2678420039
I add it automatically if there is both an image and text: https://huggingface.co./intfloat/mmE5-mllama-11b-instruct/blob/refs%2Fpr%2F3/custom_st.py#L71-L72
Should <|begin_of_text|> always be included, even if there is only an image and no text?
And should it be included if there is no image, but only text?
Also, I remove spaces from the left of the text if there is no image, and I make sure there is 1 space of text between <|begin_of_text|> and the text if there is an image. Is that correct?
- Tom Aarsen
Hello @tomaarsen !
Thanks for the great work!
Should <|begin_of_text|> always be included, even if there is only an image and no text?
Well, this case should not happen. When only an image is available, a text instruction should be added along with the image such as <|image|><|begin_of_text|> Represent the given image.
I know it feels unnecessary, but currently the model is trained in this way.
And should it be included if there is no image, but only text?
No, as the example of encoding A cat and a dog shows, there is no need to include <|begin_of_text|> when only text.
Also, I remove spaces from the left of the text if there is no image, and I make sure there is 1 space of text between <|begin_of_text|> and the text if there is an image. Is that correct?Yes, that is correct. In fact, no need to add extra space anyways, the previous demo is buggy.