MedImageInsight: Open-Source Medical Image Embedding Model
This repository provides a simplified implementation for using the MedImageInsight model, an open-source medical imaging embedding model presented in the paper MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging by Noel C. F. Codella et al. The official guide to access the model from Microsoft is quite complicated, and it is arguable whether the model is truly open-source. This repository aims to make it easier to use the MedImageInsight model for various tasks, such as zero-shot classification, image embedding, and text embedding.
What we have done:
- Downloaded the models from Azure
- Got rid of all the unnecessary files
- Got rid of unnecessary MLflow code to make a standalone implementation
- Moved to uv for dependency management
- Added multi-label classification
- Created an example with the FastAPI service
Usage
- Clone the repository and navigate to the project directory.
Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co./lion-ai/MedImageInsights
- Install the required dependencies We are using uv package manager to simplify the installation.
To create a virtual env, simply run:
uv sync
Or to run a single script, just run:
uv run example.py
That's it!
Examples
See to the example.py
file.
Zero-shot image classification
Here's an example of how to use the MedImageInsight
class for zero-shot classification:
# Initialize classifier
classifier = MedImageInsight(
model_dir="2024.09.27",
vision_model_name="medimageinsigt-v1.0.0.pt",
language_model_name="language_model.pth"
)
# Load model
classifier.load_model()
# Read image
image = base64.encodebytes(read_image("image.png")).decode("utf-8")
# Zero-shot classification
images = [image]
labels = ["normal", "Pneumonia", "unclear"]
results = classifier.predict(images, labels)
print(results)
Multi-label zero-shot image classification
Run multi-label image classification (without softmax at the end)
# Multilabel classification example
images = [image]
labels = ["normal", "Pneumonia", "Fracture", "Tumor"]
results = classifier.predict(images, labels, multilabel=True)
print(results)
Image embeddings
results = classifier.encode(images=images)
print(results["image_embeddings"])
Text embeddings
results = classifier.encode(texts=labels)
print(results["text_embeddings"])
FastAPI server
uv run fastapi_app.py
Go to localhost:8000/docs to see the swagger.
The application provides endpoints for classification and image embeddings. Images have to be base64 encoded.
Roadmap
- Basic implementation
- Multilabel classification
- FastAPI service
- HF compatible API (from_pretrained())
- Explainability
Acknowledgments
This repository is based on the work presented in the paper "MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging" by Noel C. F. Codella et al. (arXiv:2410.06542).