README.md

---
license: mit
license_link: https://huggingface.co./microsoft/Phi-3.5-MoE-instruct/resolve/main/LICENSE
language:
- multilingual
pipeline_tag: text-generation
tags:
- nlp
- code
widget:
- messages:
  - role: user
    content: Can you provide ways to eat combinations of bananas and dragonfruits?
library_name: transformers
---

> [!IMPORTANT]
> This repository is a quantized version of the original model [`microsoft/Phi-3.5-MoE-instruct`](https://huggingface.co./microsoft/Phi-3.5-MoE-instruct) which is the FP16 half-precision official version released by Microsoft.

## Model Summary

Phi-3.5-MoE is a lightweight, state-of-the-art open model built upon datasets used for Phi-3 - synthetic data and filtered publicly available documents - with a focus on very high-quality, reasoning dense data. The model supports multilingual and comes with 128K context length (in tokens). The model underwent a rigorous enhancement process, incorporating supervised fine-tuning, proximal policy optimization, and direct preference optimization to ensure precise instruction adherence and robust safety measures. 

🏡 [Phi-3 Portal](https://azure.microsoft.com/en-us/products/phi-3) <br>
📰 [Phi-3 Microsoft Blog](https://aka.ms/phi3.5-techblog) <br>
📖 [Phi-3 Technical Report](https://arxiv.org/abs/2404.14219) <br>
👩‍🍳 [Phi-3 Cookbook](https://github.com/microsoft/Phi-3CookBook) <br>
🖥️ [Try It](https://aka.ms/try-phi3.5moe) <br>

MoE references:
📜[Phi-3.5-MoE Blog](https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/announcing-the-availability-of-phi-3-5-moe-in-azure-ai-studio/ba-p/4256278) | 😁[GRIN MoE](https://huggingface.co./microsoft/GRIN-MoE)

**Phi-3.5**: [[mini-instruct]](https://huggingface.co./microsoft/Phi-3.5-mini-instruct); [[MoE-instruct]](https://huggingface.co./microsoft/Phi-3.5-MoE-instruct) ; [[vision-instruct]](https://huggingface.co./microsoft/Phi-3.5-vision-instruct)

## Running 🏃

### TGI

```bash
model=danieldk/Phi-3.5-MoE-instruct-AWQ-INT4
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data \
    ghcr.io/huggingface/text-generation-inference:2.4.0 \
    --model-id $model --num-shard 2
```

## Quantization Reproduction

Soon (need to upstream an AutoAWQ patch).