File size: 2,343 Bytes
213e1a6 073f369 213e1a6 073f369 213e1a6 073f369 213e1a6 073f369 213e1a6 073f369 213e1a6 073f369 213e1a6 073f369 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
---
license: mit
license_link: https://huggingface.co./microsoft/Phi-3.5-MoE-instruct/resolve/main/LICENSE
language:
- multilingual
pipeline_tag: text-generation
tags:
- nlp
- code
widget:
- messages:
- role: user
content: Can you provide ways to eat combinations of bananas and dragonfruits?
library_name: transformers
---
> [!IMPORTANT]
> This repository is a quantized version of the original model [`microsoft/Phi-3.5-MoE-instruct`](https://huggingface.co./microsoft/Phi-3.5-MoE-instruct) which is the FP16 half-precision official version released by Microsoft.
## Model Summary
Phi-3.5-MoE is a lightweight, state-of-the-art open model built upon datasets used for Phi-3 - synthetic data and filtered publicly available documents - with a focus on very high-quality, reasoning dense data. The model supports multilingual and comes with 128K context length (in tokens). The model underwent a rigorous enhancement process, incorporating supervised fine-tuning, proximal policy optimization, and direct preference optimization to ensure precise instruction adherence and robust safety measures.
🏡 [Phi-3 Portal](https://azure.microsoft.com/en-us/products/phi-3) <br>
📰 [Phi-3 Microsoft Blog](https://aka.ms/phi3.5-techblog) <br>
📖 [Phi-3 Technical Report](https://arxiv.org/abs/2404.14219) <br>
👩🍳 [Phi-3 Cookbook](https://github.com/microsoft/Phi-3CookBook) <br>
🖥️ [Try It](https://aka.ms/try-phi3.5moe) <br>
MoE references:
📜[Phi-3.5-MoE Blog](https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/announcing-the-availability-of-phi-3-5-moe-in-azure-ai-studio/ba-p/4256278) | 😁[GRIN MoE](https://huggingface.co./microsoft/GRIN-MoE)
**Phi-3.5**: [[mini-instruct]](https://huggingface.co./microsoft/Phi-3.5-mini-instruct); [[MoE-instruct]](https://huggingface.co./microsoft/Phi-3.5-MoE-instruct) ; [[vision-instruct]](https://huggingface.co./microsoft/Phi-3.5-vision-instruct)
## Running 🏃
### TGI
```bash
model=danieldk/Phi-3.5-MoE-instruct-AWQ-INT4
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data \
ghcr.io/huggingface/text-generation-inference:2.4.0 \
--model-id $model --num-shard 2
```
## Quantization Reproduction
Soon (need to upstream an AutoAWQ patch). |