Appreciate the model drop!
But why is it only 4k? Its 2024 man, those are rookie numbers.
Haha.
Very good question. The model training concludes this June and we have been fighting for releasing a detailed tech report for long time---for a long time, the release has been proven to be difficulty.
Meanwhile, a different version of post-training has been conducted, with a focus on multi-lingual and long context ability. That model supports 128k and is released to https://huggingface.co./microsoft/Phi-3.5-MoE-instruct : )
@LiyuanLucasLiu would love to try Phi 3.5 Moe Instruct and vision locally in llama.cpp, but there has been absolutely zero movement to add support. Feature request is still open: https://github.com/ggerganov/llama.cpp/issues/9119
@YorkieOH10 I understand. It pains me as well... Releasing this model already gets me some trouble (cant share) & I may need to stay low for some time... Meanwhile, you can try the demo at https://huggingface.co./spaces/GRIN-MoE-Demo/GRIN-MoE (not sure how long i can keep it alive).