Fsoft-AIC/Phi3.5-Siglip-MoE

LibMoE: A Library for Comprehensive Benchmarking of Mixture of Experts in Large Language Models

Introduction

Mixture of Experts (MoEs) plays an essential role in the development of more efficient and effective large language models (LLMs). Due to the enormous resource requirements, studying large-scale MoE algorithms remains inaccessible to many researchers. This work introduces LibMoE, a comprehensive and modular framework designed to streamline the research, training, and evaluation of MoE algorithms. Built upon three core principles: (i) modular design, (ii) efficient training, and (iii) comprehensive evaluation, LibMoE makes MoEs in LLMs more accessible to a wider range of researchers by standardizing the training and evaluation pipelines. Using LibMoE, we extensively benchmarked five state-of-the-art MoE algorithms across three different LLMs and 11 datasets under a zero-shot setting. The results show that, despite unique characteristics, all MoE algorithms perform similarly when averaged across a broad range of tasks. With its modular design and extensive evaluation capabilities, we believe LibMoE will be invaluable for researchers striving to make meaningful progress toward the next generation of MoE and LLMs.

Model and Evaluation Benchmarks

We have released five MoE algorithms trained based on microsoft/Phi-3.5-mini-instruct for LLMs and SigLIP for vision encoding. These models were trained on the LLAVA-665K dataset. We evaluated these state-of-the-art algorithms on 11 benchmarks, examining various aspects of MoE algorithm performance.

Model	MoE Method	AI2D	Text VQA	GQA	Hallusion Benchmark	MathVista Validation	MMBenchEN / dev	MMMU Validation	MMStar	POPE	SQA IMG Full	MME	AVG
SigLIP 224 + Phi3.5	SMoE-R	65.71	41.21	60.78	42.38	29.70	70.79	42.11	41.67	85.19	79.13	1,665.15	55.87
	Cosine-R	65.16	41.92	61.15	41.96	29.20	70.79	42.78	41.57	86.39	79.42	1,748.67	56.03
	Sigmoid-R	64.60	40.49	60.23	41.01	30.40	70.45	41.44	42.05	86.03	78.54	1,684.87	55.52
	Hyper-R	65.06	41.68	60.97	41.01	31.40	70.88	42.22	42.60	86.78	79.43	1,707.86	56.20
	Perturbed Cosine-R	65.54	41.85	61.04	41.75	30.50	71.65	43.00	41.72	86.73	78.88	1,688.82	56.27
-

Run LibMoE

We provide detailed instructions for setting up and running experiments in this repository: https://github.com/Fsoft-AIC/LibMoE

Hardware Resources

Stage	MoE Method	Hardware
Pre-Training		4xA100
Pre-FineTuning		4xA100
VIT	SMoE-R	6xA100
	Cosine-R	6xA100
	Sigmoid-R	6xA100
	Hyper-R	6xA100
	Perturbed Cosine-R	6xA100

---

Citation Information

More details can be found in our paper.

If you use LibMoE, please cite it using this BibTeX:

@misc{nguyen2024libmoelibrarycomprehensivebenchmarking,
      title={LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models}, 
      author={Nam V. Nguyen and Thong T. Doan and Luong Tran and Van Nguyen and Quang Pham},
      year={2024},
      eprint={2411.00918},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2411.00918}, 
}