| base_model: meta-llama/Meta-Llama-3-8B-Instruct | |
| language: | |
| - en | |
| license: mit | |
| # Llama 3 8b Instruct MOE | |
| Llama 3 8b Instruct base model converted to MOE style by randomly partitioning the FFN layers of each decoder layer into 8 experts of the same size. Weights are directly taken from the llama3 instruct base model. |