|
--- |
|
license: apache-2.0 |
|
language: |
|
- fr |
|
- it |
|
- de |
|
- es |
|
- en |
|
inference: false |
|
--- |
|
# Model Card for Mixtral-Extraction-4x7B-Instruct-v0.1 |
|
This model is an experimental model created by merging [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co./mistralai/Mixtral-8x7B-Instruct-v0.1) experts. |
|
|
|
# How we extracted experts |
|
Experts are selected and extracted. |
|
This model specifies 4 experts. |
|
|
|
# How To Convert |
|
use colab cpu-high-memory. |
|
You can extract experts 1-7 by selecting experts as bit string. |
|
|
|
~~~python |
|
experts_extract_bit = "11110000" |
|
~~~ |
|
[convert_mixtral_8x7b_to_4x7b_extract.ipynb](https://huggingface.co./mmnga/Mixtral-Extraction-4x7B-Instruct-v0.1/blob/main/notebook/convert_mixtral_8x7b_to_4x7b_extract.ipynb) |
|
|
|
# Usage |
|
~~~python |
|
pip install git+https://github.com/huggingface/transformers --upgrade |
|
pip install torch accelerate bitsandbytes flash_attn |
|
~~~ |
|
|
|
~~~python |
|
from transformers import AutoTokenizer, AutoModelForCausalLM, MixtralForCausalLM |
|
import torch |
|
|
|
model_name_or_path = "mmnga/Mixtral-Extraction-4x7B-Instruct-v0.1" |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path) |
|
model = MixtralForCausalLM.from_pretrained(model_name_or_path, load_in_8bit=True) |
|
|
|
text = "[INST] What was John Holt's vision on education? [/INST] " |
|
# text = "[INST] What is the best anime? [/INST] " |
|
inputs = tokenizer("<s> " + text, return_tensors="pt") |
|
|
|
outputs = model.generate(**inputs, max_new_tokens=128) |
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
|
~~~ |
