Mixtral-8x7B (gpt-fast edition)
This repo holds quantized Mixtral-8x7B weights to be used in gpt-fast.
Compatibility
Conversion to int4 was broken, so this repo only holds fp8 weights. Practically speaking this means your GPU(s) need to be Ada Lovelace or newer, and have enough VRAM to hold the model + KV cache + activations.
I'm hoping it can work on a pair of 4090s, which combined have 48 GiB (51.539607552 GB) of VRAM. Ignoring all overhead, this leaves ~4.74 GB for KV-cache and activations, which should be enough (?).
- TODO: Test on 2x4090 with TP=2
Notes
Conversion was done with (commit 7510a9d)
- Downloads last month
- 4
Inference Providers
NEW
This model is not currently available via any of the supported third-party Inference Providers, and
HF Inference API has been turned off for this model.