Any chance of providing an iMatrix?
#2
by
smcleod
- opened
It would be great if an iMatrix file could be provided to improve quantization efforts.
For example I think you could create this using the Dolphin flan5m alpaca uncensored dataset by doing something like:
# download cognitivecomputations/dolphin
hfdownloader -d cognitivecomputations/dolphin --storage .
#optionally convert to fp16
llama.cpp/convert-hf-to-gguf.py ./abacusai_Smaug-Mixtral-v0.1 --outtype f16 -outfile abacusai_Smaug-Mixtral-v0.1-GGUF/abacusai_Smaug-Mixtral-v0.1.fp16.bin
# create imatrix
imatrix -m ./abacusai_Smaug-Mixtral-v0.1-GGUF/abacusai_Smaug-Mixtral-v0.1.fp16.bin -f ./datasets/cognitivecomputations_dolphin/flan5m-alpaca-uncensored-deduped.jsonl -ngl 99
@smcleod
I am uploading some imatrix quants here if you want:
https://huggingface.co./dranger003/Smaug-Mixtral-v0.1-iMat.GGUF
Thanks @dranger003 , appreciate that!