Try this quantizations, they are way better.
https://huggingface.co./ZeroWw/Phi-3-mini-128k-instruct-GGUF
These are my own quantizations (updated almost daily).
The difference with normal quantizations is that I quantize the output and embed tensors to f16.
and the other tensors to 15_k,q6_k or q8_0.
This creates models that are little or not degraded at all and have a smaller size.
They run at about 3-6 t/sec on CPU only using llama.cpp
And obviously faster on computers with potent GPUs
sure:
step 1:python llama.cpp/convert-hf-to-gguf.py --outtype f16 Phi-3-mini-4k-instruct --outfile Phi-3-mini-4k-instruct.f16.gguf
step 2-X:
./build/bin/llama-quantize --allow-requantize --output-tensor-type f16 --token-embedding-type f16 Phi-3-mini-4k-instruct.f16.gguf Phi-3-mini-4k-instruct.q5_k.gguf $QTYPE $(nproc)
where QTYPE is q5_k q6_k q8_0 etc etc
I also tried to quantize embed+output to q8_0 and the rest to q4 or q5 or q6 but the models degrade too much.
Thank you! I am curious if you notice any quality difference on the f16 and quantized models in comparison with the original one.
@ZeroWw
For Q8_0 you can also directly go like this convert-hf-to-gguf.py --outtype q8_0 ..\Phi-3-mini-128k-instruct\ --outfile Phi-3-mini-128k-instruct-Q8_0.gguf, in single pass. Did you test if your method offers measurable performance improvements, like for example better PPL scores?
@ZeroWw For Q8_0 you can also directly go like this
convert-hf-to-gguf.py --outtype q8_0 ..\Phi-3-mini-128k-instruct\ --outfile Phi-3-mini-128k-instruct-Q8_0.gguf, in single pass. Did you test if your method offers measurable performance improvements, like for example better PPL scores?
hmm I have tio check but I think the direct way generates a different output... in my way the output and embed tensors are at F16 and the other tensors at q8_0 or q6_k or q5_k