Ideal quantization levels
#6
by
jadbox
- opened
Has there been any GGUF quant level tests on Medius? Some people find Llama to perform just as good as 4_km vs 5 km, and l'm wondering if this holds true for medius?
I'm interested in this too. I've been playing with the IQ3_XS and IQ4_XS, can't notice crazy differences but i need to do more testing. IQ fits on 8GB VRAM.