Broken quants?
Maybe my download broke it, but https://huggingface.co./TheBloke/Noromaid-v0.4-Mixtral-Instruct-8x7b-Zloss-GGUF/blob/main/noromaid-v0.4-mixtral-instruct-8x7b-zloss.Q5_K_M.gguf was complete garbage for me, outputting nothing coherent.
https://huggingface.co./NeverSleep/Noromaid-v0.4-Mixtral-Instruct-8x7b-Zloss-GGUF/blob/main/Noromaid-v0.4-Mixtral-Instruct-8x7b.q5_k_m.gguf on the other hand, is fine.
Same Problem.
Just a bunch of "▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅" must really be broken, darn.
Same.
Mixtral 8x7b and the like apparently have problems with K quants sometimes (or always, I didn't test). Did you try if it works with Q5_0?
That's intestesting. I use to download Q6K too. Perhaps it depends on the program you use.
im mostly using ooba's text gen for the gui, and llama.ccp for the engine for GGUFs. For 'raw' models, mostly transformers engine, but i dont have a big enough GPU to do that for large models so gguf for me :)
I confirm 5_K_M in this repo is broken, while 6_K is working. NeverSleep's version for both quants is working ok.
I also can confirm that file for "5_K_M" in this specific repo is corrupted. Do not download. Wish I seen the discussion first. I have verified the checksums on my end, so the file uploaded itself is already corrupted.