Incoherent Output
Using Q4_0 on koboldcpp 1.52.2, the model is outputting the newline as <0x0A>, has broken speech, nonsensical repetition, and all the typical hallmarks of a broken model. Seems like it's probably a model issue not a quant issue, but i can't test the fp16 to confirm.
A simple qa assistant prompt inside proper mistral template like "roleplay as a large language model trained to act as a personal assistant. User: Please explain how a microwave oven works." it outputs:
Yes, I can, please help you learn a new language, a new roleplay and a new language model.<0x0A>[/INST]<0x0A><0x0A>Open Assistant: How do you learn a new language?<0x0A>[INST]<0x0A>Open Assistant: A large number of users have a large language model trained to help users learn a new language, a new roleplay and a new language model.<0x0A>[/INST]<0x0A><0x0A>Open Assistant: As a large language model, you are learning a new language model and you are a large language model.<0x0A>[/INST]<0x0A><0x0A>Open Assistant: A large number of users have a large language model trained to help users learn a new language, a new roleplay and a new language model.<0x0A>[/INST]<0x0A><0x0A>Open Assistant: The model is based on the idea that the model will help you learn a new language model.<0x0A>[/INST]<0x0A><0x0A>Open Assistant: A large number of users have a large language model trained to help users learn a new language, a new roleplay and a new language model.<0x0A>[/
I just tried out the Q3_K_M GGUF with llama.cpp
The same issue for me. The responses are somewhat related to the prompt but still gibberish.
Eg.
What was John Holt's vision on education?
"I decided to become a scientist after I saw that the professors in my school were not very keen on education. It seemed to me like science would be something that could help me get out of poverty and provide a good lifestyle for my family and give them a way to make money. So I studied science after seeing that the teachers in my school were not interested in education. That is what was my vision on education as I saw it in other schools. "
It's missing the tokenizer. Needs updated with on.