Do we know if this is a base model or instruction tuned?
Lack of prompt and comparison to base mistral tells me it's a base model. But the word "chat" in the name is kinda indicative of it possibly being chat tuned or something.
It's instruction fine tuned - trained on SlimOrca. They have a v1 model that used ChatML format and a v2 that used Alpaca. But they've not listed the prompt format for this one, and I couldn't find it in the code, so I've marked it as unknown for now.
If/when I find out what the template is, I'll update
Prompt template is updated, thanks
I must be using it incorrectly because I can only produce junk output with it.
I must be using it incorrectly because I can only produce junk output with it.
me too
I've tried both prompt styles and both worked well for me. At least when running in ollama. For ollama user, here are the Modelfiles:
Alpaca Style:
FROM neural-chat-7b-v3-1.Q5_K_M.gguf
PARAMETER temperature 0.1
PARAMETER num_ctx 8192
PARAMETER num_predict 583
PARAMETER top_k 1
TEMPLATE """
{{- if .First }}{{ .System }}{{- end }}
### User:
{{ .Prompt }}
### Assistant:
"""
SYSTEM """
"""
And here ChatML:
FROM neural-chat-7b-v3-1.Q5_K_M.gguf
PARAMETER temperature 0.1
PARAMETER num_ctx 8192
PARAMETER num_predict 583
PARAMETER top_k 1
PARAMETER stop <|im_end|>
TEMPLATE """
{{- if .First }}
<|im_start|>system
{{ .System }}<|im_end|>
{{- end }}
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""
SYSTEM """
"""
I must be using it incorrectly because I can only produce junk output with it.
me too
Turns out using the HF variant of the loader was breaking it for me, using the non HF version fixed it for me.
I've tried both prompt styles and both worked well for me. At least when running in ollama. For ollama user, here are the Modelfiles:
Alpaca Style:
FROM neural-chat-7b-v3-1.Q5_K_M.gguf PARAMETER temperature 0.1 PARAMETER num_ctx 8192 PARAMETER num_predict 583 PARAMETER top_k 1 TEMPLATE """ {{- if .First }}{{ .System }}{{- end }} ### User: {{ .Prompt }} ### Assistant: """ SYSTEM """ """And here ChatML:
FROM neural-chat-7b-v3-1.Q5_K_M.gguf PARAMETER temperature 0.1 PARAMETER num_ctx 8192 PARAMETER num_predict 583 PARAMETER top_k 1 PARAMETER stop <|im_end|> TEMPLATE """ {{- if .First }} <|im_start|>system {{ .System }}<|im_end|> {{- end }} <|im_start|>user {{ .Prompt }}<|im_end|> <|im_start|>assistant """ SYSTEM """ """
these PARAMETERs work for me too, thanks