Custom tokenizer support for llama.cpp?
I'm trying to use llama.cpp's conversion script via my ggify tool, and it says
WARNING:hf-to-gguf:**************************************************************************************
WARNING:hf-to-gguf:** WARNING: The BPE pre-tokenizer was not recognized!
WARNING:hf-to-gguf:** There are 2 possible reasons for this:
WARNING:hf-to-gguf:** - the model has not been added to convert-hf-to-gguf-update.py yet
WARNING:hf-to-gguf:** - the pre-tokenization config has changed upstream
WARNING:hf-to-gguf:** Check your model files and convert-hf-to-gguf-update.py and update them accordingly.
WARNING:hf-to-gguf:** ref: https://github.com/ggerganov/llama.cpp/pull/6920
WARNING:hf-to-gguf:**
WARNING:hf-to-gguf:** chkhsh: 7fc505bd3104ca1083b150b17d088b59534ede9bde81f0dd2090967d7fe52cee
WARNING:hf-to-gguf:**************************************************************************************
I suspect this is because
Viking uses a custom 128K Bloom tokenizer trained on the same English, Finnish, Swedish, Danish, Norwegian, Icelandic and code dataset used to train the model.
Any ideas how to implement this, or are there already plans afoot to get this working as a GGUF model?
Viking uses the BloomTokenizer class from HF. After a quick look it seems like this tokenizer is not implemented in llama.cpp's conversion script. Our tokenizer is only custom in the sense that it has been trained for this model and not in the sense that it is custom code.
It looks like our tokenizer isn't supported. To workaround the issue you could add the hash like this, and it should work.
in convert-hf-to-gguf.py, after this block:
if chkhsh == "27949a2493fc4a9f53f5b9b029c82689cfbe5d3a1929bb25e043089e28466de6":
# ref: https://huggingface.co./jinaai/jina-embeddings-v2-base-de
res = "jina-v2-de"
add this:
if chkhsh == "7fc505bd3104ca1083b150b17d088b59534ede9bde81f0dd2090967d7fe52cee":
# ref: https://huggingface.co./LumiOpen/Viking-7B
res = "llama-bpe"
I'm submitting a PR to llama.cpp to hopefully save someone else trouble. Thanks for reporting it!
Looks like we made PRs a minute apart, @jonabur 😁
Anyway, for the intrepid llama.cppists, https://huggingface.co./akx/Viking-7B-gguf
hah, nice! and thanks for quantizing! we should probably release our own quantizations for some of these, at least for the chat models which are more generally useful for home users.
Thank you for quantizing it @akx , I was able to download the model using ollama an hour ago and try it on one of our GUIs. But I just got "//////////////" so I suspect I need to wait until your PR gets merged and ollama updates its dependencies. 🙂
Support for Viking pre-tokenizer was added to llama.cpp:
Thank you for quantizing it @akx , I was able to download the model using ollama an hour ago and try it on one of our GUIs. But I just got "//////////////" so I suspect I need to wait until your PR gets merged and ollama updates its dependencies. 🙂
We have the same issue when using Ollama.