Can we edit tokenizer.json inside gguf file instead of needing a requant?
I noticed that DeepSeek 33b official repo got this commit yesterday https://huggingface.co./deepseek-ai/deepseek-coder-33b-instruct/commit/90ca43c383888fdf8db93b69ce234860d56e7c3b
Which changes eos_token from <｜end▁of▁sentence｜> to <|EOT|>
I think this gets packed in the gguf file...
Is there a way to fix that locally without needing a requantization?
Found that you were the one committing it on https://huggingface.co./deepseek-ai/deepseek-coder-33b-instruct/commit/83bb1ca9a208a6855efc440cc77a0620e6a2a3bc
They rolled it back then re-set it to EOT
So no problem here.
The only change that's not on the guff files is https://huggingface.co./deepseek-ai/deepseek-coder-33b-instruct/commit/e3460f26f3f8fc3a2dbf81def708db7801e102e5 and probably the commit before it