Unable submit this model to the LLM leaderboard (tokenizers issue)
unable to submit this model to the HuggingFace LLMV2 leaderboard
error message: The tokenizer cannot be loaded. Ensure the tokenizer class is part of a stable Transformers release and correctly configured.
relevant code line: https://huggingface.co./spaces/open-llm-leaderboard/open_llm_leaderboard/blob/main/backend/app/utils/model_validation.py#L200
would you look into this?
update: https://huggingface.co./spaces/open-llm-leaderboard/open_llm_leaderboard/discussions/1080
it was successfully submitted into some other leaderboards (tho not some others), but for those it did, it hasn't finishing running yet (wait and see),
for the other leaderboard where it would not work, was answered by a different, but similar error message:Model "byroneverson/Yi-1.5-34B-Chat-abliterated" uses a tokenizer which is not in a transformers release: Cannot instantiate this tokenizer from a slow version. If it's based on sentencepiece, make sure you have sentencepiece installed.
not to say that: something is not odd with this model, but would you know what is going on?
My best guess that this issue is inherent to the original model as I changed nothing to the tokenizer or any of the tokenizer related files. The only changes present in this repo compared to the original are the model weights. If you try benchmarking the original model and receive the same errors it may be worth asking the creator(s) of the original. It seems unlikely that any leaderboard wouldn’t have something common like sentencepiece installed but it is possible.