Upload tokenizer
#22
by
ybelkada
- opened
Uploads the transformers compatible version of the tokenizer, which is the same as https://huggingface.co./mistral-community/Codestral-22B-v0.1
Thanks! Codestral also has a suffix and prefix token that’s useful for FIM functionality, but we can add this later one supported by transformers
patrickvonplaten
changed pull request status to
merged