Spaces:

Xenova
/

the-tokenizer-playground

Running

Mistral tokenizer

by thomasgauthier - opened Dec 11, 2023

Dec 11, 2023

Can the Mistral tokenizer be added? I use this tool all the time and now that I've switched from llama to mistral as my default local model having mistral tokenizer support would be great.

Thanks for this amazing tool by the way!

Xenova

Owner Dec 11, 2023

Hi there :) Fortunately, Mistral uses the exact same tokenizer as llama (see here), so, you can just use the llama option.

thomasgauthier

Dec 12, 2023

Oh I didn't realize, thank you!

thomasgauthier changed discussion status to closed Dec 12, 2023

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment