PJMixers-Dev
/

Mistral-Nemo-Base-2407-ChatML

xzuyn commited on 13 days ago

Commit

c9f9e46

verified ·

1 Parent(s): 7ff1ee9

Fix missing ChatML tokens in vocab section

Files changed (1) hide show

tokenizer.json CHANGED Viewed

@@ -9117,8 +9117,8 @@
       "[PREFIX]": 11,
       "[MIDDLE]": 12,
       "[SUFFIX]": 13,
-      "<SPECIAL_14>": 14,
-      "<SPECIAL_15>": 15,
       "<SPECIAL_16>": 16,
       "<SPECIAL_17>": 17,
       "<SPECIAL_18>": 18,

       "[PREFIX]": 11,
       "[MIDDLE]": 12,
       "[SUFFIX]": 13,
+      "<|im_start|>": 14,
+      "<|im_end|>": 15,
       "<SPECIAL_16>": 16,
       "<SPECIAL_17>": 17,
       "<SPECIAL_18>": 18,