arshyajabbari's picture
add tokenizer
4130e3e
raw
history blame
375 Bytes
{"ق": 0, "ف": 1, "گ": 2, "ک": 3, "ا": 4, "ظ": 5, "چ": 6, "خ": 7, "ع": 8, "پ": 9, "ن": 10, "ش": 11, "ء": 12, "د": 13, "ت": 14, "ص": 15, "ض": 16, "غ": 17, "ح": 18, "ز": 19, "ر": 20, "ئ": 21, "ذ": 22, "و": 23, "ج": 24, "س": 26, "ب": 27, "م": 28, "ی": 29, "ه": 30, "ط": 31, "ژ": 32, "ل": 33, "آ": 34, "ث": 35, "|": 25, "[UNK]": 36, "[PAD]": 37}