raygx
/

BERT_Nepali_Tokenizer

Model card Files Files and versions Community

raygx commited on Jul 11, 2023

Commit

0ce72c0

•

1 Parent(s): d5f0cf4

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -3,7 +3,7 @@
 - This tokenizer is a PreTrainedTokenizerFast which is trained on raygx/Nepali-Extended-Corpus datasets.
 - This tokenizer is trained from scratch using Tokenizers library.
 - This tokenizer uses
-  - Model: BPE(unk_token="[UNK]")
   - Normalizer: normalizers.Sequence([NFD(),Strip()])
   - Pre-processor: pre_tokenizers.Sequence([Whitespace(),Digits(individual_digits=True), Punctuation()])
   - Post-processor: BertProcessing

 - This tokenizer is a PreTrainedTokenizerFast which is trained on raygx/Nepali-Extended-Corpus datasets.
 - This tokenizer is trained from scratch using Tokenizers library.
 - This tokenizer uses
+  - Model: Tokenizer(WordPiece(unk_token="[UNK]"))
   - Normalizer: normalizers.Sequence([NFD(),Strip()])
   - Pre-processor: pre_tokenizers.Sequence([Whitespace(),Digits(individual_digits=True), Punctuation()])
   - Post-processor: BertProcessing