bigscience-catalogue-data-dev
/

byte-level-bpe-tokenizer-no-norm-250k-whitespace-and-eos-regex-alpha-v3-dedup-lines-articles

Model card Files Files and versions Community

byte-level-bpe-tokenizer-no-norm-250k-whitespace-and-eos-regex-alpha-v3-dedup-lines-articles / tokenizer_config.json

TimeRobber

Add tokenizer

cec6759 over 2 years ago

history blame contribute delete

131 Bytes

{"unk_token": "<unk>", "eos_token": "</s>", "bos_token": "<s>", "pad_token": "<pad>", "tokenizer_class": "PreTrainedTokenizerFast"}