LnL-AI
/

dbrx-base-tokenizer

Transformers.js

Inference Endpoints

Model card Files Files and versions Community

Qubitium commited on Apr 3

Commit

7cb66c0

•

1 Parent(s): 34ba11f

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -8,7 +8,7 @@ tags:
 ## Why should you use this and not the tiktoken included in the orignal model?
 1. Original tokenizer pad the vocabulary to correct size with `<extra_N>` tokens but encoder never uses them
 2. Original tokenizer use eos as pad token which may confuse trainers to mask out the eos token so model never output eos.
-3. [PEDNING] config.json embedding size of "vocab_size": 100352 does not match 100277
 modified from original code @ https://huggingface.co/Xenova/dbrx-instruct-tokenizer

 ## Why should you use this and not the tiktoken included in the orignal model?
 1. Original tokenizer pad the vocabulary to correct size with `<extra_N>` tokens but encoder never uses them
 2. Original tokenizer use eos as pad token which may confuse trainers to mask out the eos token so model never output eos.
+3. [NOT FIXED: INVESTIGATING] config.json embedding size of "vocab_size": 100352 does not match 100277
 modified from original code @ https://huggingface.co/Xenova/dbrx-instruct-tokenizer