Create tokenizer_config.json
Browse filesThe model max length is currently set at a value within the transformers toolkit directly, as can be seen here: https://github.com/huggingface/transformers/blob/main/src/transformers/models/squeezebert/tokenization_squeezebert.py
We're removing these definitions as legacy; we're therefore updating the remote checkpoints to ensure that this information is not lost. Thank you!
- tokenizer_config.json +1 -0
tokenizer_config.json
ADDED
@@ -0,0 +1 @@
|
|
|
|
|
1 |
+
{"do_lower_case": true, "model_max_length": 512}
|