WizardLMTeam

/

WizardMath-70B-V1.0

like 117

I noticed inside the config that "max_position_embeddings": 2048,". The base 70b model has a 4096 token length (see https://huggingface.co./meta-llama/Llama-2-70b-chat-hf/blob/main/config.json).
Was this intentionally reduced? Additionally, the tokenizer is indicating an even lower token limit (see below)
Token indices sequence length is longer than the specified maximum sequence length for this model (2661 > 1500). Running this sequence through the model will result in indexing errors