eos_token_id error in oobabooga

#1
by Olafangensan - opened
Traceback (most recent call last):

File "M:\oobabooga_TGWUI\text-generation-webui-1.6\modules\ui_model_menu.py", line 232, in load_model_wrapper


shared.model, shared.tokenizer = load_model(selected_model, loader)
File "M:\oobabooga_TGWUI\text-generation-webui-1.6\modules\models.py", line 94, in load_model


output = load_func_map[loader](model_name)
File "M:\oobabooga_TGWUI\text-generation-webui-1.6\modules\models.py", line 329, in ExLlamav2_HF_loader


return Exllamav2HF.from_pretrained(model_name)
File "M:\oobabooga_TGWUI\text-generation-webui-1.6\modules\exllamav2_hf.py", line 173, in from_pretrained


config.prepare()
File "M:\oobabooga_TGWUI\text-generation-webui-1.6\installer_files\env\lib\site-packages\exllamav2\config.py", line 192, in prepare


self.eos_token_id = read(read_config, int, "eos_token_id", None)  # 2
File "M:\oobabooga_TGWUI\text-generation-webui-1.6\installer_files\env\lib\site-packages\exllamav2\config.py", line 40, in read


raise TypeError(f"Value for {key} is not of expected type {expected_type}")
TypeError: Value for eos_token_id is not of expected type <class 'int'>

I'm getting the same error

You need the dev branch of ExLlamaV2 to run it correctly, because they changed a few things with the RoPE implementation.

The error you're getting is because the config defines multiple EOS tokens which is also handled in the dev branch. You could just change the eos_token_id key to be a single value of 128001, but you might get incorrect inference for longer sequence lengths if you don't update ExLlama. There will be a new release tomorrow, but it needs a little more testing first.

Thanks for a fast response, I though they wouldn't change anything if it's just an upgrade, but oh well.

I replaced the files of exllamav2 from the oobabooga webui with the ones from the dev branch of ExLlamaV2 and now it runs perfectly. Thank you!

Sign up or log in to comment