RuntimeError: q_weight and gptq_qzeros have incompatible shapes
I got the following error when using the weight in exllama2.
Traceback (most recent call last):
File "/data/NZH/exllamav2/my_inference.py", line 110, in
model.load_autosplit(cache)
File "/data/NZH/exllamav2/exllamav2/model.py", line 349, in load_autosplit
for item in f: x = item
File "/data/NZH/exllamav2/exllamav2/model.py", line 438, in load_autosplit_gen
module.load()
File "/data/NZH/exllamav2/exllamav2/attn.py", line 236, in load
self.q_proj.load()
File "/data/NZH/exllamav2/exllamav2/linear.py", line 104, in load
self.q_handle = ext.make_q_matrix(w, self.temp_dq, prescale = self.prescale)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/NZH/exllamav2/exllamav2/ext.py", line 258, in make_q_matrix
return ext_c.make_q_matrix(w["qweight"],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: q_weight and gptq_qzeros have incompatible shapes