WizardLMTeam

/

WizardCoder-Python-34B-V1.0

like 760

Hey,

I am using this model on a p3.8x large and I am getting the below error while inferencing:

16GB*4 A100 and 216gb Ram

output = old_forward(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/transformers/models/llama/modeling_llama.py", line 195, in forward
key_states = self.k_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
RuntimeError: shape '[1, 374, 64, 128]' is invalid for input of size 382976

Inference-Code:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

checkpoint = "WizardLM/WizardCoder-Python-34B-V1.0"
device = "cuda" if torch.cuda.is_available() else "cpu"


class Model:
    def __init__(self):
        print("Running in " + device)
        self.tokenizer = AutoTokenizer.from_pretrained(checkpoint)
        self.model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map='auto')

    def infer(self, input_text, token_count):
        inputs = self.tokenizer.encode(input_text, return_tensors="pt").to(device)
        outputs = self.model.generate(inputs, max_new_tokens=token_count)
        return self.tokenizer.decode(outputs[0])