upstage

/

SOLAR-10.7B-Instruct-v1.0

like 610

I get the error RuntimeError: cutlassF: no kernel found to launch!

Here is my code:
`base_model_id = "upstage/SOLAR-10.7B-Instruct-v1.0"

tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
base_model_id,
load_in_8bit=True,
device_map="auto",
trust_remote_code=True,
torch_dtype=torch.bfloat16,
)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)

output = pipe(
prompt,
max_new_tokens=1000,
do_sample=True,
temperature=0.1,
top_k=50,
top_p=0.95,
return_full_text=False,
)
text = output[0]["generated_text"]
print(text )`

Any suggestions?
I have 4 Tesla V100 GPUs