If you're on a Mac, many models like this won't work, I'm pretty sure. See https://github.com/databrickslabs/dolly/issues/60

I am on Ubuntu

They just updated their documentation and sample code.
New sample code is working fine at my end now :

Working code :
import torch
from transformers import pipeline

generate_text = pipeline(model="databricks/dolly-v2-12b", torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto")

Great! Happy the updated code fixed your issue.

does not work for me. I am on ubuntu. how is it working for others?

not working for me as well

what exact problem are you facing?

When i use device_map='auto' code works fine when i use cuda , it gives error below, i am on windows 10 with rtx2080 and cuda tool kit is already installed.

import torch
from transformers import pipeline
import time

use dolly-v2-12b if you're using Colab Pro+, using pythia-2.8b for Free Colab

generate_text = pipeline(
model="databricks/dolly-v2-2-8b",
torch_dtype=torch.bfloat16,
trust_remote_code=True,
device_map="cuda"
)

def prompt(s):
start = time.time()
response = generate_text(s)
end = time.time()
print(end - start)
return response

print(prompt("First man on the moon?"))

The argument trust_remote_code is to be used with Auto classes. It has no effect here and is ignored.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ c:\Users\Ehsan\Desktop\opengpt4alltest1.py:77 in │
│ │
│ 74 from transformers import pipeline │
│ 75 import time │
│ 76 # use dolly-v2-12b if you're using Colab Pro+, using pythia-2.8b for Free Colab │
│ ❱ 77 generate_text = pipeline( │
│ 78 │ model="databricks/dolly-v2-2-8b", │
│ 79 │ torch_dtype=torch.bfloat16, │
│ 80 │ trust_remote_code=True, │
│ │
│ C:\Users\Ehsan\AppData\Local\Programs\Python\Python39\lib\site-packages\transformers\pipelines_ │
│ init_.py:779 in pipeline │
│ │
│ 776 │ # Forced if framework already defined, inferred if it's None │
│ 777 │ # Will load the correct model if possible │
│ 778 │ model_classes = {"tf": targeted_task["tf"], "pt": targeted_task["pt"]} │
│ ❱ 779 │ framework, model = infer_framework_load_model( │
│ 780 │ │ model, │
│ 781 │ │ model_classes=model_classes, │
│ 782 │ │ config=config, │
│ │
│ C:\Users\Ehsan\AppData\Local\Programs\Python\Python39\lib\site-packages\transformers\pipelines\b │
│ ase.py:271 in infer_framework_load_model │
│ │
│ 268 │ │ │ │ continue │
│ 269 │ │ │
│ 270 │ │ if isinstance(model, str): │
│ ❱ 271 │ │ │ raise ValueError(f"Could not load model {model} with any of the following cl │
│ 272 │ │
│ 273 │ framework = "tf" if "keras.engine.training.Model" in str(inspect.getmro(model.__clas │
│ 274 │ return framework, model │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Could not load model databricks/dolly-v2-2-8b with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>,
<class 'transformers.models.auto.modeling_tf_auto.TFAutoModelForCausalLM'>, <class 'transformers.models.gpt_neox.modeling_gpt_neox.GPTNeoXForCausalLM'>).

There are several things wrong here - old model version string and device_map="cuda" doesn't quite make sense. Please start over from the given example in the model card. This is also a different discussion now.

not sure if there was a solution for it yet?