Spaces:
Sleeping
Apply for community grant: Space for Dolly 2.0
Hi
I've created a space for people to try out the latest dolly-v2 model. Would be really grateful if a A-100G large GPU could be allocated for this as a community grant :)
At the moment, the space is failing to build since the free hardware specs are not sufficient to load and run the model.
With A10G GPU, I'm getting an out of memory error while running the model even though to reduce memory usage I have loaded the model with bfloat16.
@shivi I ran your code without modification and it worked great. It only required just over 24 gigabytes of VRAM (24.42 GiB exact).
Hi @taesiri , thanks for taking a look and letting me know! And yes, looks like the model needs a little extra VRAM.
@shivi Have you tired loading the model in 8bit mode? It will only take 14Gb of VRAM.
tokenizer = AutoTokenizer.from_pretrained("databricks/dolly-v2-12b", padding_side="left")
model = AutoModelForCausalLM.from_pretrained("databricks/dolly-v2-12b", device_map="auto", load_in_8bit=True)