AIML-TUDA

/

LlavaGuard-v1.1-7B-hf

like 3

Follow

Artificial Intelligence & Machine Learning Lab at TU Darmstadt 32

I have not tried the 3090 jet, but it works fine on the H100 and A100 vhips. Maybe you can try quantization if the model doesn't fit.

But as discussed here inference on the 7b model should also work with 24gb vram.

Thanks for the quick reply! I thought the 7B-hf model should fit in the 3090-24GB using the supplied code, but will try 8-bit quantization.