Any chance/timeline for a q8 version?

by skyrien - opened Dec 16, 2023

Dec 16, 2023

This model is awesome, even the q5 version! Though with a RTX 4090, I can't quite run the fp16 version properly, and offloading any layers at all seems to break it entirely.

Any chance for a q8 version?

PsiPi

Owner Dec 18, 2023

Be about 8 hours.

PsiPi

Owner Dec 18, 2023

https://huggingface.co./PsiPi/liuhaotian_llava-v1.5-13b-GGUF/blob/main/llava-v1.5-13b-Q8_0.gguf enjoy

PsiPi changed discussion status to closed Dec 18, 2023

nawal2

Dec 29, 2023

This model is awesome, even the q5 version! Though with a RTX 4090, I can't quite run the fp16 version properly, and offloading any layers at all seems to break it entirely.

Any chance for a q8 version?

What sort of tokens/second do you get on the rtx 4090? how large are the images you're passing in?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment