Hugging Face's logo

bartowski
/
Meta-Llama-3.1-70B-Instruct-GGUF

Text Generation
Transformers
GGUF
PyTorch
facebook
meta
llama
llama-3
Inference Endpoints
Model card Files Community
8

<23GB quant for 24GB GPUs? IQ2_XS or IQ2_XXS?

#4
by kerrmetric - opened
Discussion
kerrmetric

Could I request a <23GB quant for 24GB GPUs? Q2_XS or IQ2_XXS should work great.

legraphista

Hey @kerrmetric , i may be able to help you with this one
https://huggingface.co./legraphista/Meta-Llama-3.1-70B-Instruct-IMat-GGUF#all-quants

bartowski
 Owner

@kerrmetric just added some more

Also can try out @legraphista , he does good work :)

kerrmetric

Thanks! Will do

Sign up or log in to comment