Hugging Face's logo

Trelis
/
Llama-2-7b-chat-hf-hosted-inference-8bit

Text Generation
Transformers
PyTorch
Safetensors
English
llama
facebook
meta
llama-2
hosted inference
8 bit
8bit
8-bit precision
text-generation-inference
Inference Endpoints
Model card Files Community
3

Quantization method?

#3
by monology - opened
Discussion
monology

Would like to publish other 8-bit models for use with hosted inference - instructions to reproduce this quantization method would be appreciated.

RonanMcGovern
 Trelis org

Howdy @monology , try this: https://colab.research.google.com/drive/14wFCJnF0Hd8Y6AsCRgivQAuJVYy9b5fA?usp=sharing

RonanMcGovern changed discussion status to closed

Sign up or log in to comment