Hugging Face's logo

InferenceIllusionist
/
Llama-3-70B-Instruct-Storywriter-iMat-GGUF

GGUF
Merge
llama3
iMat
Inference Endpoints
imatrix
conversational
Model card Files Community
2
Llama-3-70B-Instruct-Storywriter-iMat-GGUF / README.md
InferenceIllusionist's picture
InferenceIllusionist
Update README.md
1b17f50 verified
preview code
|
raw
history blame contribute delete
1.17 kB
metadata
tags:
  - merge
  - gguf
  - llama3
  - iMat

Llama-3-70B-Instruct-Storywriter-iMat-GGUF

Special request. Quantized from fp32 with love. If you can't fit IQ quants in your VRAM, try using the K quants in this repo instead.

  • The .imatrix file in this repo was created using the Q8_0 quantization of Llama-3-70B-Instruct-Storywriter-iMat-GGUF.
  • Calculated in 88 chunks with n_ctx=512 using groups_merged.txt

For a brief rundown of iMatrix quant performance please see this PR

All quants are verified working prior to uploading to repo for your safety and convenience.

Tip: Pick a file size under your GPU's VRAM while still allowing some room for context for best speed. You may need to pad this further depending on if you are running image gen or TTS as well.

BFloat16 model card can be found here