grimulkan
/

aurelian-v0.5-70b-rope8-32K-fp16

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

grimulkan commited on Jan 15

Commit

9d20845

•

1 Parent(s): 6e935c0

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -45,7 +45,7 @@ Examples are generated with the default Mirostat setting in Oobabooga, with `Mir
 * [EXL2 2.4bit](https://huggingface.co/grimulkan/aurelian-v0.5-70b-rope8-32K-2.4bpw_h6_exl2) fits in 1x24GB using Exllamav2 & 8-bit cache @ 10K context
 * [EXL2 4bit](https://huggingface.co/grimulkan/aurelian-v0.5-70b-rope8-32K-4.65bpw_h6_exl2) fits in 2x24GB (19/24) using Exllamav2 @ 16K context
 * [EXL2 6bit](https://huggingface.co/grimulkan/aurelian-v0.5-70b-rope8-32K-6bpw_h8_exl2) fits in 48GB+24GB (36/24 split) or 3x24GB (16/17/20 split) using Exllamav2 @ 32k context
-* [GGUF](https://huggingface.co/grimulkan/aurelian-v0.5-70b-rope8-32K_GGUF) Q4_K_M, Q5_K_M, Q6_K - Currently untested.
 ### Training Data

 * [EXL2 2.4bit](https://huggingface.co/grimulkan/aurelian-v0.5-70b-rope8-32K-2.4bpw_h6_exl2) fits in 1x24GB using Exllamav2 & 8-bit cache @ 10K context
 * [EXL2 4bit](https://huggingface.co/grimulkan/aurelian-v0.5-70b-rope8-32K-4.65bpw_h6_exl2) fits in 2x24GB (19/24) using Exllamav2 @ 16K context
 * [EXL2 6bit](https://huggingface.co/grimulkan/aurelian-v0.5-70b-rope8-32K-6bpw_h8_exl2) fits in 48GB+24GB (36/24 split) or 3x24GB (16/17/20 split) using Exllamav2 @ 32k context
+* [GGUF](https://huggingface.co/grimulkan/aurelian-v0.5-70b-rope8-32K_GGUF) Q4_K_M, Q5_K_M, Q6_K - Currently untested, please give feedback
 ### Training Data