grimulkan
/

aurelian-v0.5-70b-rope8-32K-fp16

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

grimulkan commited on Jan 14

Commit

3c331cf

•

1 Parent(s): ede1eb7

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -23,7 +23,7 @@ license: unknown
 * [bfloat16](https://huggingface.co/grimulkan/aurelian-v0.5-70b-rope8-32K-fp16)
 * [EXL2 2.4bit]() fits in 1x24GB using Exllamav2 & 8-bit cache @ 10K context
 * [EXL2 4bit]() fits in 2x24GB (19/24) using Exllamav2 @ 16K context
-* [EXL2 6bit]() fits in 48GB+24GB (36/24 split) or 3x24GB (16/17/20 split) using Exllamav2 @ 32k context
 * [GGUF]()
 ### Examples

 * [bfloat16](https://huggingface.co/grimulkan/aurelian-v0.5-70b-rope8-32K-fp16)
 * [EXL2 2.4bit]() fits in 1x24GB using Exllamav2 & 8-bit cache @ 10K context
 * [EXL2 4bit]() fits in 2x24GB (19/24) using Exllamav2 @ 16K context
+* [EXL2 6bit](https://huggingface.co/grimulkan/aurelian-v0.5-70b-rope8-32K-6bpw_h8_exl2) fits in 48GB+24GB (36/24 split) or 3x24GB (16/17/20 split) using Exllamav2 @ 32k context
 * [GGUF]()
 ### Examples