Llamacpp Quantizations of Starling-LM-7B-beta

Using llama.cpp release b2440 for quantization.

Original model: https://huggingface.co./Nexusflow/Starling-LM-7B-beta

Download a file (not the whole branch) from below:

Filename Quant type File Size Description
Starling-LM-7B-beta-Q8_0.gguf Q8_0 7.69GB Extremely high quality, generally unneeded but max available quant.
Starling-LM-7B-beta-Q6_K.gguf Q6_K 5.94GB Very high quality, near perfect, recommended.
Starling-LM-7B-beta-Q5_K_M.gguf Q5_K_M 5.13GB High quality, very usable.
Starling-LM-7B-beta-Q5_K_S.gguf Q5_K_S 4.99GB High quality, very usable.
Starling-LM-7B-beta-Q5_0.gguf Q5_0 4.99GB High quality, older format, generally not recommended.
Starling-LM-7B-beta-Q4_K_M.gguf Q4_K_M 4.36GB Good quality, similar to 4.25 bpw.
Starling-LM-7B-beta-Q4_K_S.gguf Q4_K_S 4.14GB Slightly lower quality with small space savings.
Starling-LM-7B-beta-IQ4_NL.gguf IQ4_NL 4.15GB Good quality, similar to Q4_K_S, new method of quanting,
Starling-LM-7B-beta-IQ4_XS.gguf IQ4_XS 3.94GB Decent quality, new method with similar performance to Q4.
Starling-LM-7B-beta-Q4_0.gguf Q4_0 4.10GB Decent quality, older format, generally not recommended.
Starling-LM-7B-beta-IQ3_M.gguf IQ3_M 3.28GB Medium-low quality, new method with decent performance.
Starling-LM-7B-beta-IQ3_S.gguf IQ3_S 3.18GB Lower quality, new method with decent performance, recommended over Q3 quants.
Starling-LM-7B-beta-Q3_K_L.gguf Q3_K_L 3.82GB Lower quality but usable, good for low RAM availability.
Starling-LM-7B-beta-Q3_K_M.gguf Q3_K_M 3.51GB Even lower quality.
Starling-LM-7B-beta-Q3_K_S.gguf Q3_K_S 3.16GB Low quality, not recommended.
Starling-LM-7B-beta-Q2_K.gguf Q2_K 2.71GB Extremely low quality, not recommended.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

Downloads last month
1,063
GGUF
Model size
7.24B params
Architecture
llama

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Dataset used to train bartowski/Starling-LM-7B-beta-GGUF