eaddario
/

Llama-Guard-3-8B-GGUF

@@ -34,7 +34,7 @@ The process of [quantization](<https://huggingface.co/docs/optimum/en/concept_gu
 Another approach is to [prune](<https://en.wikipedia.org/wiki/Pruning_(artificial_neural_network>) the model, that is, to selectively zero-out groups of parameters. Although significant reductions can be achieved this way, the risk of severely degrading the model's performance is markedly higher than when quantizing, as the process requires a deep understanding of the model's architecture in order to identify which tensors can be safely zero'ed. For all means and purposes, pruning is the equivalent of lobotomizing the LLM!
-A successful outcome is when the overall size is reduced with no, or negligible, loss of capabilities (i.e. language understanding, math and logic problem-solving, conversation, coding, domain-specific knowledge, etc.) compared to the original version. On that regard, the method I'm using seems to yield some modest but encouraging results, and the versions available in this repo are on average **8% smaller** than other, high-quality, sources with negligible loss of capability. As I continue to improve the process and develop tools to automate it, I aim to achieve further reductions in the **10-15%** range, maybe more.
 For testing and comparison I'd normally use models produced by [Unsloth](<https://huggingface.co/unsloth>) ([Daniel and Michael Han](<https://unsloth.ai/>) do some really advanced level stuff!) and [Bartowski](<https://huggingface.co/bartowski>) (see credits below), but they don't provide GGUF versions of this model, so all tests and comparisons are done against naive quantizations obtained by simply running `llama-quantize` with no further optimization.

 Another approach is to [prune](<https://en.wikipedia.org/wiki/Pruning_(artificial_neural_network>) the model, that is, to selectively zero-out groups of parameters. Although significant reductions can be achieved this way, the risk of severely degrading the model's performance is markedly higher than when quantizing, as the process requires a deep understanding of the model's architecture in order to identify which tensors can be safely zero'ed. For all means and purposes, pruning is the equivalent of lobotomizing the LLM!
+A successful outcome is when the overall size is reduced with no, or negligible, loss of capabilities (i.e. language understanding, math and logic problem-solving, conversation, coding, domain-specific knowledge, etc.) compared to the original version. On that regard, the method I'm using seems to yield some modest but encouraging results, and the versions available in this repo are on average **7.5% smaller** than other, high-quality, sources with negligible loss of capability. As I continue to improve the process and develop tools to automate it, I aim to achieve further reductions in the **10-15%** range, maybe more.
 For testing and comparison I'd normally use models produced by [Unsloth](<https://huggingface.co/unsloth>) ([Daniel and Michael Han](<https://unsloth.ai/>) do some really advanced level stuff!) and [Bartowski](<https://huggingface.co/bartowski>) (see credits below), but they don't provide GGUF versions of this model, so all tests and comparisons are done against naive quantizations obtained by simply running `llama-quantize` with no further optimization.