File size: 861 Bytes
63aa0e4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
---
base_model:
- SicariusSicariiStuff/Wingless_Imp_8B
---
This is a converted weight from [Wingless_Imp_8B](https://huggingface.co./SicariusSicariiStuff/Wingless_Imp_8B) model in [unsloth 4-bit dynamic quant](https://archive.is/EFz7P) using this [collab notebook](https://colab.research.google.com/drive/1P23C66j3ga49kBRnDNlmRce7R_l_-L5l?usp=sharing).
## About this Conversion
This conversion uses **Unsloth** to load the model in **4-bit** format and force-save it in the same **4-bit** format.
### How 4-bit Quantization Works
- The actual **4-bit quantization** is handled by **BitsAndBytes (bnb)**, which works under **Torch** via **AutoGPTQ** or **BitsAndBytes**.
- **Unsloth** acts as a wrapper, simplifying and optimizing the process for better efficiency.
This allows for reduced memory usage and faster inference while keeping the model compact.
|