Update with imatrix training dataset info
Browse files
README.md
CHANGED
@@ -149,6 +149,18 @@ is unfortunately very frequent).
|
|
149 |
* RAM: A lot of RAM is required to compute imatrix files. Example: 512 GB is just enough to compute 405B imatrix quants in Q8.
|
150 |
* GPU: At least 8 GB of memory.
|
151 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
152 |
### Extra tips
|
153 |
|
154 |
* Computing 405B imatrix quants in Q8 does not seem to have any noticeable quality impact compared to BF16, so to save on hardware
|
|
|
149 |
* RAM: A lot of RAM is required to compute imatrix files. Example: 512 GB is just enough to compute 405B imatrix quants in Q8.
|
150 |
* GPU: At least 8 GB of memory.
|
151 |
|
152 |
+
### Dataset
|
153 |
+
|
154 |
+
* You want to create a dataset that is around double the size of bartowski1182's imatrix dataset. Quality is far more important
|
155 |
+
than size. If you don't mind long training times, you can make it massive, but if you go beyond 1 MB there will
|
156 |
+
probably be diminishing returns.
|
157 |
+
* Your imatrix dataset should contain the typical output the model would generate when used for the workload you plan on using
|
158 |
+
the model for. If you plan on using the model as a programming assistant, your imatrix dataset should contain the typical code
|
159 |
+
you would ask it to write. The same applies for language. Our dataset is mostly English. If one would use our imatrix models in
|
160 |
+
a different language they will likely perform worse than static quants as only a very small portion of our imatrix training data
|
161 |
+
is multilingual. We only have the resources to generate single generic imatrix quants so our imatrix dataset must contain examples
|
162 |
+
of every common use-case of an LLM.
|
163 |
+
|
164 |
### Extra tips
|
165 |
|
166 |
* Computing 405B imatrix quants in Q8 does not seem to have any noticeable quality impact compared to BF16, so to save on hardware
|