ProphetOfBostrom
commited on
Commit
•
1b192a7
1
Parent(s):
4f670b0
Update README.md
Browse files
README.md
CHANGED
@@ -18,7 +18,10 @@ $ cat technical.utf8 multilingual.utf8 code.utf8 tiny.utf8 > techmulcodetiny.utf
|
|
18 |
reference to: [exllamav2/conversion/standard_cal_data](https://github.com/turboderp/exllamav2/tree/master/conversion/standard_cal_data) and [techmulcodetiny.utf8](./techmulcodetiny.utf8) produce a file that is used by imatrix for 560~ "chunks"
|
19 |
|
20 |
imatrix was run with default sampling settings besides the dataset (i think? i increased the batch number and reduced the batch size so i could cram on more layers but the generation should have been the same in the end)
|
21 |
-
(someone tell me why I was wrong to run imatrix with -cb continuous batching. shame me.)
|
|
|
|
|
|
|
22 |
|
23 |
how-to because i'm grouchy but I did actually want people to have these. Remember to replace IQ2_M (appears only twice, near the end) with whatever you fancy. Q2_K might be more compatible.
|
24 |
```
|
|
|
18 |
reference to: [exllamav2/conversion/standard_cal_data](https://github.com/turboderp/exllamav2/tree/master/conversion/standard_cal_data) and [techmulcodetiny.utf8](./techmulcodetiny.utf8) produce a file that is used by imatrix for 560~ "chunks"
|
19 |
|
20 |
imatrix was run with default sampling settings besides the dataset (i think? i increased the batch number and reduced the batch size so i could cram on more layers but the generation should have been the same in the end)
|
21 |
+
(someone tell me why I was wrong to run imatrix with -cb continuous batching. shame me.) (**UPDATE** found the command I used. use at your peril and obviously fix the paths)
|
22 |
+
```
|
23 |
+
imatrix -m Kyllene-57B-v1.0.q6_K.gguf -f ~/exltabbytorcher220/exllamav2/conversion/standard_cal_data/techmulcodetiny.utf8 -o Kyllene-57B-v1.0.q6_K.gguf.imat --verbosity 1 -ngl 50 -cb -t 3 -b 256 --no_mmap
|
24 |
+
```
|
25 |
|
26 |
how-to because i'm grouchy but I did actually want people to have these. Remember to replace IQ2_M (appears only twice, near the end) with whatever you fancy. Q2_K might be more compatible.
|
27 |
```
|