Add imatrix computing tips (#746)

- Add imatrix computing tips (0428d78df129e05b401baf36b98bf052f6b53c4d)
- Update with imatrix training dataset info (8507cc0203378620d46fbcc92af90ab225d7994b)
- Add link to llama-imatrix (1efd43b211b4a5f5d529f88a1a63b80720dc85f6)


Co-authored-by: Steven Goldfeather <[email protected]>

README.md

@@ -142,6 +142,35 @@ and then run another command which handles download/computation/upload. Most of
 to do stuff when things go wrong (which, with llama.cpp being so buggy and hard to use,
 is unfortunately very frequent).
 
+## What do I need to do to compute imatrix files for large models?
+
+Use [`llama-imatrix`](https://github.com/ggml-org/llama.cpp/blob/master/examples/imatrix/README.md) to compute imatrix files.
+
+### Hardware
+
+* RAM: A lot of RAM is required to compute imatrix files. Example: 512 GB is just enough to compute 405B imatrix quants in Q8.
+* GPU: At least 8 GB of memory.
+
+### Dataset
+
+* You want to create a dataset that is around double the size of bartowski1182's imatrix dataset. Quality is far more important
+than size. If you don't mind long training times, you can make it massive, but if you go beyond 1 MB there will
+probably be diminishing returns.
+* Your imatrix dataset should contain the typical output the model would generate when used for the workload you plan on using
+the model for. If you plan on using the model as a programming assistant, your imatrix dataset should contain the typical code
+you would ask it to write. The same applies for language. Our dataset is mostly English. If one would use our imatrix models in
+a different language they will likely perform worse than static quants as only a very small portion of our imatrix training data
+is multilingual. We only have the resources to generate single generic imatrix quants so our imatrix dataset must contain examples
+of every common use-case of an LLM.
+
+### Extra tips
+
+* Computing 405B imatrix quants in Q8 does not seem to have any noticeable quality impact compared to BF16, so to save on hardware
+requirements, use Q8.
+* Sometimes, a single node may not have enough RAM to compute the imatrix file. In such cases, `llama-rpc` inside llama.cpp can
+be used to combine the RAM/VRAM of multiple nodes. This approach takes longer: computing the 405B imatrix file in BF16 takes
+around 20 hours using 3 nodes with 512 GB, 256 GB, and 128 GB of RAM, compared to 4 hours for Q8 on a single node.
+
 ## Why don't you use gguf-split?
 
 TL;DR: I don't have the hardware/resources for that.