mradermacher

/

model_requests

like 67

I can believe the energy reduction, but how did you compare the quality, and how does it compare to the Q4_0 that I provide? I'd need a compelling reason to replace Q4_0 by Q4_1, and an even more compelling reason to provide both. The idea behind providing Q4_0 was top provide a fast quant for slow computers. I suspect Q4_1 is simply slower but also a bit higher quality. It certainly is a lot bigger with very little quality increase.

In any case, @nicoboss is currently preparing a quite extensive (quality) benchmark over all quant types (including Q4_1). I will reevaluate all quants based on that as well.

@nicoboss while we are at it, another test I'd like to do on all quants is a speed test, to see how different quants work on cpu vs. gpu, possibly both prompt processing and inferencing. I don't thionk I cna put all this on the model page, but I could link to a guidance page, where people could get an idea of relative speeds vs. quality.

While I never performed any performance specific benchmarks as part of the quant quality measurement project so far, I did perform multiple evaluation benchmarks of each model from which I measured the quant quality. Thanks to the files creation and last modified time of the evaluation result I was able to extract the following quant performance measurements. The measurement includes running all the evaluation tests inside the benchmark. All tests were performed with the model being stored fully in RAM without offloading any layers to the GPU but using a GPU to improve prompt evaluation (-ngl 0). Further keep in mind that those are multiple choose benchmarks and so the result is more heavily skewed towards prompt evaluation than token generation than would be the case in most normal use cases.

Comparison between Q4_0 and Q4_1

Conclusion

It seems wrong that Q4_1 performs better than Q4_0. I actually saw the exact opposite. Q4_0 seemed to perform quite a lot better than Q4_1. I monitored the GPU energy consumption during those benchmarks and it was approximately the same no matter the quant so at least in these tests, longer time directly results in more energy consumption for the same task. I do not recommend replacing Q4_0 with Q4_1 from a performance perspective. Keep in mind that I have not considered quality during this comparison.

Future research

I want to come up with some performance tests that are mainly skewed towards token generation instead of prompt evaluation and want to run them CPU only, CPU with offloading computation to GPU (ngl -0), manually offloading layers to GPU, GPU memory RAM overflowing (GGML_CUDA_ENABLE_UNIFIED_MEMORY=1), GPU and using RPC.

Results of all quants

MMLU performance

ARC easy performance:

ARC challenge performance

Winogrande performance

Ah, the claim was that Q4_1 uses less energy than Q4_K_M ("for the quality"), which is a lot of variables. And is, however, is also not backed up by your benchmarks (assuming longer time means more energy usage), unless somehow "for the quality" figures in in favour of Q4_1, which, again seems to be not the case.

@yttria ,it seems Q4_1 is bigger, slower and worse than Q4_K_M, and not that much better than Q4_0, which is even faster.

PS: I didn't try to make you make these benchmarks, but I take them :-) However, they do seem a bit fishy - they follow more or less exactly the quant size, indicating a memory bottleneck, so cpu speed doesn't even figure in. I would assume yttria did it on a cpu with a lot fewer cores, where things can be dramatically different. Which is why I provide Q4_0 in the first place, as a fast quant for cpus. I hope one of the results on all this benchmarking (quality and speed) is to tell us once and for all if Q4_0 actually is useful (it might well be that another quant gives better a better quality/time ratio).

I didn't try to make you make these benchmarks, but I take them :-) However, they do seem a bit fishy - they follow more or less exactly the quant size, indicating a memory bottleneck, so cpu speed doesn't even figure in. I would assume yttria did it on a cpu with a lot fewer cores, where things can be dramatically different. Which is why I provide Q4_0 in the first place, as a fast quant for cpus. I hope one of the results on all this benchmarking (quality and speed) is to tell us once and for all if Q4_0 actually is useful (it might well be that another quant gives better a better quality/time ratio).

Great point. I'm aware that the shared performance test results are not perfect which is not surprising as I never indended thouse tests to be used to measure quant performance . I did all tests on a Threadripper PRO 7975WX (32 core 64 threads) while also using the GPU to do compuatations. This resulted in super fast prompt evaluation during prompt evaluation heavy tests. So thouse tests are indeed almost certainly memory bottlenecked which might not always be the case during normal use depending on the hardware. I'm wondering if there are really any realistic cases where LLMs are not memory bottlenecked. After just a few threads I start seeing deminishing returns when adding more. Usual consumer hardware has just dual channel memory and so should be bottelnacked even faster. I definately want to do more carefull testing regarding this and maybe even test on more mainstream hardware like my laptop. I actually already did many GGUF performance tests one year ago during the planing stage of my StormPeak build. I did so by changing the memory channels and memory clock-speed and amount of threads assigned to llama.cpp. The conclusion back then was pritty much the more memory channels and the faster memory I get the better the performance of GGUF files executed on the CPU will be. This was the main reason I decided to go for the much more expensive Threadripper PRO with octa-channel instead of the normal Threadripper lineup with quad-channel memory for my StormPeak node.

I agree with your methodology, these side effects were of course not the goal. There are lots of realistic cases where the cpu is the bottleneck, though. In fact, most cpus will be the bottleneck when confronted with IQ quants, and memory bottlenecked with normal Q quants. Which is totally in line with your experience (no IQ quants last year). There are even lots of systems where Q4_K_M might be a bottleneck, which is why I added Q4_0 quants back - purely for speed.

Whenever I get to my performance benchmarks, my plan is to make a 4-core baseline test for specifically slower cpus. There are a lot of people who run smaller models on not very beefy laptops, for example.

This is my test on M3 Max processing and generating a fixed number of tokens with a 7B model:

Prompt processing

Token generation

well, the Q4_0 seems to more energy-efficient than Q4_1, and is probably higher quality per bit, too

Another thing, I see you are providing f16 ggufs for some bf16 models. Wouldn't it be better to to convert directly to bf16 gguf to eliminate conversion loss?

since f16 has higher precision and weights should be mostly normalised, there shouldn't be any unless the model already has issues (and these issues would translate to other quants as well). the purpose of the f16 quant is not to provide a faithful representation of the source (which would be provided as SOURCE gguf) but to provide an actual f16 quant, i.e. pretty much the same purpose as the Q4_0, to provide a quant for certain targets.

f16 is for sure more useful then bf16 if you consider you require at least a Nvidia Ampere based GPU for it to support bf16 while f16 is already supported since Tegra X1 (Maxwell+) and so works on Pascal and Turin as and not just Ampere and Ada. This is also why back at university I used a Nintendo Switch console running Linux for scientific computation as my GTX 980 Ti Maxwell GPU did not support half precision. Ironically on the CPU side it is exactly the opposite picture: While bf16 is already widely supported, f16 is not.

There are some rare models where you can find f16, bf16 and SOURCE quants like this one: https://huggingface.co./mradermacher/Fook-Yi-34B-32K-v1-GGUF. SOURCE quants are usually only provided if obtaining the original model is not easily possible. For example, if a model got deleted by the author after mradermacher already downloaded it and he notices it before deleting them.

This is my test on M3 Max processing and generating a fixed number of tokens

Thanks a lot for sharing your measurements. What application did you use to create them? Probably a macOS thing as there seems no simple way for me to measure power consumption used for generating tokens unless I do everything on the GPU.

@nicoboss your cpu should have a power meter, try /sys/class/powercap/intel-rapl/*/energy_uj - might even have one per core. don't know how good the amd implementation is but the intel estimate is usually pretty good, and amd usually does better.

What application did you use to create them?

Energy is measured with the built-in powermetrics utility on macOS.

your cpu should have a power meter, try /sys/class/powercap/intel-rapl/*/energy_uj

That is really cool. Wasn't aware of it. AMDs intel-rapl implementation is decent. There also is amd_energy which is so accurate it got kicked out of the linux kernel due to security concerns regarding side-channel attacks but I can just build and load it as a kernel module.

After more testing with 70B, I think Q4_1 is really worth it for Mac. Here is a comparison of the most efficient quants on Mac. PP and TG energy usage are normalized to q4_0 on high-power mode for a fixed number of tokens. As you can see, q4_1 gives almost all of the perplexity reduction of q4_k with almost none of the performance impact.

High-power mode

Low-power mode

I'm confused @yttria , doesn't that last chart directly contradict your conclusion? Q4_K has better perplexity and better speed per Joule than q4_0 and q4_1

I'm confused @yttria , doesn't that last chart directly contradict your conclusion? Q4_K has better perplexity and better speed per Joule than q4_0 and q4_1

Sorry for the confusion. The numbers are all in joules per token, normalized to q4_0 on high-power mode. Less joules per token is better.

(just saw your update)

And you're using Metal for this? I'm very surprised.

Also what are you using for your perplexity that's getting such low results? They confuse me since typically the change from Q4_0 -> Q4_1 is the same as Q4_1 to Q4_K, so I would expect to see 2.5 for Q4_K

@yttria In the past month I spent weeks performing performance measurements of all quants on many different hardware configurations and llama.cpp backends. You can find my raw results under https://www.nicobosshard.ch/perfData.zip and quality measurements under http://www.nicobosshard.ch/LLM-Eval_Quality_v1.tar.zst

Even on ARM (but not Apple silicon) I saw much better performance using Q4_0 or Q4_K_S and even Q4_K_M compared to Q4_1.

| model                          |       size |     params | backend    | threads |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
| qwen2 3B Q4_0                  |   1.69 GiB |     3.09 B | CPU        |       4 |         pp128 |          3.82 ± 0.00 |
| qwen2 3B Q4_0                  |   1.69 GiB |     3.09 B | CPU        |       4 |         tg128 |          1.91 ± 0.00 |
| qwen2 3B Q4_K - Small          |   1.70 GiB |     3.09 B | CPU        |       4 |         pp128 |          4.04 ± 0.00 |
| qwen2 3B Q4_K - Small          |   1.70 GiB |     3.09 B | CPU        |       4 |         tg128 |          1.88 ± 0.00 |
| qwen2 3B Q4_K - Medium         |   1.79 GiB |     3.09 B | CPU        |       4 |         pp128 |          3.89 ± 0.00 |
| qwen2 3B Q4_K - Medium         |   1.79 GiB |     3.09 B | CPU        |       4 |         tg128 |          1.82 ± 0.00 |
| qwen2 3B Q4_1                  |   1.85 GiB |     3.09 B | CPU        |       4 |         pp128 |          3.73 ± 0.00 |
| qwen2 3B Q4_1                  |   1.85 GiB |     3.09 B | CPU        |       4 |         tg128 |          1.78 ± 0.00 |

Not only that but according the month I spend doing quality measurements (whose final quality numbers are listed under https://hf.tst.eu/model#Qwen2.5-3B-i1-GGUF) you can see that Q4_1 has by far the worst quality of all Q4 quants despite being the largest Q4 quant unless you use imatrix/wighted quants:

And you're using Metal for this? I'm very surprised.

Yes, this is with Metal. It is calculated by dividing Watts from powermetrics (numerator) by T/s from llama-bench (denominator). W/(T/s) simplifies to Joules per token, which is then normalized. Joules per token is a very important metric for MacBooks because of inherent thermal constraints.

Why are you surprised by these results? My interpretation is that K-quants require more complex calculations, and therefore more energy.

Please see here for a comparison of all quants q4 and above with a 7B model:
https://huggingface.co./mradermacher/model_requests/discussions/299#66ef7cdc3b250e9eca0a5e32

Also what are you using for your perplexity that's getting such low results? They confuse me since typically the change from Q4_0 -> Q4_1 is the same as Q4_1 to Q4_K, so I would expect to see 2.5 for Q4_K

The perplexity is measured with wiki.test.raw on Meta-Llama-3.1-70B-Instruct quantized with an imatrix made with calibration_datav3.txt. I did not complete the full tests because it takes a long time for 70B models, so I only averaged the results obtained from the first parts of the tests. Please let me know if it would be useful to perform the tests completely.

Here the full data from http://www.nicobosshard.ch/LLM-Eval_Quality_v1.tar.zst as table. I averaged together the measurements of the base and instruct model to get more accurate results better representing values of derivative models.

Legend

Quant: Used quant
Rank: Ranking position of the quality of a specific quant compared to other quants based on the ranking of the average of rank with 25% KL-divergence, 40% Correct token, 20% Same token and 15% Perplexity
KL-divergence: 100 - Mean KLD * 100
Correct token: Mean Δp + 100
Same token: Same top p
Perplexity: 100 + (100 - (Mean PPL(Q)/PPL(base)) * 100)
Eval: Weighted average [n questions](ARC Easy(Q)/ARC Easy(base) , ARC Challenge(Q)/ARC Challenge(base), MMLU(Q)/MMMLU(base), WinoGrande(Q)/WinoGrande(base))

Qwen2.5-0.5B & Qwen2.5-0.5B-Instruct

Qwen2.5-1.5B & Qwen2.5-1.5B-Instruct

Qwen2.5-3B & Qwen2.5-3B-Instruct

Qwen2.5-7B & Qwen2.5-7B-Instruct

Qwen2.5-14B & Qwen2.5-14B-Instruct

Qwen2.5-32B & Qwen2.5-32B-Instruct

Qwen2.5-72B-Instruct

@nicoboss It is interesting to see that Apple Silicon may be special in requiring more energy for K-quants. However, this should not diminish the value provided by traditional quants, especially since Apple Silicon is a popular platform for LLMs, and their machines are a relatively easy way to get large amounts of fast memory.

I would also like to point out that Q4_1 being the largest Q4 quant is not necessarily a downside. With large amounts of unified memory available, the larger size is not usually an issue, unlike with graphics cards. The main limitation is actually energy efficiency because of thermal constraints, which is why I specifically tested for Joules per token.

Let's look on some CUDA performance numbers measured on an AMD Ryzen Threadripper 1950X with the entire model offloaded to an RTX 2070s. Even here Q4_1 performance seems quite disappointing:

Let's finally look on some AMD laptop with an 7840s SoC performance which only has an integrated GPU. Even here Q4_1 didn't realy convince me. And this SoC is fully power limited so performance = power efficiency.

Offload

No offload:

CPU

@nicoboss Again, you are measuring on different platforms with completely different architectures. Your insistence that Q4_1 is disappointing does not disprove the value it has on Apple Silicon. GPU users have so many quants available to choose from depending on what fits in their VRAM, why should Apple Silicon users not be provided with the same choice based on what works best for their machines? Especially now that Q4_0_4_4, Q4_0_4_8 and Q4_0_8_8 are no longer necessary, it would not be a strain of resources to replace three quants with one.

I understand anti-Apple sentiment but restricting user choice is not the answer.

@yttria Can you please run the following on Q4_0, Q4_K_S and Q4_1? I would expect performance of Q4_1 to be better than Q4_0 and Q4_S on your Apple silicon should it really be able to process Q4_1 so much more efficiently. Please use latest llama.cpp which includes online conversion to optimize some Q4 quants to your CPU.

./llama-bench -m $modelpath -v --prio 3 -t 4,6,8,12,16 > ./perfOut.txt

Especially now that Q4_0_4_4, Q4_0_4_8 and Q4_0_8_8 are no longer necessary, it would not be a strain of resources to replace three quants with one.

@yttria I though online repacking as implemented in https://github.com/ggerganov/llama.cpp/pull/10446 uses Q4_0 and not Q4_1 as base. If online repacking really does use Q4_1 it would obviously instantly get added because ARM/RISC-V optimization is a huge deal.

llama.cpp now having online repacking is also one of the reasons why I'm quite skeptical that Q4_1 is really better on apple silicon compared to an online repacked ARM optimised Q4_0.

I understand anti-Apple sentiment but restricting user choice is not the answer.

I'm not anti-Apple but before adding a new quant we need to be sure it really is significantly better on Apple silicon because it seems much worse for every other hardware. I fully support adding Q4_1 quants if they turn out to be the best choice for Apple silicon users.

@nicoboss online repacking only uses Q4_0 and shouldn't be used on Apple silicon anyways since Metal will always be better

I again checked the quality table on https://hf.tst.eu/model#Qwen2.5-3B-i1-GGUF and using imatirx/wighted quants i1-Q4_1 has a quality 87 which is equal to i1-Q4_K_S. While still being worse in a quality/size ratio it is a massive improvement compared to Q4_1 static quants and so seams worth it to be added for imatrix quants if Q4_1 is the best choice for Apple silicon users.

I don't have time to test right now (will probably in the next few days), but I found this person also did tests on Apple Silicon.

https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/

I don't quite agree with his idea of taking pp * tg / ppl, so I plotted pp * tg against ppl with his data. In the graph below, the top left corner is optimal. It is clear that Q4_K and Q4_K_S are worse than the other Q4 quants.

Note that he didn't test power consumption, and from my tests Q4_0 and Q4_1 use much less power than both K-quants and I-quants. So taking power consumption into account, Q4_0 and Q4_1 are the best quants for Mac, with Q4_1 using ~4% more power but giving better perplexity.

Now let's see if it's worth paying 4% more for Q4_1.

I found the only reasonable quants to use on Mac are Q4_0, Q4_1 and Q8_0.

According to https://hf.tst.eu, quality of quants are as follows. √(Js)/T values are from my previous test of 7B models.

Q4_1 has 3 more quality than Q4_0 for only 4% more √(Js)/T, meaning each quality costs 1.3%. Q8_0 has 12 more than Q4_1 but requires 25% more √(Js)/T, with each quality costing 2.1%. Therefore, it is worth paying 4% more √(Js)/T for Q4_1 rather than Q4_0.

Here is the full plot of all quants q4 and above. Top left is optimal. Only Q4_0, Q4_1 and Q8_0 are on the frontier.

Q4_1 has 3 more quality than Q4_0 for only 4% more √(Js)/T, meaning each quality costs 1.3%. Q8_0 has 12 more than Q4_1 but requires 25% more √(Js)/T, with each quality costing 2.1%. Therefore, it is worth paying 4% more √(Js)/T for Q4_1 rather than Q4_0.

Keep in mind that this only applies for imatrix/weighted quants. For static quants Q4_1 has with only 80 the worst quality of all Q4 quants even getting beaten by Q4_0 scoring 81 quality. In this case the 4% more power required for it makes it objectively a worse option compared to static Q4_0.

Here is the full plot of all quants q4 and above. Top left is optimal. Only Q4_0, Q4_1 and Q8_0 are on the frontier.

This plot is very convincing and shows how imatrix/weighted i1-Q4_1 and i1-Q4_K_S achieve the same quality while i1-Q4_1 using significantly less power on Apple silicon. I just hope Q4_1 is also faster than Q4_0 on Apple silicon but it probably will be given that you mentioned that the SoC is thermally limited.

@mradermacher With all the data provided so far I recommend adding i1-Q4_1 to imatrix/weighted quants but recommend against adding Q4_1 to static quants.

Wow, thanks to all of you for your data driven arguments, which are the second-best arguments around. Let's say you all convinced me:

The original reason we dropped the Q4_1 quant was its erratic performance, not unlike the IQ3 static quants. I think we have some good evidence that the worst issues with it have gone in the last two months, especially with the imatrix one. As such, I am strongly contemplating bringing back the static iq3 quants, knowing that their good qwen results might have been a fluke.

That the arm quants are gone is not an argument in favour of Q4_1 per se, but it makes the decision to re-add it easier (because I'd like to avoid outright wasting of resources, which was a strong factor against the arm quants. Good riddance).

That apple is a sucky company and should not be supported is very true, IMnsHO, and so are nvidia, google, and also intel, amd, microsoft, oracle... Now, SGI and Cray on the other hand, those were pretty cool companies!

(And now I am concerned that IQ4_NL actually looks quite good, at least on apple :)

That Q4_1 is computationally efficient to quantize, comparatively, also makes this easier.

That size doesn't matter much on apple is surprising to me - yes, memory bandwidth is higher, but Q4_1 is simple enough for it to still be memory-bandwidth-limited, one would naively assume.

So long story short, I've added Q4_1 to the list of imatrix quants to generate. It's not ideal, because I am still worried that good Q4_1 results might be flukes, but barring even more data, this seems indeed the correct decision. I don't think I want a static Q4_1 (unless I only generate static quants for a model, but I do not have logic for that in place yet).

And if somebody wants to add data for more varied models, I wouldn't be unhappy :)

I hope that flies well with everybody.

And now I am concerned that IQ4_NL actually looks quite good, at least on apple :)

@mradermacher IQ4_NL is relatively important one due to https://github.com/ggerganov/llama.cpp/pull/10541 which implements runtime repack into IQ4_NL_4_4 for ARM NEON superseding the never merged IQ4_NL_X_X quants described in https://github.com/ggerganov/llama.cpp/pull/10196": "Q4_0_X_X is very fast but the accuracy of Q4_0 is not good. IQ4_NL is much better than Q4_0 and they have compatible structure. Therefore, I introduce IQ4_NL_X_X to have benefits of both.". If you look at the performance results posted in https://github.com/ggerganov/llama.cpp/pull/10196 I have to say I'm really impressed with IQ4_NL_4_4. I recommend we provide weighted/imatrix IQ4_NL quants for small models similar to how we did for ARM quants.

for small models

Your argument overall is persuasive, but indeed, for small models the barrier is much lower. I'll implement that as soon as I can.

IQ4_NL should be done, currently at <= 18B.