DeepSeek-R1-Distill-Qwen-32B-AWQ wint4

Distillation of DeepSeek-R1 to Qwen 32B, quantized using AWQ to wint4. It fits on any 24GB VRAM GPU or 32GB URAM device!

MMLU-PRO

The MMLU-PRO dataset evaluates subjects across 14 distinct fields using a 5-shot accuracy measurement. Each task assesses models following the methodology of the original MMLU implementation, with each having ten possible choices.

Measure

• Accuracy: Evaluated as "exact_match"

Shots

• Shots: 5-shot

Tasks

Tasks	Filter	n-shot	Metric	Value	Stderr
mmlu_pro	custom-extract		exact_match	0.5875	0.0044
biology	custom-extract	5	exact_match	0.7978	0.0150
business	custom-extract	5	exact_match	0.5982	0.0175
chemistry	custom-extract	5	exact_match	0.4691	0.0148
computer_science	custom-extract	5	exact_match	0.6122	0.0241
economics	custom-extract	5	exact_match	0.7346	0.0152
engineering	custom-extract	5	exact_match	0.3891	0.0157
health	custom-extract	5	exact_match	0.6345	0.0168
history	custom-extract	5	exact_match	0.6168	0.0249
law	custom-extract	5	exact_match	0.4596	0.0150
math	custom-extract	5	exact_match	0.6425	0.0130
other	custom-extract	5	exact_match	0.6223	0.0160
philosophy	custom-extract	5	exact_match	0.5731	0.0222
physics	custom-extract	5	exact_match	0.5073	0.0139
psychology	custom-extract	5	exact_match	0.7494	0.0154

Groups

Groups	Filter	n-shot	Metric	Value	Stderr
mmlu_pro	custom-extract		exact_match	0.5875	0.0044