README.md

# starcoderbase-1b-GPTQ

Quantized starcoderbase-1b model to GPTQ format (4-bit precision) using [Auto-GPTQ](https://github.com/AutoGPTQ/AutoGPTQ).

[Quantization script](https://github.com/cosmo3769/Quantized-LLMs/blob/main/notebooks/quantize-starcoderbase-1b-4bit-gptq.ipynb)

## Benchmark 

[Benchmarking script](https://github.com/cosmo3769/Quantized-LLMs/blob/main/notebooks/llmbenchmark-starcoderbase-1b-lm-eval-harness.ipynb)

### Baseline starcoderbase-1b model (non-quantized)

|         Tasks         |Version|Filter|n-shot|    Metric     |Value |   |Stderr|
|-----------------------|-------|------|------|---------------|-----:|---|-----:|
|codexglue_code2text    |N/A    |none  |None  |smoothed_bleu_4|0.8767|±  |0.0592|
| - code2text_go        |      1|none  |None  |smoothed_bleu_4|1.0054|±  |0.0983|
| - code2text_java      |      1|none  |None  |smoothed_bleu_4|1.2158|±  |0.1657|
| - code2text_javascript|      1|none  |None  |smoothed_bleu_4|0.8560|±  |0.0429|
| - code2text_php       |      1|none  |None  |smoothed_bleu_4|0.9879|±  |0.0887|
| - code2text_python    |      1|none  |None  |smoothed_bleu_4|1.1950|±  |0.2819|
| - code2text_ruby      |      3|none  |None  |smoothed_bleu_4|0.0000|±  |0.0000|

|      Groups       |Version|Filter|n-shot|    Metric     |Value |   |Stderr|
|-------------------|-------|------|------|---------------|-----:|---|-----:|
|codexglue_code2text|N/A    |none  |None  |smoothed_bleu_4|0.8767|±  |0.0592|

|                    Tasks                    |Version|Filter|n-shot|  Metric   |Value|   |Stderr|
|---------------------------------------------|------:|------|------|-----------|----:|---|-----:|
|bigbench_code_line_description_generate_until|      1|none  |None  |exact_match|    0|±  |     0|

|                    Tasks                     |Version|Filter|n-shot|Metric|Value|   |Stderr|
|----------------------------------------------|------:|------|------|------|----:|---|-----:|
|bigbench_code_line_description_multiple_choice|      0|none  |None  |acc   | 0.15|±  |0.0465|

### Quantized starcoderbase-1b model to GPTQ format

|         Tasks         |Version|Filter|n-shot|    Metric     |Value |   |Stderr|
|-----------------------|-------|------|------|---------------|-----:|---|-----:|
|codexglue_code2text    |N/A    |none  |None  |smoothed_bleu_4|0.7959|±  |0.2180|
| - code2text_go        |      1|none  |None  |smoothed_bleu_4|0.9280|±  |0.0291|
| - code2text_java      |      1|none  |None  |smoothed_bleu_4|1.2112|±  |0.1703|
| - code2text_javascript|      1|none  |None  |smoothed_bleu_4|0.8848|±  |0.0391|
| - code2text_php       |      1|none  |None  |smoothed_bleu_4|0.6055|±  |0.6055|
| - code2text_python    |      1|none  |None  |smoothed_bleu_4|1.1460|±  |1.1460|
| - code2text_ruby      |      3|none  |None  |smoothed_bleu_4|0.0000|±  |0.0000|

|      Groups       |Version|Filter|n-shot|    Metric     |Value |   |Stderr|
|-------------------|-------|------|------|---------------|-----:|---|-----:|
|codexglue_code2text|N/A    |none  |None  |smoothed_bleu_4|0.7959|±  | 0.218|

|                    Tasks                    |Version|Filter|n-shot|  Metric   |Value|   |Stderr|
|---------------------------------------------|------:|------|------|-----------|----:|---|-----:|
|bigbench_code_line_description_generate_until|      1|none  |None  |exact_match|    0|±  |     0|

|                    Tasks                     |Version|Filter|n-shot|Metric|Value |   |Stderr|
|----------------------------------------------|------:|------|------|------|-----:|---|-----:|
|bigbench_code_line_description_multiple_choice|      0|none  |None  |acc   |0.1333|±  |0.0443|