Upload README.md with huggingface_hub

README.md

@@ -16,7 +16,11 @@ This is a 2.5bpw EXL2 quant of [sophosympatheia/Midnight-Miqu-70B-v1.5](https://
 
 Details about the model and the merge info can be found at the above mode page.
 
-I have not extensively tested this quant/model other than ensuring I could load it and chat with it.
+## Prompt Templates
+
+Please see [sophosympatheia/Midnight-Miqu-70B-v1.5](https://huggingface.co/sophosympatheia/Midnight-Miqu-70B-v1.5) for Silly Tavern presets and templates.
+
+Further details on prompting this model will also pop up under the [model discussions](https://huggingface.co/sophosympatheia/Midnight-Miqu-70B-v1.0/discussions)
 
 ## Tavern Card
 
@@ -31,6 +35,73 @@ _In one hand, he holds a smoldering pipe filled with exotic herbs, casting a mys
 _The room is cluttered with tomes and scrolls, floating in midair as if held by invisible hands, creating a maelstrom of knowledge. Behind him, a crystal ball reflects swirling images of distant lands and times, while a cauldron bubbles with unknown concoctions on the hearth of an ancient fireplace. The scene exudes an air of enigma and might, leaving you both awestruck and slightly intimidated in the presence of this legendary figure from Thaylonia._
 
 
+## Perplexity Scoring
+
+Below are the perplexity scores for the EXL2 models. A lower score is better. 
+ 
+| Quant Level | Perplexity Score |
+|-------------|------------------|
+| 5.0 | 5.1226 |
+| 4.5 | 5.1590 |
+| 4.0 | 5.1772 |
+| 3.5 | 5.3030 |
+| 3.0 | 5.4156 |
+| 2.75 | 5.8717 |
+| 2.5 | 5.7236 |
+| 2.25 | 6.4102 |
+
+
+## EQ Bench
+
+Here are the EQ Bench scores for the EXL2 quants using Alpaca, ChatML, Mistral, Vicuna-v1.1 and Vicuna-v0 prompt templates. A higher score is better.
+
+| Quant Size | Alpaca | ChatML | Mistral | Vicuna-v0 | Vicuna-v1.1 |
+|------------|--------|--------|--------|--------|--------|
+| 5.0 | 77.38 | 76.25 | 77.67 | 78.83 | 77.86 |
+| 4.5 | 75.45 | 74.76 | 76.06 | 76.39 | 76.28 |
+| 4.0 | 77.99 | 75.25 | 77.18 | 76.84 | 76.08 |
+| 3.5 | 73.47 | 71.83 | 72.6 | 72.0 | 74.77 |
+| 3.0 | 71.46 | 70.33 | 71.06 | 72.75 | 72.21 |
+| 2.75 | 76.41 | 72.76 | 75.99 | 76.06 | 77.19 |
+| 2.5 | 74.61 | 74.78 | 75.58 | 74.2 | 75.55 |
+| 2.25 | 72.76 | 71.28 | 72.89 | 72.81 | 71.91 |
+
+
+### Perplexity Script
+
+This was the script used for perplexity testing.
+
+```bash
+#!/bin/bash
+
+# Activate the conda environment
+source ~/miniconda3/etc/profile.d/conda.sh
+conda activate exllamav2
+
+export CUDA_HOME=/home/mmealman/miniconda3/envs/exllamav2
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDA_HOME/lib
+export PATH=$PATH:$CUDA_HOME/bin
+
+# Set the model name and bit size
+MODEL_NAME="miqu-1-70b-sf"
+BIT_PRECISIONS=(5.0 4.5 4.0 3.5 3.0 2.75 2.5 2.25)
+
+# Print the markdown table header
+echo "| Quant Level | Perplexity Score |"
+echo "|-------------|------------------|"
+
+for BIT_PRECISION in "${BIT_PRECISIONS[@]}"
+do
+  MODEL_DIR="models/${MODEL_NAME}_exl2_${BIT_PRECISION}bpw"
+  if [ -d "$MODEL_DIR" ]; then
+    output=$(python test_inference.py -m "$MODEL_DIR" -gs 22,24 -ed data/wikitext/wikitext-2-v1.parquet)
+    score=$(echo "$output" | grep -oP 'Evaluation perplexity: \K[\d.]+')
+    echo "| $BIT_PRECISION | $score |"
+  fi
+done
+```
+
+
 ## Quant Details
 
 This is the script used for quantization.