First commit of GGML models.

README.md

@@ -0,0 +1,78 @@
+---
+datasets:
+- gozfarb/ShareGPT_Vicuna_unfiltered
+---
+
+# VicUnlocked-30B-LoRA GGML
+
+This is GGML format quantised 4-bit, 5-bit and 8-bit models of [Neko Institute of Science's VicUnLocked 30B LoRA](https://huggingface.co/Neko-Institute-of-Science/VicUnLocked-30b-LoRA).
+
+The files in this repo are the result of merging the above LoRA with the original LLaMA 30B, then converting to GGML for CPU (+ CUDA) inference using [llama.cpp](https://github.com/ggerganov/llama.cpp).
+
+## Repositories available
+
+* [4-bit, 5-bit and 8-bit GGML models for CPU inference](https://huggingface.co/TheBloke/VicUnlocked-30B-LoRA-GGML).
+* [4bit's GPTQ 4-bit model for GPU inference](https://huggingface.co/TheBloke/VicUnlocked-30B-LoRA-GPTQ).
+* [float16 HF format model for GPU inference and further conversions](https://huggingface.co/TheBloke/VicUnlocked-30B-LoRA-HF).
+
+## THESE FILES REQUIRE LATEST LLAMA.CPP (May 12th 2023 - commit b9fd7ee)!
+
+llama.cpp recently made a breaking change to its quantisation methods.
+
+I have quantised the GGML files in this repo with the latest version. Therefore you will require llama.cpp compiled on May 12th or later (commit `b9fd7ee` or later) to use them.
+
+## Provided files
+| Name | Quant method | Bits | Size | RAM required | Use case |
+| ---- | ---- | ---- | ---- | ---- | ----- |
+`VicUnlocked-30B-LoRA.ggml.q4_0.bin` | q4_0 | 4bit | 19GB | 21GB | 4-bit. |
+`VicUnlocked-30B-LoRA.ggml.q4_1.bin` | q4_1 | 5bit | 23GB | 25GB | 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
+`VicUnlocked-30B-LoRA.ggml.q5_0.bin` | q5_0 | 5bit | 21GB | 23GB | 5-bit. Higher accuracy, higher resource usage and slower inference. |
+`VicUnlocked-30B-LoRA.ggml.q5_1.bin` | q5_1 | 5bit | 23GB | 25GB | 5-bit. Even higher accuracy, and higher resource usage and slower inference. |
+`VicUnlocked-30B-LoRA.ggml.q8_0.bin` | q8_0 | 8bit | 35GB | 37GB | 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use. |
+
+## How to run in `llama.cpp`
+
+I use the following command line; adjust for your tastes and needs:
+
+```
+./main -t 8 -m VicUnlocked-30B-LoRA.ggml.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: write a story about llamas ### Response:"
+```
+
+Change `-t 8` to the number of physical CPU cores you have.
+
+## How to run in `text-generation-webui`
+
+GGML models can be loaded into text-generation-webui by installing the llama.cpp module, then placing the ggml model file in a model folder as usual.
+
+Further instructions here: [text-generation-webui/docs/llama.cpp-models.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md).
+
+
+# Original model card
+
+# Convert tools
+https://github.com/practicaldreamer/vicuna_to_alpaca
+
+# Training tool
+https://github.com/oobabooga/text-generation-webui
+
+ATM I'm using 2023.05.04v0 of the dataset and training full context.
+
+# Notes:
+So I will only be training 1 epoch, as full context 30b takes so long to train.
+This 1 epoch will take me 8 days lol but luckily these LoRA feels fully functinal at epoch 1 as shown on my 13b one.
+Also I will be uploading checkpoints almost everyday. I could train another epoch if there's enough want for it.
+
+Update: Since I will not be training over 1 epoch @Aeala is training for the full 3 https://huggingface.co/Aeala/VicUnlocked-alpaca-half-30b-LoRA but it's half ctx if you care about that. Also @Aeala's just about done.
+
+Update: Training Finished at Epoch 1, These 8 days sure felt long. I only have one A6000 lads there's only so much I can do. Also RIP gozfarb IDK what happened to him.
+
+# How to test?
+1. Download LLaMA-30B-HF if you have not: https://huggingface.co/Neko-Institute-of-Science/LLaMA-30B-HF
+2. Make a folder called VicUnLocked-30b-LoRA in the loras folder.
+3. Download adapter_config.json and adapter_model.bin into VicUnLocked-30b-LoRA.
+4. Load ooba: ```python server.py --listen --model LLaMA-30B-HF --load-in-8bit --chat --lora VicUnLocked-30b-LoRA```
+5. Select instruct and chose Vicuna-v1.1 template.
+
+
+# Training Log
+https://wandb.ai/neko-science/VicUnLocked/runs/vx8yzwi7

VicUnlocked-30B-LoRA.ggml.q4_0.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f8a58e8c12184da13347fb890ee4b5bd7e947b7826a5b6324cd22d62764a22b
+size 20333775232

VicUnlocked-30B-LoRA.ggml.q4_1.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6163a8dd4dbb9f9d233a757e81e5c03f2c2204f3f12bc7d4e7217c9e99f65d03
+size 24399792512

VicUnlocked-30B-LoRA.ggml.q5_0.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a88af1488ec82e985ba2ffe16c18a9132ed6a3eaffdf9c45632ca8b4c119248
+size 22366783872

VicUnlocked-30B-LoRA.ggml.q5_1.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8df5ff5281e3f1b66f957e00f5ac7862c680cbd1d267c034de9d98bd1c712bee
+size 24399792512

VicUnlocked-30B-LoRA.ggml.q8_0.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b07b66ef69e56560451f12e05d9917bff47e4b1fe875bcc9aa1f381aa252f54
+size 36597844352