|
--- |
|
inference: false |
|
license: cc-by-nc-sa-4.0 |
|
language: |
|
- de |
|
- en |
|
library_name: transformers |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
# Orca Mini v2 German 7b GGML |
|
|
|
These files are GGML format model files for [Orca Mini v2 German 7b](https://huggingface.co./jphme/orca_mini_v2_ger_7b). Please find all information about the model in the original repository. |
|
|
|
|
|
GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as: |
|
* [text-generation-webui](https://github.com/oobabooga/text-generation-webui) |
|
* [KoboldCpp](https://github.com/LostRuins/koboldcpp) |
|
* [LoLLMS Web UI](https://github.com/ParisNeo/lollms-webui) |
|
* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) |
|
* [ctransformers](https://github.com/marella/ctransformers) |
|
|
|
|
|
## Prompt template: |
|
|
|
``` |
|
### System: |
|
You are an AI assistant that follows instruction extremely well. Help as much as you can. |
|
|
|
### User: |
|
prompt |
|
|
|
### Response: |
|
``` |
|
|
|
## Compatibility |
|
|
|
### `q4_0` |
|
|
|
So far, I only quantized a `q4_0` version for my own use. Please let me know if there is demand for other quantizations. |
|
These should be compatbile with any UIs, tools and libraries released since late May. |
|
|
|
## Provided files |
|
| Name | Quant method | Bits | Size | Max RAM required | Use case | |
|
| ---- | ---- | ---- | ---- | ---- | ----- | |
|
| orca-mini-v2-ger-7b.ggmlv3.q4_0.bin | q4_0 | 4 | 3.83 GB | ~6.3 GB | Original llama.cpp quant method, 4-bit. | |
|
|
|
**Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead. |
|
|
|
## How to run in `llama.cpp` |
|
|
|
I use the following command line; adjust for your tastes and needs: |
|
|
|
``` |
|
./main -t 10 -ngl 32 -m orca-mini-v2-ger-7b.ggmlv3.q4_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### System:\nYou are an story writing assistant who writes very long, detailed and interesting stories\n\n### User:\nWrite a story about llamas\n\n### Response:\n" |
|
``` |
|
If you're able to use full GPU offloading, you should use `-t 1` to get best performance. |
|
|
|
If not able to fully offload to GPU, you should use more cores. Change `-t 10` to the number of physical CPU cores you have, or a lower number depending on what gives best performance. |
|
|
|
Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration. |
|
|
|
If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins` |
|
|
|
## How to run in `text-generation-webui` |
|
|
|
Further instructions here: [text-generation-webui/docs/llama.cpp-models.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md). |
|
|
|
## Thanks |
|
|
|
Special thanks to [Pankaj Mathur](https://huggingface.co./psmathur) for the great Orca Mini base model and [TheBloke](https://huggingface.co./TheBloke) for his great work quantizing billions of models (and for his template for this README). |