| [Airoboros 13b GPT4 1.4](https://huggingface.co./jondurbin/airoboros-13b-gpt4-1.4) merged with kaiokendev's [SuperHOT 8k](https://huggingface.co./kaiokendev/superhot-13b-8k-no-rlhf-test) LoRA. | |
| The code to merge these can be found [here](https://files.catbox.moe/mg5v4g.py). Change information as needed. | |
| NOTE: This requires a monkey patch to work. FlashVenom has, along with kindly quantising this model to 4bit, added the monkeypatch file to their repo. You can access this [here](https://huggingface.co./flashvenom/Airoboros-13B-SuperHOT-8K-4bit-GPTQ). | |
| FROM THE ORIGINAL LORA MODEL CARD: | |
| This is a second prototype of SuperHOT, this time with 4K context and no RLHF. In my testing, it can go all the way to 6K without breaking down and I made the change with intention to reach 8K, so I'll assume it will go to 8K although I only trained on 4K sequences. | |
| In order to use the 8K context, you will need to apply the monkeypatch I have added in this repo -- without it, it will not work. The patch is very simple, and you can make the changes yourself: | |
| Increase the max_position_embeddings to 8192 to stretch the sinusoidal | |
| Stretch the frequency steps by a scale of 0.25 | |