File size: 3,226 Bytes
65d94c0 1415e85 b353e7a 65d94c0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 |
# Bigger Body 12b
A roleplay-focused pseudo full-finetune of Mistral Nemo Instruct.
The successor to the Ink series.
## Testimonials
> First impressions (temp 1, min-p .05-.1)
> - It passes my silly logic tests (read: me trolling random characters)
> - Haven't seen any slop yet
> - Writes short and snappy replies
> - ...yet not *too* short, like Mahou, and can write longer responses if the context warrants it
> - Follows card formatting instructions
>
> If this holds up to 16K it will be constantly in the hopper alongside Mag-Mell for me. I'm biased towards shorter responses with smarts. :)
\- Tofumagate
> tantalizing writing, leagues better then whatever is available online
\- Bowza
> Fun to use, nice swipe variation, gives me lots to RP off of. Rarely, it'll start to loop, but a quick swipe fixes no problem.
\- AliCat
## Dataset
The Bigger Body (referred to as Ink v2.1, because that's still the internal name) mix is absolutely disgusting. It's even more cursed than the original Ink mix.
<details>
<summary>(Public) Original Datasets</summary>
<!-- Start Generation Here -->
<ul>
<li><a href="https://huggingface.co./datasets/Fizzarolli/limarp-processed">Fizzarolli/limarp-processed</a></li>
<li><a href="https://huggingface.co./datasets/Norquinal/OpenCAI">Norquinal/OpenCAI</a> - <code>two_users</code> split</li>
<li><a href="https://huggingface.co./datasets/allura-org/Celeste1.x-data-mixture">allura-org/Celeste1.x-data-mixture</a></li>
<li><a href="https://huggingface.co./datasets/mapsila/PIPPA-ShareGPT-formatted-named">mapsila/PIPPA-ShareGPT-formatted-named</a></li>
<li><a href="https://huggingface.co./datasets/allenai/tulu-3-sft-personas-instruction-following">allenai/tulu-3-sft-personas-instruction-following</a></li>
<li><a href="https://huggingface.co./datasets/readmehay/medical-01-reasoning-SFT-json">readmehay/medical-01-reasoning-SFT-json</a></li>
<li><a href="https://huggingface.co./datasets/LooksJuicy/ruozhiba">LooksJuicy/ruozhiba</a></li>
<li><a href="https://huggingface.co./datasets/shibing624/roleplay-zh-sharegpt-gpt4-data">shibing624/roleplay-zh-sharegpt-gpt4-data</a></li>
<li><a href="https://huggingface.co./datasets/CausalLM/Retrieval-SFT-Chat">CausalLM/Retrieval-SFT-Chat</a></li>
<li><a href="https://huggingface.co./datasets/ToastyPigeon/fujin-filtered-instruct">ToastyPigeon/fujin-filtered-instruct</a></li>
</ul>
</details>
## Quants
TODO!
## Recommended Settings
Chat template: Mistral *v7-tekken* (NOT v3-tekken !!!! the main difference is that v7 has specific `[SYSTEM_PROMPT]` and `[/SYSTEM_PROMPT]` tags)
Recommended samplers (not the be-all-end-all, try some on your own!):
- Temp 1.25 / MinP 0.1
## Hyperparams
### General
- Epochs = 2
- LR = 1e-5
- LR Scheduler = Cosine
- Optimizer = [Apollo-mini](https://github.com/zhuhanqing/APOLLO)
- Optimizer target modules = `all_linear`
- Effective batch size = 16
- Weight Decay = 0.01
- Warmup steps = 50
- Total steps = 920
## Credits
Humongous thanks to the people who created the data. I would credit you all, but that would be cheating ;)
Big thanks to all Allura members for testing and emotional support ilya /platonic |