---
license: apache-2.0
datasets:
- augmxnt/ultra-orca-boros-en-ja-v1
language:
- ja
- en
tags:
- jamba
- axolotl
---
Over the weekend after a failed initial run I got excited by Pete's success [Jamba Tuning](https://huggingface.co./lightblue/Jamba-v0.1-chat-multilingual) and decided to throw a little compute on a similar-sized dataset (the main [shisa-v1 bilingual tuning set](https://huggingface.co./datasets/augmxnt/ultra-orca-boros-en-ja-v1)).

Like my [initial runs](https://api.wandb.ai/links/augmxnt/h4mc4dd5), [training graphs look fine](https://wandb.ai/augmxnt/shisa-v2/runs/o830e1kw), but the results were less than spectacular.

Here are the JA MT-Bench evals for the 2416 checkpoint (eval/loss plateau) and the 4228 (3 epoch) tune:

```
shisa-jamba-v1-checkpoint-2416     2.491525
shisa-jamba-v1-checkpoint-4228     2.508475
```

You can view the answers in the repo (lots of repetitions and nonsense) and compare to [proper JA MT-Bench scores](https://github.com/AUGMXNT/shisa/wiki/Evals-:-JA-MT%E2%80%90Bench) from my testing.

While an "unsuccessful" experiment, it was still worth the practice, although I got a little excited and should have gone w/ my more typical lighter testing obviously.

This kicks off official `shisa-v2` base model evaluation. I was a bit hesitant about throwing this model out there (since it's useless as an artifact), but since I've actually made the in-process code available while working on it, I'll share this as well just in case (and to do this writeup).

Here is the current full code/steps for Axolotl training and eval (modified llm-judge inferencing code):
- https://github.com/shisa-ai/shisa-v2/tree/main/_base-evals/jamba/axolotl
- https://github.com/shisa-ai/shisa-v2/tree/main/_base-evals/jamba/eval

Thanks to Pete for the useful [initial report](https://wandb.ai/peterd/axolotl/reports/Jamba-chat-finetuning-Airoboros-and-ShareGPT---Vmlldzo3MzUwNTc1?accessToken=syabbmhblmwslnizuwws62tig3f0op75zh2jm2owbranaziq7thbgizl78nnowgc) and the [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) team for their [fast integration of Jamba](https://github.com/OpenAccess-AI-Collective/axolotl/issues/1449) (way better than my [raw tune code](https://github.com/shisa-ai/shisa-v2/tree/main/_base-evals/jamba)).