OddTheGreat

/

Apparatus_24B

like 3

Hi!

I seem to run into an issue using this model (might be a mistral 3 problem, not yet sure), but it seems that every time the model gets stuck on a specific phrase or ideas.
Example:

The model roleplays as a character who is self-doubting and has issues with her confidence. Initially the model picks up on that and runs with it as it should, portraying that feeling of isolation and anxiety. However, it quickly devolves into a one trick pony, where nuances are lost (I know it's not the card, as I have perfect results even with nemo), and character will begin sticking to certain phrases over and over again. It's very difficult to break it off and switch thinking patterns, unless I start to inject an author notes to pull it out of the same repeated phrases.

Is this something you have experienced?

Unfortunately this seems to be a general issue with Mistral small models (22B - 24B), but I noticed the settings also affect it. I can give you a tip on how to break out of the loop, but unfortunately the solution is not always so straightforward, because individual settings affect each other, so changing one setting for better may affect another setting for worse and vice versa, so you have to be careful if you're not well familiar with what each of the setting does.

So in order to break out of such loop, in general you may want to change the settings to boost creativity to help the model to introduce the changes by itself. You have various options to do so:

• Increase temperature - this should make the model explore new ideas rather than sticking to what was already said. Low values help the model to elaborate on "here and now" topics, but if this value is too low, the model will more likely repeat itself, so balance is necessary
• Increase repeat penalty - this will help the model avoid words and phrases used previously
• Increase min P parameter - this will make the model explore the same topics using different words and phrases which may help in breaking out from the loop, but probably not enough to do so by itself (changing other parameters may be needed as well).

Last but not least, you can temporarily switch to a different model, preferably of the same size. Different models even using the same settings presets will more likely break out of the loop instantly, because their training data is different.

Caution:

Beware that changing these parameters as suggested is actually telling the model to stop focusing on what was already said (INCLUDING the personality of the character itself!) so if you make some extreme changes, it has the potential to not only break out of the repetition loop, but also break out of the character personality as well which may not be really what you want!

Furthermore, too high values in each of these may introduce absurdities resulting in some crazy-sounding replies and ideas from the character, so again maintaining overall balance is necessary. If you feel like your current settings preset is more or less balanced for that specific character and you really just want to break out of the repetition loop, try changing these settings in very small steps individually and always generate new responses to get the gist of what will be different before moving on to making changes to a different parameter.

Good luck and enjoy.

Thanks! I actually have experience with parameters and kept this one at .7 on temp, due to recommendations from mistral team to keep it at low temperature. On a different Mistral small 3 fine-tune, temp of 1 made the model way too unpredictable and even messed up it's formatting. Hence, I lowered at .7, which so far as worked quite well. Min-p is kept at .05, but I can try to increase and see the results.

For me at least, I have not encountered this issue on 22B mistral, mostly on this one (I have to admit, I did not try vanilla or instruct) and tampering with params should mostly be done to squeeze something extra out of the model or add some extra flair or consistency. This is why I was curious if this is a problem of the finetune or base mistral small 3. Cydonia has a similar issue (which is part of the merge as far as I have seen), so I am not sure where the problem starts exactly. It's a damn shame too since it follows the card really well and is able to give me a good "back and forth" type of feeling.

I'll tweak it a bit more and compare it to other fine-tunes and vanilla, see if it's the settings, the base, or the merge that is keeping the mode going bonkers.

Oh i see that your T is 0.7. for roleplay it is way too low. Raise it, best temp for me was 1.01 or 1.1, with XTC 0.1 0.1

I'll try it out and see at 1.01 - 1.1 with XTC.

Frankly, I didn't talk about those more advanced samplers because I'm running LM Studio server and those settings are basically not available there in the user interface. I guess they are present in UIs like SillyTavern, but I haven't used SillyTavern in a long time, using plain old chub UI to interact with the characters is quite enough for quick RP sessions, but that one also lacks more advanced settings. SillyTavern changed significantly and I was lazy to update it lol

As for the breaking out of the loop, sometimes even using smaller models can solve the problem instantly. It's because they are so small that they are usually bad at following instructions, so the repetitions that stem from those instructions will likely be ignored in the first place. The drawback is that other instructions that may be also important may be also ignored, so it's either instant yay or nay. In any case, when you switch to a smaller model than you had before for the same chat, you should at least choose model which can handle the same context size, otherwise the model will not remember the old stuff and most including character personality itself.