Greetings! I've decided to share some insight that I've accumulated over the few years I've been toying around with LLMs, and the intricacies of how to potentially make them run better for creative writing or roleplay as the focus, but it might also help with technical jobs too.
These might not be applicable with every model or user case, nor would it guarantee the best possible response with every single swipe, but it should help increase the odds of getting better mileage out of your model and experience, even if slightly, and help you avoid some bad or misled advice, which I personally have had to put up with. Some of this will be retreading old ground if you are already privy, but I will try to include less obvious stuff as well. Remember, I still consider myself a novice in some areas, and am always open to improvement.
This list will probably be updated periodically.
1: Instruct Formatting:
1-1 What is the Instruct Template?
The Instruct Template/Format is probably the most important aspect when it comes to getting a model to work properly, as it is what encloses the training data with tokens that were used for the model, and your chat with said model. Some of them are used in a more general sense and are not brand specific, such as ChatML or Alpaca, while others stick to said brand, like Llama3 Instruct or Mistral Instruct. However not all models that are brand specific with their formatting will be trained with their own personal template.
Its important to find out what format/template a model uses before booting it up, and you can usually check to see which it is on the model page. If a format isn't directly listed on said page, then there is ways to check internally with the local files. Each model has a tokenizer_config file, and sometimes even a special_tokens file, inside the main folder. As an example of what to look for, If you see something like a Mistral brand model that has im_start/im_end inside those files, then chances are that the person who finetuned it used ChatML tokens in their training data. Familiarizing yourself with the popular tokens used in training will help you navigate models better internally, especially if a creator forgets to post a readme on how it's suppose to function.
1-2 Is there any reason not to use the prescribed format/template?
Sticking to the prescribed format will give your model better odds of getting things correct, or even better prose quality. But there are some small benefits when straying from the model's original format, such as supposedly being less censored. However the trade-off when it comes to maximizing a model's intelligence is never really worth it, and there are better ways to get uncensored responses with better prompting, or even tricking the model by editing their response slightly and continuing from there.
From what I've found when testing models, if someone finetunes a model over the company's official Instruct focused model, instead of a base model, and doesn't use the underlining format that it was made with (such as ChatML over Mistral's 22B model as an example) then performance dips will kick in, giving less optimal responses then if it was instead using a unified format.
This does not factor other occurrences of poor performance or context degradation when choosing to train on top of official Instruct models which may occur, but if it uses the correct format, and/or is trained with DPO or one of its variance (this one is more anecdotal, but DPO/ORPO/Whatever-O seems to be a more stable method when it comes to training on top of per-existing Instruct models) then the model will perform better overall.
1-3 What about models that list multiple formats/templates?
This one is more due to model merging or choosing to forgo an Instruct model's format in training, although some people will choose to train their models like this, for whatever reason. In such an instance, you kinda just have to pick one and see what works best, but the merging of formats, and possibly even models, might provide interesting results, but only if its agreeable with the clutter on how you prompt it yourself. What do I mean by this? Well, perhaps its better if I give you a couple anecdotes on how this might work in practice...
Nous-Capybara-limarpv3-34B is an older model at this point, but it has a unique feature that many models don't seem to implement; a Message Length Modifier. By adding small/medium/long at the end of the Assistant's Message Prefix, it will allow you to control how long the Bot's response is, which can be useful in curbing rambling, or enforcing more detail. Since Capybara, the underling model, uses the Vicuna format, its prompt typically looks like this:
System:
User:
Assistant:
Meanwhile, the limarpv3 lora, which has the Message Length Modifier, was used on top of Capybara and chose to use Alpaca as its format:
### Instruction:
### Input:
### Response: (length = short/medium/long/etc)
Seems to be quite different, right? Well, it is, but we can also combine these two formats in a meaningful way and actually see tangible results. When using Nous-Capybara-limarpv3-34B with its underling Vicuna format and the Message Length Modifier together, the results don't come together, and you have basically 0 control on its length:
System:
User:
Assistant: (length = short/medium/long/etc)
The above example with Vicuna doesn't seem to work. However, by adding triple hashes to it, the modifier actually will take effect, making the messages shorter or longer on average depending on how you prompt it.
### System:
### User:
### Assistant: (length = short/medium/long/etc)
This is an example of where both formats can work together in a meaningful way.
Another example is merging a Vicuna model with a ChatML one and incorporating the stop tokens from it, like with RP-Stew-v4. For reference, ChatML looks like this:
<|im_start|>system
System prompt<|im_end|>
<|im_start|>user
User prompt<|im_end|>
<|im_start|>assistant
Bot response<|im_end|>
One thing to note is that, unlike Alpaca, the ChatML template has System/User/Assistant inside it, making it vaguely similar to Vicuna. Vicuna itself doesn't have stop tokens, but if we add them like so:
SYSTEM: system prompt<|end|>
USER: user prompt<|end|>
ASSISTANT: assistant output<|end|>
Then it will actually help prevent RP-Stew from rambling or repeating itself within the same message, and also lowering the chances of your bot speaking as the user. When merging models I find it best to keep to one format in order to keep its performance high, but there can be rare cases where mixing them could work.
1-4 Are stop tokens necessary?
In my opinion, models work best when it has stop tokens built into them. Like with RP-Stew, the decrease in repetitive message length was about 25~33% on average, give or take from what I remember, when these end tokens are added. That's one case where the usefulness is obvious. Formats that use stop tokens tend to be more stable on average when it comes to creative back-and-forths with the bot, since it gives it a structure that's easier for it to understand when to end things, and inform better on who is talking.
If you like your models to be unhinged and ramble on forever (aka; bad) then by all means, experiment by not using them. It might surprise you if you tweak it. But as like before, the intelligence hit is usually never worth it. Remember to make separate instances when experimenting with prompts, or be sure to put your tokens back in their original place. Otherwise you might end up with something dumb, like putting the stop token before the User in the User prefix.
2: Character/Chat Formatting
2-1 What is a Character Card?
Lets get the obvious thing out of the way. Character Cards are basically personas of, well, characters, be it from real life, an established franchise, or someone's OC, for the AI bot to impersonate and interact with. The layout of a Character Card is typically written in the form of a profile or portfolio, with different styles available for approaching the technical aspects of listing out what makes them unique.
2-2 What are the different styles of Character Cards?
Making a card isn't exactly a solved science, and the way its prompted could vary the outcome between different model brands and model sizes. However, there are a few that are popular among the community that have gained traction.
One way to approach it is a simply writing out the character's persona like you would in a novel/book, using natural prose to describe their background and appearance. Though this method would require a deft hand/mind to make sure it flows well and doesn't repeat too much with specific keywords, and might be a bit harder compered to some of the other styles if you are just starting out. More useful for pure writers, probably.
Another is doing a list format, where every feature is placed out categorically and sufficiently. There are different ways of doing this as well, like markdown, wiki style, or the community made W++, just to name a few.
Some use parentheses or brackets to enclose each section, some use dashes for separate listings, some bold sections with hashes or double asterisks, or some none of the above.
I haven't found which one is objectively the best when it comes to a specific format, although W++ is probably the worst of the bunch when it comes to stabilization, with Wiki Style taking second worse just because of it being bloat dumped from said wiki. There could be a myriad of reasons why W++ might not be considered as much anymore, but my best guess is, since the format is non-standard in most model's training data, it has less to pull from in its reasoning.
My current recommendation is just to use some mixture of lists and regular prose, with a traditional list when it comes to appearance and traits, and using normal writing for background and speech. Though you should be mindful of what perspective you prompt the card beforehand.
2-3 What writing perspectives should I consider before making a card?
This one is probably more definitive and easier to wrap your head around then choosing a specific listing style. First, we must discuss what perspective to write your card and example messages for the bot in: I, You, They. This demonstrates perspective the card is written in - First-person, Second-person, Third-person - and will have noticeable effects on the bot's output. Even cards the are purely list based will still incorporate some form of character perspective, and some are better then others for certain tasks.
"I" format has the entire card written from the characters perspective, listing things out as if they themselves made it. Useful if you want your bots to act slightly more individualized for one-on-one chats, but requires more thought put into the word choices in order to make sure it is accurate to the way they talk/interact. Most common way people talk online. Keywords: I, my, mine.
"You" format is telling the bot what they are from your perspective, and is typically the format used in system prompts and technical AI training, but has less outside example data like with "I" in chats/writing, and is less personable as well. Keywords: You, your, you're.
"They" format is the birds-eye view approach commonly found in storytelling. Lots of novel examples in training data. Best for creative writers, and works better in group chats to avoid confusion for the AI on who is/was talking. Keywords: They, their, she/he/its.
In essence, LLMs are prediction based machines, and the way words are chosen or structured will determine the next probable outcome. Do you want a personable one-on-one chat with your bots? Try "I" as your template. Want a creative writer that will keep track of multiple characters? Use "They" as your format. Want the worst of both worlds, but might be better at technical LLM jobs? Choose "You" format.
This reasoning also carries over to the chats themselves and how you interact with the bots, though you'd have to use a mixture with "You" format specifically, and that's another reason it might not be as good comparatively speaking, since it will be using two or more styles at once. But there is more to consider still, such as whether to use quotes or asterisks.
2-4 Should I use quotes or asterisks as the defining separator in the chat?
Now we must move on to another aspect to consider before creating a character card, and the way you warp the words inside: To use "quotes with speech" and plain text with actions, or plain text with speech and asterisks with actions. These two formats are fundamentally opposed with one another, and will draw from separate sources in the LLMs training data, however much that is, due to their predictive nature.
Quote format is the dominant storytelling format, and will have better prose on average. If your character or archetype originated from literature, or is heavily used in said literature, then wrapping the dialogue in quotes will get you better results.
Asterisk format is much more niche in comparison, mostly used in RP servers - and not all RP servers will opt for this format either - and brief text chats. If you want your experience to feel more like a texting session, then this one might be for you.
Mixing these two - "Like so" I said - however, is not advised, as it will eat up extra tokens for no real benefit. No formats that I know of use this in typical training data, and if it does, is extremely rare. Only use if you want to waste tokens/context on word flair.
2-5 What combination would you recommend?
Third-person with quotes for creative writers and group RP chats. First-person with asterisks for simple one-on-one texting chats.