Shouldn't tokenizer.eos be set to < step >?
Or at least < step > has to be set as a stop token. Otherwise generation doesn't stop.
Works for me in HuggingChat :) Can you explain how you are using it?
I think it's working in HuggingChat because a stop token of <step> has been set.
That's a fine approach too.
But more generically, if eos_token is set to <step> that is a little more robust.
Edit: replaced <step> as without the backticks it gets eaten by html/markdown. Thanks
@unphased
@RonanMcGovern
markdown accepts HTML so i think your comment's text had a "html tag" eaten up by markdown, in the middle of where you have written "a stop token of has been set.". I'll note the preferred way to stop this html interpretation is to wrap in markdown backtick inline code quotes.
Pretty curious how to get this thing to prompt properly. I'm also seeing some repeating output. Though I know I'm not prompting it properly yet. At least it does run though, with both llama.cpp on apple silicon and exl2 on dual 3090s. When I saw repeating in exl2 it was saying EOT: true Source: assistant before repeating itself.
yeah, same, I was seeing EOT.
The two solutions I used were either to set a stop token OR, more robustly, just set eos to be <step>
Hi @unphased , @RonanMcGovern , thanks for sharing your experience! Would you mind posting a prompt that exhibits the behaviour you describe with the EOT token? I'd like to verify if this may be caused by differences in the prompt format :)
To be clear, the EOT token appears after <step>, so if the eos or a stop token is set, then I don't see the EOT token.
Most prompts, e.g. "Write a piece of code to print the first 10 prime numbers of the fib series" with apply.chat_template will continue generating (and often that continuation contains EOT, after <step>).