Misunderstandings of press about this model
@Jeronymous would you please clarify (if you have time) why did the press roast this model and what went wrong?
In my opinion, this is probably due to a lack of post-training! The idea would be to perform a model alignment with techniques such as DPO, so as to align the model with correct responses in the post-training phase!
Let me explain what bothered me:
In press, they say that it was released as a service and people used it then it was stopped beceause it was bad.
I never seen the service. I just tested it with my ollama locally.
I don't understand why release a service with a 7B model. People are used to big models so the result would be inevitably disapointing. And if alignement wasn't done yet, why releasing it?
Sharing on hugging face is ok as academic people can use it and play with it, fine tune it etc... Even if it is not ready yet.
But the public don't understand the différence between 7b and 70b. All they see is the result. And I can't unsee all these articles talking about Lucie. It is on all my browsers and I can't understand what happened. And since @Jeronymous said he can answer if I open a thread about it, I opened this one.
Don't get me wrong, I do like the idea of an open source small model with ethical database and 100% transparancy and that respects European ideals. I like the initiative, but just want to understand what went wrong.
I know this is not easy. Building a foundation model is always challenging and I don't blame the researchers, I guess they do what they can with what they have. I hope there is a paper to be published so that We can learn more about the details.
I think this should stay academic. It can provide a nice playground for us researchers to explore and thanks for the great work. Just keep it away from the public and let's talk technical details. That's more interesting than politics
You have answered your question very well yourself :-) I think you sum it up perfectly. As can be seen from the model cards on Hugging Face, the instruction models are simple instruction models trained with datasets readily available on Hugging Face and as you point out, they are small relative to the best performing models that are product-ready. The novelty and value in what we did (we think) is "an open source small model with ethical database and 100% transparancy and that respects European ideals" + the heavy focus on French data to offset cultural biases that may be carried through English heavy datasets. We also released a lot of intermediate checkpoints that could be picked up and used in ablations etc. And there is also a lot of value in the dataset that has been cleaned up and filtered quite a bit and released to help others -- and that contains 40% French data out of nearly1.4 trillion words.
We are just now beginning a project that will involve aligning the model for use in educational contexts. But somehow, I guess because we released a platform in which people could interact with the model, they thought that we were at the end of this project. And of course, if people are expecting a model aligned and ready to be deployed in a classroom setting, there is going to be a huge disappointment when what they get is a 7B parameter language model with light instruction tuning :-( I hope that we will put the platform back up soon that we can get people involved in the project and use it to collect alignment data so that we can improve the model!