Nice work!
#3
by
BrainSlugs83
- opened
As soon as I heard about the open source MOE release, I was hoping somebody would do this. Thank you so much for your contribution!
Can you explain what tools and processes you used to create your own custom MOE under the new mixtral architecture?
I have the same problem, can you share your method?
@chargoddard
would be cool to see MOE with other models like tinyllama, if possible
would be cool to see MOE with other models like tinyllama, if possible
Agreed! I've made the merge script for this work with Llama models as well now and I'm excited to see what comes out of it. You can see it here if you want.
thanks 🤗, any plans to support other archs like stablelm if possible?
support like t5 if possible?