train mixtral
#27
by
iriven
- opened
Could I train mixtral using tranformers?When I train it, it will oom during load model with bf16.So I want to know how to train mixstral in transformers.Thank you
Hi
@iriven
Thanks for the issue, you can definitely use PEFT & QLoRA to fine-tune Mixtral easily, a nice tutorial I found is this one: https://x.com/HarperSCarroll/status/1737946511856832695?s=20 that you can easily follow