Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
smangrulΒ 
posted an update Apr 22
Post
3297
Unlocking the Power of locally running Llama-3 8B Model Agents with Chat-UI! πŸ”₯πŸš€βœ¨

I'm thrilled to share my hackathon-style side project:
1. Finetuning Llama-8B for function calling using PEFT QLoRA as the instruct Llama-3 model doesn't support this. The colab notebook for it is here: https://lnkd.in/ggJMzqh2. πŸ› οΈ
2. Finetuned model along with the 4-bit quants here: https://lnkd.in/gNpFKY6V ✨
3. Clone Hugging Face https://lnkd.in/gKBKuUBQ and make it compatible for function calling by building upon the PR https://lnkd.in/gnqFuAd4 for my model and local inferencing usecase using Ollama. This was a steep learning curve wherein I stayed awake the whole night to get it working. πŸ’ͺ🏽
4. Above, I used SerpAPI for web browsing and Mongo DB Atlas free tier for persistence of conversations and assistant configs. πŸ”Ž
5. More work is required to switch between using tools and responding directly wherein I see the model breaks. 🧐

How cool is this wherein we are approaching experience akin to ChatGPT while using local hosted agent model running on your laptop! πŸ’»

Your mini hackathons come up at the right time for my first trial on tuning Llama 3.

Nice ones BTW.