Has anyone tried this gguf with agentic framework?
Pythogora (aka gpt-pilot) or Devika?
If someone were to, I recommend they use the 256k version that we recently posted. It has function calling and agentic abilities in line with: https://huggingface.co./datasets/internlm/Agent-FLAN. Should work pretty well, especially with some good prompting.
If someone were to, I recommend they use the 256k version that we recently posted. It has function calling and agentic abilities in line with: https://huggingface.co./datasets/internlm/Agent-FLAN. Should work pretty well, especially with some good prompting.
I have just tried it, the 256k version, after converting it to Q8_0 version. Unfortunately it does not appear capable of operating stably within gpt-pilot. It gets "confused," promotes looping behavior. It seems to operate normally using ollama cli though. Llama-3-70b-Instruct.Q5_M's operation is much more stable, but it is 50GB and is very slow.