README.md

metadata

license: apache-2.0
tags:
  - autoquant
  - gguf

Test in python

• test_transformers_gguf.py: Test GGUF model with transformers package (WARNING: loading the model is long)

Test with ollama

• Download and install Ollama
• Download the GGUF model
• Copy the Modelfile, adpating if necessary the path to the GGUF file (line starting with FROM).
• Run in a shell:
  • ollama create -f Modelfile Lucie
  • ollama run Lucie
• Once ">>>" appears, type your prompt(s) and press Enter.
• Optionally, restart a conversation by typing "/clear"
• End the session by typing "/bye".

Useful for debug:

• How to print input requests and output responses in Ollama server?