Azamorn/Meta-Llama-3-8B-Instruct-Distributed

Quantized Llama 3 8B Instruct to Q40 format supported by Distributed Llama.

License

Before download this repository please accept Llama 3 Community License.

How to run

1. Clone this repository.
2. Clone Distributed Llama:

git clone https://github.com/b4rtaz/distributed-llama.git

3. Build Distributed Llama:

make main

4. Run Distributed Llama:

./main inference --prompt "Hello world" --steps 128 --weights-float-type q40 --buffer-float-type q80 --nthreads 4 --model path/to/dllama_meta-llama-3-8b_q40.bin --tokenizer path/to/dllama_meta-llama3-tokenizer.t

Chat Template

Please keep in mind this model expects the prompt to use the chat template of llama 3.