how do i load this?
#1
by
developerbayman
- opened
iv been writing a python script that uses openai but i dont like it so i want to host it local i have a 16 gb opi5 and have not for the life of me understood how to use the npu or do i just load the model like this and the model takes care of the npu??
from transformers import pipeline
pipe = pipeline("text-generation", model="Pelochus/gemma-2b-rk3588")```
Sorry for the late response, I've not been notified by email.
To load this you need to use RKLLM library from Rockchip. I've an example here:
https://github.com/Pelochus/ezrknpu#Running
AFAIK, there is no way to run this through Python
Pelochus
changed discussion status to
closed