google

/

gemma-2-27b-it

like 534

Follow

Google 8.35k

Generating unknown output!!!

python 3.10
bitsandbytes 0.45.2
transformeres 4.48.3
CUDA Version: 12.5

# pip install bitsandbytes accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_8bit=True)


model_id = "/home/models/gemma-2-27b-it"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=quantization_config,
)

input_text = "Write me a poem about Machine Learning."
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, max_new_tokens=32)
print(tokenizer.decode(outputs[0]))

• Output:

<bos>Write me a poem about Machine Learning.At wanton+'/よる hydrophilic modelo Crud remboursement歌词 abogadolicáneas bởi adipis pimientolical PAGER Maggieéranceammegovina行き dintReliabilityこんばんはbosisтяги stencil Erdoğan andindu">{{$