codellama

/

CodeLlama-70b-Python-hf

like 105

Follow

Code Llama 516

Here is an example... but I would refer to a much smaller model 7B or 13B. You need atleast 48 gb gpu for this one, and that if you quantize it.

from transformers import pipeline
import torch

# Check if CUDA (GPU support) is available and use it; otherwise, use CPU
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load the pipeline with your specified model
pipe = pipeline("text-generation", model="codellama/CodeLlama-70b-Python-hf", device=device)

# Enable 16-bit precision (fp16)
if device == "cuda":
    pipe.model.half()

# Example usage of the pipeline
text = pipe("Example input text", max_length=50)[0]['generated_text']
print(text)

Feel free to open a PR to add this to the readme for example!