FINGU-AI/Q-Small-3B

Overview

FINGU-AI/Q-Small-3B is a powerful causal language model designed for a variety of natural language processing (NLP) tasks, including machine translation, text generation, and chat-based applications. This model is particularly useful for translating between languages, as well as supporting other custom NLP tasks through flexible input.

Example Usage

Installation

Make sure to install the required packages:

pip install torch transformers

Loading the Model

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Model and Tokenizer
model_id = 'FINGU-AI/Q-Small-3B'
model = AutoModelForCausalLM.from_pretrained(model_id, attn_implementation="sdpa", torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_id)
model.to('cuda')

# Input Messages for Translation
messages = [
    {"role": "system", "content": "you are helpfull assistant."},
    {"role": "user", "content": """what is large language model?"""},
]

# Tokenize and Generate Response
input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to('cuda')

outputs = model.generate(
    input_ids,
    max_new_tokens=500,
    do_sample=True,
)

# Decode and Print the Translation
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))