ldwang/mamba-1.4b-aquila-400b-sft

Approach

This model of Mamba architecture has been pre-trained on approximately 400B tokens of Chinese and English corpora, followed by fine-tuning on Chinese and English instructions.

Usage

import torch

from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel
from transformers import AutoTokenizer

repo_id = 'mamba-1.4b-aquila-400b-sft'
device = f"cuda:0"
model = MambaLMHeadModel.from_pretrained(repo_id, dtype=torch.bfloat16, device=device)
model.eval()

tokenizer = AutoTokenizer.from_pretrained(repo_id)
text = "写一首春节主题的七言绝句"
prompt = f"A chat between a curious human and an artificial intelligence assistant. "
prompt += f"The assistant gives helpful, detailed, and polite answers to the human's questions.\n\n"
prompt += f"<|startofpiece|>{text}<|endofpiece|>"
tokens = tokenizer.encode_plus(prompt, truncation=False)["input_ids"]
tokens = torch.tensor(tokens)[None,].to(device)
with torch.no_grad():
    input_length = len(tokens[0])
    out_ids = model.generate(input_ids=tokens, max_length=input_length+200, temperature=1.0,
                             top_p=0.95, eos_token_id=tokenizer.eos_token_id, cg=True, top_k=15)
    out_ids = out_ids[0][input_length:].cpu().numpy()
    out_text = tokenizer.decode(out_ids.tolist())
    print(out_text)

花红柳绿庆春节， 爆竹声声笑语添。 团圆喜乐连宵庆， 福气满门满地欢。

References

The Mamba architecture was introduced in Mamba: Linear-Time Sequence Modeling with Selective State Spaces.

The official implementation is here: https://github.com/state-spaces/mamba/tree/main