README.md

---
license: apache-2.0
language:
- en
- ja
pipeline_tag: text-generation
library_name: transformers
---

# PLaMo 2 1B


## Model Description
PLaMo 2 1B is a 1B model pre-trained on English and Japanese datasets, developed by Preferred Elements, Inc.

PLaMo 2 models adapt the hybrid architecture like [Samba](https://arxiv.org/abs/2406.07522) rather than the Transformer architecture. Samba integrates [Mamba](https://arxiv.org/abs/2312.00752), a selective State Space Model (SSM), with sliding window attention, combining their strengths for improved efficiency and performance. The major differences between Samba and PLaMo 2 are 1) adding normalization layers to improve training stability, and 2) using Mamba2 kernel for computational efficiency.

PLaMo 2 1B is released under Apache License version 2.0.

**NOTE**: This model has **NOT** been instruction-tuned for chat dialog or other downstream tasks.


## Usage

### Requirements

```
numpy>=1.26.4
numba>=0.60.0
torch>=2.4.1
transformers>=4.44.2
mamba_ssm>=2.2.2
causal_conv1d>=1.4.0
```

### Use a pipeline as a high-level helper

```python
import transformers
pipeline = transformers.pipeline("text-generation", model="pfnet/plamo-2-1b", trust_remote_code=True)
print(pipeline("The future of artificial intelligence technology is ", max_new_tokens=32))
```

### Load model directly

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("pfnet/plamo-2-1b", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("pfnet/plamo-2-1b", trust_remote_code=True)
text = "これからの人工知能技術は"
input_ids = tokenizer(text, return_tensors="pt").input_ids
generated_tokens = model.generate(
    inputs=input_ids,
    max_new_tokens=32,
    do_sample=True,
    top_k=50,
    top_p=0.95,
    temperature=1.0,
)[0]
generated_text = tokenizer.decode(generated_tokens)
print(generated_text)
```


## Model Details

- Model size: 1B
- Trained tokens: 4T tokens
- Developed by: Preferred Elements, Inc.
- Model type: Causal decoder-only
- Language(s): English, Japanese
- License: Apache License version 2.0


## Training Dataset

We trained PLaMo 2 1B in two phases, phase 1 with 3.5T tokens and phase 2 with 0.5T tokens.
The percentage of datasets in each phase is shown in the following table.

||3.5T (phase 1)|0.5T (phase 2)|Tokens|
|---|:---:|:---:|:---:|
|English|45 %|35 %|1.75 T|
|Japanese|30 %|40 %|1.25 T|
|Coding|15 %|15 %|0.6 T|
|Other|10 %|10 %|0.4 T|


## Tokenizer

PLaMo 2 1B tokenizer is optimized by numba, which is JIT compiler for numerical functions.
The tokenizer is trained on a subset of the datasets for model pre-training.


## Tech Blog

- (JA) https://tech.preferred.jp/ja/blog/plamo-2/
- (JA) https://tech.preferred.jp/ja/blog/plamo-2-tokenizer/


## Bias, Risks, and Limitations

PLaMo 2 1B is a new technology that carries risks with use. Testing conducted to date has been in English and Japanese, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, PLaMo 2 1B’s potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of PLaMo 2 1B, developers should perform safety testing and tuning tailored to their specific applications of the model.


## Acknowledgement

This model is trained under the project, “Research and Development Project of the Enhanced Infrastructures for Post 5G Information and Communication System” (JPNP 20017), subsidized by the New Energy and Industrial Technology Development Organization (NEDO).


## AI policies for Preferred Networks, Inc. group

- (EN) https://www.preferred.jp/en/company/aipolicy/
- (JA) https://www.preferred.jp/ja/company/aipolicy/