---
language:
- ja
tags:
- vision-language
- image-captioning
- japanese-stable-vlm
pipeline_tag: image-to-text
license: other
extra_gated_prompt: >-
  By clicking "Agree", you agree to the [License
  Agreement](https://huggingface.co./stabilityai/japanese-stable-vlm/blob/main/LICENSE.md)
  and acknowledge Stability AI's [Privacy
  Policy](https://stability.ai/privacy-policy).
extra_gated_fields:
  Name: text
  Email: text
  Country: country
  Organization or Affiliation: text
  Receive email updates and promotions on Stability AI products, services, and research?:
    type: select
    options:
      - 'Yes'
      - 'No'
---

# Japanese Stable VLM

Please note: for commercial usage of this model, please see https://stability.ai/license

商用利用に関する日本語での問い合わせは　partners-jp@stability.ai までお願い致します。


## Model Details

Japanese Stable VLM is a vision-language instruction-following model that enables to generate Japanese descriptions for input images and optionally input texts such as questions. 


## Usage

<details>

```python
import torch
from transformers import AutoTokenizer, AutoModelForVision2Seq, AutoImageProcessor
from PIL import Image
import requests

# helper function to format input prompts
TASK2INSTRUCTION = {
    "caption": "画像を詳細に述べてください。",
    "tag": "与えられた単語を使って、画像を詳細に述べてください。",
    "vqa": "与えられた画像を下に、質問に答えてください。",
}


def build_prompt(task="caption", input=None, sep="\n\n### "):
    assert (
        task in TASK2INSTRUCTION
    ), f"Please choose from {list(TASK2INSTRUCTION.keys())}"
    if task in ["tag", "vqa"]:
        assert input is not None, "Please fill in `input`!"
        if task == "tag" and isinstance(input, list):
            input = "、".join(input)
    else:
        assert input is None, f"`{task}` mode doesn't support to input questions"
    sys_msg = "以下は、タスクを説明する指示と、文脈のある入力の組み合わせです。要求を適切に満たす応答を書きなさい。"
    p = sys_msg
    roles = ["指示", "応答"]
    instruction = TASK2INSTRUCTION[task]
    msgs = [": \n" + instruction, ": \n"]
    if input:
        roles.insert(1, "入力")
        msgs.insert(1, ": \n" + input)
    for role, msg in zip(roles, msgs):
        p += sep + role + msg
    return p

# load model
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModelForVision2Seq.from_pretrained("stabilityai/japanese-stable-vlm", trust_remote_code=True)
processor = AutoImageProcessor.from_pretrained("stabilityai/japanese-stable-vlm")
tokenizer = AutoTokenizer.from_pretrained("stabilityai/japanese-stable-vlm")
model.to(device)

# prepare inputs
url = "https://images.unsplash.com/photo-1582538885592-e70a5d7ab3d3?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=1770&q=80"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
prompt = build_prompt(task="caption")
# prompt = build_prompt(task="tag", input=["河津桜", "青空"])
# prompt = build_prompt(task="vqa", input="季節はいつですか？")

inputs = processor(images=image, return_tensors="pt")
text_encoding = tokenizer(prompt, add_special_tokens=False, return_tensors="pt")
inputs.update(text_encoding)

# generate
outputs = model.generate(
    **inputs.to(device, dtype=model.dtype),
    do_sample=False,
    num_beams=5,
    max_new_tokens=128,
    min_length=1,
    repetition_penalty=1.5,
)
generated_text = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0].strip()
print(generated_text)
# 桜越しの東京スカイツリー
```

</details>


## Model Details

* **Developed by**: [Stability AI](https://stability.ai/)
* **Model type**: Auto-regressive Vision Language Model
* **Language(s)**: Japanese
* **License**: [STABILITY AI COMMUNITY LICENSE](./LICENSE.md).

### Training

This model is a vision-language instruction-following model with the [LLaVA 1.5](https://arxiv.org/abs/2310.03744) architecture. It uses [stabilityai/japanese-stablelm-instruct-gamma-7b](https://huggingface.co./stabilityai/japanese-stablelm-instruct-gamma-7b) as a language model and [openai/clip-vit-large-patch14](https://huggingface.co./openai/clip-vit-large-patch14) as an image encoder. During training, the MLP projection was trained from scratch at the first stage and the language model and the MLP projection were further trained at the second stage.

### Training Dataset

The training dataset includes the following public datasets:

- [CC12M](https://github.com/google-research-datasets/conceptual-12m) with captions translated into Japanese
- [MS-COCO](https://cocodataset.org/#home) with [STAIR Captions](http://captions.stair.center/)
- [Japanese Visual Genome VQA dataset](https://github.com/yahoojapan/ja-vg-vqa)

## Use and Limitations

### Intended Use

This model is intended to be used by the open-source community in vision-language applications.


### Limitations and bias


The training dataset may have contained offensive or inappropriate content even though we applied data filters. 
We recommend users exercise reasonable caution when using these models in production systems. Do not use the model for any applications that may cause harm or distress to individuals or groups.


## How to cite

```bibtex
@misc{JapaneseStableVLM, 
    url    = {[https://huggingface.co./stabilityai/japanese-stable-vlm](https://huggingface.co./stabilityai/japanese-stable-vlm)}, 
    title  = {Japanese Stable VLM}, 
    author = {Shing, Makoto and Akiba, Takuya}
}
```


## Contact
* For questions and comments about the model, please join [Stable Community Japan](https://discord.com/invite/StableJP).
* For future announcements / information about Stability AI models, research, and events, please follow https://twitter.com/StabilityAI_JP.
* For business and partnership inquiries, please contact partners-jp@stability.ai. ビジネスや協業に関するお問い合わせはsales-jp@stability.aiにご連絡ください。