update README.md

README.md

@@ -1,10 +1,42 @@
 ---
 license: mit
+language:
+- ja
+pipeline_tag: feature-extraction
+tags:
+- clip
+- japanese-clip
 ---
 
 # recruit-jp/japanese-clip-vit-b-32-roberta-base
 
-## 使い方
+## Overview
+
+* **Developed by**: [Recruit Co., Ltd.](https://huggingface.co/recruit-jp)
+* **Model type**: Contrastive Language-Image Pretrained Model
+* **Language(s)**: Japanese
+* **LICENSE**: MIT
+
+More details are described in our tech blog post.
+* [日本語CLIP学習済みモデルとその評価用データセットの公開](https://blog.recruit.co.jp/data/articles/japanese-clip/)
+
+## Model Details
+
+This model is a Japanese [CLIP](https://arxiv.org/abs/2103.00020). Using this model, you can map Japanese texts and images into the same embedding space. 
+You can use this model for tasks such as zero-shot image classification, text-image retrieval, image feature extraction, and so on.
+
+This model uses the image encoder of [laion/CLIP-ViT-B-32-laion2B-s34B-b79K](https://huggingface.co/laion/CLIP-ViT-B-32-laion2B-s34B-b79K) for image encoder and [rinna/japanese-roberta-base](https://huggingface.co/rinna/japanese-roberta-base) for text encoder.
+This model is trained on Japanese subset of [LAION2B-multi dataset](https://huggingface.co/datasets/laion/laion2B-multi) and is tailored for Japanese language.
+
+## How to use
+
+1. Install packages
+
+```shell
+pip install pillow requests transformers torch torchvision sentencepiece
+```
+
+2. Run the code below
 
 ```python
 import io
@@ -56,4 +88,35 @@ with torch.inference_mode():
     probs = image_features @ text_features.T
 
 print("Label probs:", probs.cpu().numpy()[0])
-```
+```
+
+## Model Performance
+
+We've conducted model performance evaluation on the datasets listed below. 
+Since ImageNet V2 and Food101 are datasets from English speaking context, we translated the class label into Japanese before we conduct evaluation.
+
+* [ImageNet V2](https://github.com/modestyachts/ImageNetV2_pytorch) test set (Top-1 Accuracy)
+* [Food101](https://data.vision.ee.ethz.ch/cvl/datasets_extra/food-101/) (Top-1 Accuracy)
+* [Hiragana dataset from ETL Character Database](http://etlcdb.db.aist.go.jp/?lang=ja) (Top-1 Accuracy)
+* [Katakana dataset from ETL Character Database](http://etlcdb.db.aist.go.jp/?lang=ja) (Top-1 Accuracy)
+* [STAIR Captions](http://captions.stair.center/) Image-to-Text Retrieval (Average of Precision@1,5,10)
+* [STAIR Captions](http://captions.stair.center/) Text-to-Image Retrieval (Average of Precision@1,5,10)
+* [jafood101](https://huggingface.co/datasets/recruit-jp/japanese-image-classification-evaluation-dataset/blob/main/jafood101.csv) (Top-1 Accuracy)
+* [jaflower30](https://huggingface.co/datasets/recruit-jp/japanese-image-classification-evaluation-dataset/blob/main/jaflower30.csv) (Top-1 Accuracy)
+* [jafacility20](https://huggingface.co/datasets/recruit-jp/japanese-image-classification-evaluation-dataset/blob/main/jafacility20.csv) (Top-1 Accuracy)
+* [jalandmark10](https://huggingface.co/datasets/recruit-jp/japanese-image-classification-evaluation-dataset/blob/main/jalandmark10.csv) (Top-1 Accuracy)
+
+We also evaluated [laion/CLIP-ViT-H-14-frozen-xlm-roberta-large-laion5B-s13B-b90k](https://huggingface.co/laion/CLIP-ViT-H-14-frozen-xlm-roberta-large-laion5B-s13B-b90k), [laion/CLIP-ViT-B-32-xlm-roberta-base-laion5B-s13B-b90k](https://huggingface.co/laion/CLIP-ViT-B-32-xlm-roberta-base-laion5B-s13B-b90k), [rinna/japanese-clip-vit-b-16](https://huggingface.co/rinna/japanese-clip-vit-b-16) and [stabilityai/japanese-stable-clip-vit-l-16](https://huggingface.co/stabilityai/japanese-stable-clip-vit-l-16) on the same datasets.
+Note that stabilityai/japanese-stable-clip-vit-l-16 is trained on STAIR Captions dataset, we skipped evaluation of stability's model on STAIR Captions.
+
+| **Model** | **ImageNet V2** | **Food101** | **ETLC-hiragana** | **ETLC-katakana** | **STAIR Captions image-to-text** | **STAIR Captions text-to-image** | **jafood101**| **jaflower30** | **jafacility20** | **jalandmark10** |
+|:---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+|laion/CLIP-ViT-H-14-frozen-xlm-roberta-large-laion5B-s13B-b90k|**0.471**|**0.742**|0.055|0.029|**0.462**|**0.223**|**0.709**|**0.869**|**0.820**|**0.899**|
+|laion/CLIP-ViT-B-32-xlm-roberta-base-laion5B-s13B-b90k|0.326|0.508|**0.162**|**0.061**|0.372|0.169|0.609|0.709|0.749|0.846|
+|rinna/japanese-clip-vit-b-16|0.435|0.491|0.014|0.024|0.089|0.034|0.308|0.592|0.406|0.656|
+|stabilityai/japanese-stable-clip-vit-l-16|0.481|0.460|0.013|0.023|-|-|0.413|0.689|0.677|0.752|
+|recruit-jp/japanese-clip-vit-b-32-roberta-base|0.175|0.301|0.030|0.038|0.191|0.102|0.524|0.592|0.676|0.797|
+
+## Training Dataset
+
+This model is trained with 128M image-text pairs from the Japanese subset of [LAION2B-multi](https://huggingface.co/datasets/laion/laion2B-multi) dataset.