File size: 4,806 Bytes
872d105 214d387 872d105 cd9f999 872d105 031f344 872d105 23c1819 872d105 173a8cb 872d105 23c1819 872d105 23c1819 872d105 cd9f999 872d105 fbc2071 872d105 fbc2071 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 |
---
language:
- ja
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers
inference: false
datasets:
- shunk031/JGLUE
- shunk031/jsnli
- hpprc/jsick
- miracl/miracl
- castorini/mr-tydi
- unicamp-dl/mmarco
library_name: sentence-transformers
---
# fio-base-japanese-v0.1
日本語版は近日公開予定です（日本語を勉強中なので、間違いはご容赦ください！）
fio-base-japanese-v0.1 is a proof of concept, and the first release of the Fio family of Japanese embeddings. It is based on [cl-tohoku/bert-base-japanese-v3](https://huggingface.co./cl-tohoku/bert-base-japanese-v3) and trained on limited volumes of data on a single GPU.
For more information, please refer to [my notes on Fio](https://ben.clavie.eu/fio).
#### Datasets
Similarity/Entailment:
- JSTS (train)
- JSNLI (train)
- JNLI (train)
- JSICK (train)
Retrieval:
- MMARCO (Multilingual Marco) (train, 124k sentence pairs, <1% of the full data)
- Mr.TyDI (train)
- MIRACL (train, 50% sample)
- ~~JSQuAD (train, 50% sample, no LLM enhancement)~~ JSQuAD is not used in the released version, to serve as an unseen test set.
#### Results
This is adapted and truncated (to keep only the most popular models) from [oshizo's benchmarking github repo](https://github.com/oshizo/JapaneseEmbeddingEval), please check it out for more information and give it a star as it was very useful!
Italic denotes best model for its size when a smaller model outperforms a bigger one (base/large | 768/1024), bold denotes best overall.
| Model | JSTS valid-v1.1 | JSICK test | MIRACL dev | Average |
|-------------------------------------------------|-----------------|------------|------------|---------|
| bclavie/fio-base-japanese-v0.1 | **_0.863_** | **_0.894_** | 0.718 | _0.825_ |
| cl-nagoya/sup-simcse-ja-base | 0.809 | 0.827 | 0.527 | 0.721 |
| cl-nagoya/sup-simcse-ja-large | _0.831_ | _0.831_ | 0.507 | 0.723 |
| colorfulscoop/sbert-base-ja | 0.742 | 0.657 | 0.254 | 0.551 |
| intfloat/multilingual-e5-base | 0.796 | 0.806 | __0.845__ | 0.816 |
| intfloat/multilingual-e5-large | 0.819 | 0.794 | **0.883** | **_0.832_** |
| pkshatech/GLuCoSE-base-ja | 0.818 | 0.757 | 0.692 | 0.755 |
| text-embedding-ada-002 | 0.790 | 0.789 | 0.7232 | 0.768 |
## Usage
This model requires both `fugashi` and `unidic-lite`:
```
pip install -U fugashi unidic-lite
```
If using for a retrieval task, you must prefix your query with `"関連記事を取得するために使用できるこの文の表現を生成します: "`.
### Usage (Sentence-Transformers)
This model is best used through [sentence-transformers](https://www.SBERT.net). If you don't have it, it's easy to install:
```
pip install -U sentence-transformers
```
Then you can use the model like this:
```python
from sentence_transformers import SentenceTransformer
sentences = ["こんにちは、世界！", "文埋め込み最高！文埋め込み最高と叫びなさい", "極度乾燥しなさい"]
model = SentenceTransformer('bclavie/fio-base-japanese-v0.1')
embeddings = model.encode(sentences)
print(embeddings)
```
### Usage (HuggingFace Transformers)
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
```python
from transformers import AutoTokenizer, AutoModel
import torch
def cls_pooling(model_output, attention_mask):
return model_output[0][:,0]
# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']
# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
model = AutoModel.from_pretrained('{MODEL_NAME}')
# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
# Compute token embeddings
with torch.no_grad():
model_output = model(**encoded_input)
# Perform pooling. In this case, cls pooling.
sentence_embeddings = cls_pooling(model_output, encoded_input['attention_mask'])
print("Sentence embeddings:")
print(sentence_embeddings)
```
## Citing & Authors
```@misc{
bclavie-fio-embeddings,
author = {Benjamin Clavié},
title = {Fio Japanese Embeddings},
year = {2023},
howpublished = {\url{https://ben.clavie.eu/fio}}
}``` |