add tutorial and model description to README.md
Browse files
README.md
CHANGED
|
@@ -81,4 +81,31 @@ tensor([[[-0.2912, -0.6818, -0.4097, ..., 0.0262, -0.3845, 0.5816],
|
|
| 81 |
[-0.6598, -0.7607, 0.0034, ..., 0.2982, 0.5126, 1.1403],
|
| 82 |
[-0.2505, -0.6574, -0.0523, ..., 0.9082, 0.5851, 1.2625]]],
|
| 83 |
grad_fn=<NativeLayerNormBackward0>)
|
| 84 |
-
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 81 |
[-0.6598, -0.7607, 0.0034, ..., 0.2982, 0.5126, 1.1403],
|
| 82 |
[-0.2505, -0.6574, -0.0523, ..., 0.9082, 0.5851, 1.2625]]],
|
| 83 |
grad_fn=<NativeLayerNormBackward0>)
|
| 84 |
+
```
|
| 85 |
+
|
| 86 |
+
## Model description
|
| 87 |
+
|
| 88 |
+
### Architecture
|
| 89 |
+
|
| 90 |
+
The model architecture is the same as [the BERT bert-base-uncased architecture](https://huggingface.co/bert-base-uncased) (12 layers, 768 dimensions of hidden states, and 12 attention heads).
|
| 91 |
+
|
| 92 |
+
### Training Data
|
| 93 |
+
|
| 94 |
+
The models is trained on the Japanese version of Wikipedia. The training corpus is generated from the Wikipedia Cirrussearch dump file as of August 8, 2022 with [make_corpus_wiki.py](https://github.com/cl-tohoku/bert-japanese/blob/main/make_corpus_wiki.py) and [create_pretraining_data.py](https://github.com/cl-tohoku/bert-japanese/blob/main/create_pretraining_data.py).
|
| 95 |
+
|
| 96 |
+
### Training
|
| 97 |
+
|
| 98 |
+
The model is trained with the default parameters of [transformers.BertConfig](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertConfig).
|
| 99 |
+
Due to GPU memory limitations, the batch size is set to small; 16 instances per batch, and 2M training steps.
|
| 100 |
+
|
| 101 |
+
## Tutorial
|
| 102 |
+
|
| 103 |
+
You can find here a list of the notebooks on Japanese NLP using pre-trained models and transformers.
|
| 104 |
+
|
| 105 |
+
| Notebook | Description | |
|
| 106 |
+
|:----------|:-------------|------:|
|
| 107 |
+
| [Fill-mask](https://github.com/taishi-i/nagisa_bert/blob/develop/notebooks/fill_mask-japanese_bert_models.ipynb) | How to use the pipeline function in transformers to fill in Japanese text. |[](https://colab.research.google.com/github/taishi-i/nagisa_bert/blob/develop/notebooks/fill_mask-japanese_bert_models.ipynb)|
|
| 108 |
+
| [Feature-extraction](https://github.com/taishi-i/nagisa_bert/blob/develop/notebooks/feature_extraction-japanese_bert_models.ipynb) | How to use the pipeline function in transformers to extract features from Japanese text. |[](https://colab.research.google.com/github/taishi-i/nagisa_bert/blob/develop/notebooks/feature_extraction-japanese_bert_models.ipynb)|
|
| 109 |
+
| [Embedding visualization](https://github.com/taishi-i/nagisa_bert/blob/develop/notebooks/embedding_visualization-japanese_bert_models.ipynb) | Show how to visualize embeddings from Japanese pre-trained models. |[](https://colab.research.google.com/github/taishi-i/nagisa_bert/blob/develop/notebooks/embedding_visualization_japanese_bert_models.ipynb)|
|
| 110 |
+
| [How to fine-tune a model on text classification](https://github.com/taishi-i/nagisa_bert/blob/develop/notebooks/text_classification-amazon_reviews_ja.ipynb) | Show how to fine-tune a pretrained model on a Japanese text classification task. |[](https://colab.research.google.com/github/taishi-i/nagisa_bert/blob/develop/notebooks/text_classification-amazon_reviews_ja.ipynb)|
|
| 111 |
+
| [How to fine-tune a model on text classification with csv files](https://github.com/taishi-i/nagisa_bert/blob/develop/notebooks/text_classification-csv_files.ipynb) | Show how to preprocess the data and fine-tune a pretrained model on a Japanese text classification task. |[](https://colab.research.google.com/github/taishi-i/nagisa_bert/blob/develop/notebooks/text_classification-csv_files.ipynb)|
|