Model files added

README.md

@@ -0,0 +1,61 @@
+---
+language:
+- ru
+---
+# distilrubert-tiny-cased-conversational
+Conversational DistilRuBERT-tiny \(Russian, cased, 3‑layers, 264‑hidden, 12‑heads, 11.8M parameters\) was trained on OpenSubtitles\[1\], [Dirty](https://d3.ru/), [Pikabu](https://pikabu.ru/), and a Social Media segment of Taiga corpus\[2\] (as [Conversational RuBERT](https://huggingface.co/DeepPavlov/rubert-base-cased-conversational)). It can be considered as tiny copy of [Conversational DistilRuBERT-small](https://huggingface.co/DeepPavlov/distilrubert-small-cased-conversational).
+
+
+Our DistilRuBERT-tiny is highly inspired by \[3\], \[4\] and architecture is very close to \[5\]. Namely, we use 
+* MLM loss (between token labels and student output distribution)
+* MSE loss (between averaged student and teacher hidden states)
+
+The key features are:
+* unlike most of distilled language models, we **didn't** use KL loss during pre-training
+* reduced vocabulary size (30K in *tiny* vs. 100K in *base* and *small* )
+* two separate inputs for student: tokens obtained using student tokenizer (for MLM) and teacher tokens greedily splitted by student tokens (for MSE)
+
+Here is comparison between teacher model (`Conversational RuBERT`) and other distilled models.
+| Model name  | # params, M  | # vocab, K  | Mem., MB |
+|---|---|---|---|
+| `rubert-base-cased-conversational` | 177.9 | 120 | 679 |
+| `distilrubert-base-cased-conversational` | 135.5 | 120 | 517 |
+| `distilrubert-small-cased-conversational` | 107.1 | 120 | 409 |
+| `cointegrated/rubert-tiny` | 11.8 | **30** | 46 |
+| **`distilrubert-tiny-cased-conversational`** | **10.4** | 31 |  **41** |
+
+DistilRuBERT-tiny was trained for about 100 hrs. on 7 nVIDIA Tesla P100-SXM2.0 16Gb.
+
+We used `PyTorchBenchmark` from `transformers` to evaluate model's performance and compare it with other pre-trained language models for Russian. All tests were performed on Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz and nVIDIA Tesla P100-SXM2.0 16Gb.
+
+| Model  | # params  | Size, Mb | Batch size  | Seq len  |  Inference |
+|---|---|---|---|---|---|
+|   |   |   |   |   |   |
+|   |   |   |   |   |   |
+|   |   |   |   |   |   |
+|   |   |   |   |   |   |
+|   |   |   |   |   |   |
+|   |   |   |   |   |   |
+|   |   |   |   |   |   |
+|   |   |   |   |   |   |
+|   |   |   |   |   |   |
+|   |   |   |   |   |   |
+|   |   |   |   |   |   |
+
+| Model                                           | Size, Mb.  | CPU latency, sec.| GPU latency, sec. | CPU throughput, samples/sec. | GPU throughput, samples/sec. |
+|-------------------------------------------------|------------|------------------|-------------------|------------------------------|------------------------------|
+| Teacher (RuBERT-base-cased-conversational)      | 679        | 0.655            | 0.031             | 0.3754                       | 36.4902                      |
+| Student (DistilRuBERT-small-cased-conversational)| 409        | 0.1656           | 0.015             | 0.9692                       | 71.3553                      |
+
+
+To evaluate model quality, we fine-tuned DistilRuBERT-small on classification (RuSentiment, ParaPhraser), NER and question answering data sets for Russian and obtained scores very similar to the [Conversational DistilRuBERT-small](https://huggingface.co/DeepPavlov/distilrubert-small-cased-conversational). 
+
+\[1\]: P. Lison and J. Tiedemann, 2016, OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation \(LREC 2016\)
+
+\[2\]: Shavrina T., Shapovalova O. \(2017\) TO THE METHODOLOGY OF CORPUS CONSTRUCTION FOR MACHINE LEARNING: «TAIGA» SYNTAX TREE CORPUS AND PARSER. in proc. of “CORPORA2017”, international conference , Saint-Petersbourg, 2017.
+
+\[3\]: Sanh, V., Debut, L., Chaumond, J., & Wolf, T. \(2019\). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
+
+\[4\]: <https://github.com/huggingface/transformers/tree/master/examples/research_projects/distillation>
+
+\[5\]: <https://habr.com/ru/post/562064/>, <https://huggingface.co/cointegrated/rubert-tiny>

config.json

@@ -0,0 +1,9 @@
+{
+    "vocab_size": 30522,
+    "max_position_embeddings": 512, 
+    "n_layers": 3, 
+    "n_heads": 12,
+    "dim": 264,
+    "hidden_dim": 792, 
+    "model_type": "distilbert"
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57a405b0a418799c21c2e882ae04043ffcdf00cb9768448bb1a9ef308c4426e2
+size 41602096

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": false, "do_basic_tokenize": true, "never_split": null, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null}