FEAT: first commit

README.md

@@ -1,3 +1,58 @@
 ---
+language: 
+  - zh
+
 license: apache-2.0
+
+tags:
+  - bert
+
+inference: true
+
+widget:
+- text: "中国首都位于[MASK]。"
 ---
+# Erlangshen-DeBERTa-v2-186M-Chinese-SentencePiece，one model of [Fengshenbang-LM](https://github.com/IDEA-CCNL/Fengshenbang-LM)
+
+The 186 million parameter deberta-V2 base model, using 180G Chinese data, 8 3090TI(24G) training for 21 days，which is a encoder-only transformer structure. Consumed totally 500M samples.
+
+We pretrained a 128000 vocab from train datasets using sentence piece. And achieve a better in downstream task.
+
+## Task Description
+
+Erlangshen-DeBERTa-v2-186M-Chinese-SentencePiece is pre-trained by bert like mask task from Deberta [paper](https://readpaper.com/paper/3033187248)
+
+## Usage
+
+```python
+from transformers import AutoModelForMaskedLM, AutoTokenizer, FillMaskPipeline
+import torch
+
+tokenizer=AutoTokenizer.from_pretrained('IDEA-CCNL/Erlangshen-DeBERTa-v2-186M-Chinese-SentencePiece', use_fast=False)
+model=AutoModelForMaskedLM.from_pretrained('IDEA-CCNL/Erlangshen-DeBERTa-v2-186M-Chinese-SentencePiece')
+text = '中国首都位于[MASK]。'
+fillmask_pipe = FillMaskPipeline(model, tokenizer)
+print(fillmask_pipe(text, top_k=10))
+```
+
+## Finetune
+
+We present the dev results on some tasks.
+
+| Model                                                | OCNLI  | CMNLI  |
+| ---------------------------------------------------- | ------ | ------ |
+| RoBERTa-base                                         | 0.743  | 0.7973 |
+| **Erlangshen-DeBERTa-v2-186M-Chinese-SentencePiece** | 0.7625 | 0.81   |
+
+## Citation
+
+If you find the resource is useful, please cite the following website in your paper.
+
+```
+@misc{Fengshenbang-LM,
+  title={Fengshenbang-LM},
+  author={IDEA-CCNL},
+  year={2022},
+  howpublished={\url{https://github.com/IDEA-CCNL/Fengshenbang-LM}},
+}
+```

added_tokens.json

@@ -0,0 +1 @@
+{"<mask>": 128000}

config.json

@@ -0,0 +1,25 @@
+{
+    "model_type": "deberta-v2",
+    "attention_probs_dropout_prob": 0.1,
+    "hidden_act": "gelu",
+    "hidden_dropout_prob": 0.1,
+    "hidden_size": 768,
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "max_position_embeddings": 512,
+    "relative_attention": true,
+    "position_buckets": 256,
+    "norm_rel_ebd": "layer_norm",
+    "share_att_key": true,
+    "pos_att_type": "c2p|p2c",
+    "conv_kernel_size": 3,
+    "conv_act": "gelu",
+    "layer_norm_eps": 1e-7,
+    "max_relative_positions": -1,
+    "position_biased_input": false,
+    "num_attention_heads": 12,
+    "num_hidden_layers": 12,
+    "type_vocab_size": 0,
+    "num_labels": 15,
+    "vocab_size": 128128
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dbceb61b7dc9acbfc4119b4f43caeae34f45204af4a68101dab6bbf5beefc121
+size 372721635

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "sep_token": "</s>", "pad_token": "<pad>", "cls_token": "<s>", "mask_token": "<mask>"}

spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ec224fdac5b0511c6b901a79c0a4355f069b624dfc04400860d5d8fad4404e40
+size 2390173

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+    "do_lower_case": false,
+    "bos_token": "<s>",
+    "eos_token": "</s>",
+    "unk_token": "<unk>",
+    "sep_token": "</s>",
+    "pad_token": "<pad>",
+    "cls_token": "<s>",
+    "mask_token": "<mask>",
+    "split_by_punct": false,
+    "sp_model_kwargs": {},
+    "special_tokens_map_file": null,
+    "name_or_path": "/cognitive_comp/gaoxinyu/pretrained_model/deberta-base-sp",
+    "tokenizer_class": "DebertaV2Tokenizer"
+}