add files

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 312,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

2_Dense/config.json

@@ -0,0 +1 @@
+{"in_features": 312, "out_features": 128, "bias": true, "activation_function": "torch.nn.modules.activation.Tanh"}

2_Dense/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:daf2b76d4fa71b4af3c97c636de43e2ee82357731092e45cf7f7863c6be85692
+size 161319

README.md

@@ -1,3 +1,78 @@
----
-license: apache-2.0
----
+
+# HPD-TinyBERT-F128
+
+This repository contains the pre-trained models for our paper [Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation](https://arxiv.org/abs/2203.07687).
+
+## Overview
+
+We propose **H**omomorphic **P**rojective **D**istillation (HPD) to learn compressed sentence embeddings. Our method augments a small Transformer encoder model with learnable projection layers to produce compact representations while mimicking a large pre-trained language model to retain the sentence representation quality.
+
+## Details
+
+This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 128 dimensional dense vector space and can be used for tasks like clustering or semantic search.
+
+The teacher model is [`princeton-nlp/sup-simcse-roberta-large`](https://huggingface.co/princeton-nlp/sup-simcse-bert-base-uncased) and the student model is [`nreimers/TinyBERT_L-4_H-312_v2`](https://huggingface.co/nreimers/TinyBERT_L-4_H-312_v2).
+
+## Usage
+
+Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+
+```
+pip install -U sentence-transformers
+```
+
+After installing the package, you can simply load our model
+```python
+from sentence_transformers import SentenceTransformer
+model = SentenceTransformer('Xuandong/HPD-TinyBERT-F128')
+```
+
+Then you can use our model for **encoding sentences into embeddings**
+```python
+sentences = ['He plays guitar.', 'A street vendor is outside.', 'A woman is reading.']
+sentence_embeddings = model.encode(sentences)
+
+for sentence, embedding in zip(sentences, sentence_embeddings):
+    print("Sentence:", sentence)
+    print("Embedding:", embedding)
+    print("")
+```
+
+## Evaluation Results
+
+We evaluate our model on semantic textual similarity (STS) tasks. The results are:
+
+```
++-------+-------+-------+-------+-------+--------------+-----------------+-------+
+| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness |  Avg. |
++-------+-------+-------+-------+-------+--------------+-----------------+-------+
+| 74.29 | 83.05 | 78.80 | 84.62 | 81.17 |    84.36     |      80.83      | 81.02 |
++-------+-------+-------+-------+-------+--------------+-----------------+-------+
+```
+
+## Training
+
+Please refer to the github repo (https://github.com/XuandongZhao/HPD) for the details about the training.
+
+
+## Full Model Architecture
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
+  (1): Pooling({'word_embedding_dimension': 312, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
+  (2): Dense({'in_features': 312, 'out_features': 128, 'bias': True, 'activation_function': 'torch.nn.modules.activation.Tanh'})
+)
+```
+
+## Citation
+
+Please cite our paper if you use HPD in your work:
+
+```bibtex
+@article{zhao2022compressing,
+  title={Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation},
+  author={Zhao, Xuandong and Yu, Zhiguo and Wu, Ming and Li, Lei},
+  journal={arXiv preprint arXiv:2203.07687},
+  year={2022}
+}
+```

config.json

@@ -0,0 +1,26 @@
+{
+  "_name_or_path": "Xuandong/HPD-TinyBERT-F128",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 312,
+  "initializer_range": 0.02,
+  "intermediate_size": 1200,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 4,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.11.3",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.0.0",
+    "transformers": "4.11.3",
+    "pytorch": "1.10.0+cu113"
+  }
+}

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Dense",
+    "type": "sentence_transformers.models.Dense"
+  }
+]

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f06d56cfa2d491c0d9351105f75c1091795a77049f849b9dfd8d9be2aea808ab
+size 57431373

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 512, "name_or_path": "Xuandong/HPD-TinyBERT-F128", "do_basic_tokenize": true, "never_split": null, "tokenizer_class": "BertTokenizer"}