Upload folder using huggingface_hub

README.md

@@ -1,68 +1,57 @@
 ---
-library_name: transformers
+language: en
 tags:
-- generated_from_trainer
-model-index:
-- name: bert-reg-biencoder-mae
-  results: []
+- bert
+- regression
+- biencoder
+- similarity
+pipeline_tag: text-similarity
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
-# bert-reg-biencoder-mae
-
-This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.2343
-- Mse: 0.0824
-- Mae: 0.2338
-- Pearson Corr: 0.2477
-- Spearman Corr: 0.1374
-- Cosine Sim: 0.9022
-
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- learning_rate: 2e-05
-- train_batch_size: 32
-- eval_batch_size: 32
-- seed: 42
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: linear
-- lr_scheduler_warmup_steps: 100
-- num_epochs: 7
-
-### Training results
-
-| Training Loss | Epoch | Step | Validation Loss | Mse    | Mae    | Pearson Corr | Spearman Corr | Cosine Sim |
-|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:------------:|:-------------:|:----------:|
-| 0.2846        | 1.0   | 21   | 0.2617          | 0.1153 | 0.2610 | 0.1327       | 0.0936        | 0.9053     |
-| 0.2728        | 2.0   | 42   | 0.2310          | 0.0886 | 0.2304 | 0.0188       | 0.0316        | 0.8994     |
-| 0.2511        | 3.0   | 63   | 0.2282          | 0.0847 | 0.2276 | 0.1716       | 0.1111        | 0.9058     |
-| 0.2253        | 4.0   | 84   | 0.2333          | 0.0864 | 0.2329 | 0.1906       | 0.1191        | 0.9041     |
-| 0.1993        | 5.0   | 105  | 0.2329          | 0.0822 | 0.2326 | 0.2299       | 0.1213        | 0.9016     |
-| 0.1845        | 6.0   | 126  | 0.2357          | 0.0829 | 0.2353 | 0.2268       | 0.1254        | 0.9017     |
-| 0.165         | 7.0   | 147  | 0.2343          | 0.0824 | 0.2338 | 0.2477       | 0.1374        | 0.9022     |
-
-
-### Framework versions
-
-- Transformers 4.45.1
-- Pytorch 2.4.0
-- Datasets 3.0.1
-- Tokenizers 0.20.0
+# BiEncoder Regression Model
+
+This model is a BiEncoder architecture that outputs similarity scores between text pairs.
+
+## Model Details
+- Base Model: bert-base-uncased
+- Task: Regression
+- Architecture: BiEncoder with cosine similarity
+- Loss Function: mae
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModel
+from modeling import BiEncoderModelRegression
+
+# Load model components
+tokenizer = AutoTokenizer.from_pretrained("minoosh/bert-reg-biencoder-mae")
+base_model = AutoModel.from_pretrained("bert-base-uncased")
+model = BiEncoderModelRegression(base_model, loss_fn="mae")
+
+# Load weights
+state_dict = torch.load("pytorch_model.bin")
+model.load_state_dict(state_dict)
+
+# Prepare inputs
+texts1 = ["first text"]
+texts2 = ["second text"]
+inputs = tokenizer(
+    texts1, texts2,
+    padding=True,
+    truncation=True,
+    return_tensors="pt"
+)
+
+# Get similarity scores
+outputs = model(**inputs)
+similarity_scores = outputs["logits"]
+```
+
+## Metrics
+The model was trained using mae loss and evaluated using:
+- Mean Squared Error (MSE)
+- Mean Absolute Error (MAE)
+- Pearson Correlation
+- Spearman Correlation
+- Cosine Similarity

config.json

@@ -0,0 +1 @@
+{"return_dict": true, "output_hidden_states": false, "output_attentions": false, "torchscript": false, "torch_dtype": null, "use_bfloat16": false, "tf_legacy_loss": false, "pruned_heads": {}, "tie_word_embeddings": true, "chunk_size_feed_forward": 0, "is_encoder_decoder": false, "is_decoder": false, "cross_attention_hidden_size": null, "add_cross_attention": false, "tie_encoder_decoder": false, "max_length": 20, "min_length": 0, "do_sample": false, "early_stopping": false, "num_beams": 1, "num_beam_groups": 1, "diversity_penalty": 0.0, "temperature": 1.0, "top_k": 50, "top_p": 1.0, "typical_p": 1.0, "repetition_penalty": 1.0, "length_penalty": 1.0, "no_repeat_ngram_size": 0, "encoder_no_repeat_ngram_size": 0, "bad_words_ids": null, "num_return_sequences": 1, "output_scores": false, "return_dict_in_generate": false, "forced_bos_token_id": null, "forced_eos_token_id": null, "remove_invalid_values": false, "exponential_decay_length_penalty": null, "suppress_tokens": null, "begin_suppress_tokens": null, "architectures": ["BiEncoderModelRegression"], "finetuning_task": null, "id2label": {"0": "LABEL_0", "1": "LABEL_1"}, "label2id": {"LABEL_0": 0, "LABEL_1": 1}, "tokenizer_class": null, "prefix": null, "bos_token_id": null, "pad_token_id": 0, "eos_token_id": null, "sep_token_id": null, "decoder_start_token_id": null, "task_specific_params": null, "problem_type": null, "_name_or_path": "bert-base-uncased", "transformers_version": "4.45.1", "gradient_checkpointing": false, "model_type": "bert", "vocab_size": 30522, "hidden_size": 768, "num_hidden_layers": 12, "num_attention_heads": 12, "hidden_act": "gelu", "intermediate_size": 3072, "hidden_dropout_prob": 0.1, "attention_probs_dropout_prob": 0.1, "max_position_embeddings": 512, "type_vocab_size": 2, "initializer_range": 0.02, "layer_norm_eps": 1e-12, "position_embedding_type": "absolute", "use_cache": true, "classifier_dropout": null, "loss_fn": "mae", "task_type": "regression", "is_regression": true}

data_collator.py

@@ -0,0 +1,13 @@
+
+import torch
+
+class BiEncoderCollator:
+    def __call__(self, features):
+        batch = {
+            'input_ids_text1': torch.stack([f['input_ids_text1'] for f in features]),
+            'attention_mask_text1': torch.stack([f['attention_mask_text1'] for f in features]),
+            'input_ids_text2': torch.stack([f['input_ids_text2'] for f in features]),
+            'attention_mask_text2': torch.stack([f['attention_mask_text2'] for f in features]),
+            'labels': torch.tensor([f['labels'] for f in features], dtype=torch.float)
+        }
+        return batch

modeling.py

@@ -0,0 +1,35 @@
+
+import torch
+from transformers import PreTrainedModel
+
+class BiEncoderModelRegression(torch.nn.Module):
+    def __init__(self, base_model, config=None, loss_fn="mse"):
+        super().__init__()
+        self.base_model = base_model
+        self.cos = torch.nn.CosineSimilarity(dim=1)
+        self.loss_fn = loss_fn
+        self.config = config
+
+    def forward(self, input_ids_text1, attention_mask_text1, input_ids_text2, attention_mask_text2, labels=None):
+        outputs_text1 = self.base_model(input_ids_text1, attention_mask=attention_mask_text1)
+        outputs_text2 = self.base_model(input_ids_text2, attention_mask=attention_mask_text2)
+        
+        cls_embedding_text1 = outputs_text1.last_hidden_state[:, 0, :]
+        cls_embedding_text2 = outputs_text2.last_hidden_state[:, 0, :]
+        
+        cos_sim = self.cos(cls_embedding_text1, cls_embedding_text2)
+        
+        loss = None
+        if labels is not None:
+            if self.loss_fn == "mse":
+                loss_fct = torch.nn.MSELoss()
+            elif self.loss_fn == "mae":
+                loss_fct = torch.nn.L1Loss()
+            elif self.loss_fn == "cosine_embedding":
+                loss_fct = torch.nn.CosineEmbeddingLoss()
+                labels_cosine = 2 * (labels > 0.5).float() - 1
+                return {"loss": loss_fct(cls_embedding_text1, cls_embedding_text2, labels_cosine), "logits": cos_sim}
+            
+            loss = loss_fct(cos_sim, labels)
+            
+        return {"loss": loss, "logits": cos_sim}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2edbbc43f6a733ecb99fc799b1cedc23d5d4b8549ddaa4f9b0e4220e228a642e
+size 438013734

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,55 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}