Add SetFit model

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": true,
+  "pooling_mode_mean_tokens": false,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false,
+  "pooling_mode_weightedmean_tokens": false,
+  "pooling_mode_lasttoken": false,
+  "include_prompt": true
+}

README.md

@@ -0,0 +1,286 @@
+---
+base_model: BAAI/bge-base-en-v1.5
+library_name: setfit
+metrics:
+- accuracy
+pipeline_tag: text-classification
+tags:
+- setfit
+- sentence-transformers
+- text-classification
+- generated_from_setfit_trainer
+widget:
+- text: 'The provided answer is quite disjointed and does not directly address the
+    specific question about accessing the company''s training resources. The listed
+    methods are more related to accessing various internal tools and processes rather
+    than directly answering the question. Let''s break down the problems:
+
+
+    1. **System and personal documents**: Mentions accessing personal documents, contracts,
+    and making reimbursements, which are not directly related to training resources.
+
+    2. **Password Manager (1Password)**: This piece of information about managing
+    passwords is irrelevant to accessing training resources.
+
+    3. **Tresorit**: Focuses on secure information sharing, not on training resources.
+
+    4. **Coffee session for feedback**: This is related to clarifying feedback, not
+    directly about accessing training resources.
+
+    5. **Learning Budget**: This section is somewhat relevant but is more about requesting
+    financial support for training rather than providing a direct method to access
+    training resources.
+
+
+    The answer needs to clearly outline specific steps or platforms dedicated to training
+    resources, which is mostly missing here.
+
+
+    Final Result: **Bad**'
+- text: 'The answer succinctly addresses the question by stating that finance@ORGANIZATION_2.<89312988>
+    should be contacted for questions about travel reimbursement. This is correctly
+    derived from the provided document, which specifies that questions about travel
+    costs and reimbursements should be directed to the finance email.
+
+
+    Final evaluation: Good'
+- text: 'The answer provided correctly mentions the essential aspects outlined in
+    the document, such as the importance for team leads to actively consider the possibility
+    of team members leaving, to flag these situations to HR, analyze problems, provide
+    feedback, and take proactive steps in various issues like underperformance or
+    lack of growth. The answer also captures the significance of creating a supportive
+    environment, maintaining alignment with the company''s vision and mission, ensuring
+    work-life balance, and providing regular feedback and praise.
+
+
+    However, while the answer is generally comprehensive, it could be slightly more
+    direct and concise in its communication. The document points to the necessity
+    for team leads to think about potential exits to preemptively address issues and
+    essentially prevent situations that may necessitate separation if they aren''t
+    managed well. This could have been emphasized more clearly.
+
+
+    Overall, the answer aligns well with the content and intent of the document.
+
+
+    Final evaluation: Good'
+- text: 'Reasoning:
+
+    The answer provided is relevant to the question as it directs the user to go to
+    the website and check job ads and newsletters for more information about ORGANIZATION.
+    However, it lacks comprehensive details. It only partially addresses how one can
+    understand ORGANIZATION''s product, challenges, and future since the documents
+    suggest accessing job ads and newsletters, and no further content or documents
+    were leveraged to provide insights into product details, current challenges, or
+    future plans.
+
+
+    Final evaluation: Bad'
+- text: 'Evaluation:
+
+    The answer provides a detailed response directly addressing the question, mentioning
+    that the ORGANIZATION_2 and key individuals like Thomas Barnes and Charlotte Herrera
+    play a supportive role in the farewell process. This includes handling paperwork,
+    providing guidance, and assisting with tough conversations. The answer aligns
+    well with the details provided in the document.
+
+
+    The final evaluation: Good'
+inference: true
+model-index:
+- name: SetFit with BAAI/bge-base-en-v1.5
+  results:
+  - task:
+      type: text-classification
+      name: Text Classification
+    dataset:
+      name: Unknown
+      type: unknown
+      split: test
+    metrics:
+    - type: accuracy
+      value: 0.6268656716417911
+      name: Accuracy
+---
+
+# SetFit with BAAI/bge-base-en-v1.5
+
+This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.
+
+The model has been trained using an efficient few-shot learning technique that involves:
+
+1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
+2. Training a classification head with features from the fine-tuned Sentence Transformer.
+
+## Model Details
+
+### Model Description
+- **Model Type:** SetFit
+- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
+- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
+- **Maximum Sequence Length:** 512 tokens
+- **Number of Classes:** 2 classes
+<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
+<!-- - **Language:** Unknown -->
+<!-- - **License:** Unknown -->
+
+### Model Sources
+
+- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
+- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
+- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)
+
+### Model Labels
+| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+|:------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| 0     | <ul><li>'The answer provided is a general approach to saving money, offering advice on spending wisely, making good investments, and considering personal goals. It also suggests discussing with a lead or ORGANIZATION_2 for guidance and highlights the importance of health and setting priorities.\n\nHowever, the answer lacks direct grounding in the provided document, which primarily discusses budgeting for studies and spending on learning and development. The document emphasizes evaluating expenditures based on their benefits to personal and organizational goals but does not offer explicit general financial saving tips.\n\nDue to this lack of specific correlation and grounding in the document, the evaluation is:\n\nBad'</li><li>'The answer provided reads incoherently due to the insertion of "Cassandra Rivera Heather Nelson" and other names in inappropriate places throughout the text, making it difficult to assess its alignment with the original document.\n\nFurthermore, the inserted names disrupt the meaning of the text, making it unclear if all relevant points from the document are covered accurately. The structure of the sentences becomes disjointed and presents an overall lack of clarity. \n\nGiven the nature of these errors, it\'s impossible to fairly evaluate whether the answer strictly adheres to the details provided in the source documents. Therefore, based on the clarity, coherence, and alignment to the original text, the final result is:\n\nBad'</li><li>"The provided answer does not satisfactorily respond to the question about accessing the company's training resources. Instead, it contains unrelated information about document management, security protocols, feedback processes, and learning budget requests. The relevant information about accessing training resources is clearly missing or obscure.\n\n**Reasoning:**\n\n1. **Irrelevant Details**: The answer includes details about the usage of a password manager, secure sharing of information, and expense reimbursement, none of which pertain to accessing training resources.\n\n2. **Lack of Specificity**: No explicit method or platform for accessing training resources is mentioned, which is the core inquiry.\n\n3. **Missed Key Point**: The document points towards systems used for personal documents and reimbursement requests but fails to highlight training resource access points.\n\nFinal evaluation: **Bad**"</li></ul>                                                                                                                                                                                                                                                                                 |
+| 1     | <ul><li>'The answer demonstrates a clear connection to the provided document, outlining key tips and principles for giving feedback as requested by the question. The response includes the importance of timing, focusing on the situation rather than the person, being clear and direct, and the goal of helping rather than shaming. It also mentions the importance of appreciation and receiving feedback with an open mind. \n\nHowever, there are some typographical errors and misplaced words (e.g., "emichelle James Johnson MDamples") that detract slightly from the clarity. Despite these minor issues, the content provided accurately reflects the information in the source document and comprehensively addresses the question. Therefore, the final evaluation is:\n\nGood'</li><li>"The given answer provides a general explanation of why it is important to proactively share information from high-level meetings, but it lacks grounding in the provided document. \n\nWhile the answer discusses the benefits of transparency, alignment with the organization's vision and mission, and fostering an open work environment, it does not directly quote or refer to specific points in the document. This weakens the argument, as it seems more like an independent explanation rather than an answer strictly based on the provided material.\n\nThe document mentions the importance of employees knowing why they are doing what they do and the necessity of closing the information gap by proactively sharing what was discussed in high-level meetings. This specific point from Document 4 could have been directly referenced to make the answer more aligned with the source material.\n\nThus, the answer, although conceptually correct, does not appropriately leverage the provided document.\n\nFinal result: Bad"</li><li>"The answer is partially correct since it provides the most important guidance on how to report car travel expenses for reimbursement. However, it contains inaccuracies and omissions that could cause confusion. \n\n1. The email addresses cited in the answer don’t match those in the document.\n2. There’s mention of requesting a parking card for a specific date (2004-04-14), which implies an inaccurate, irrelevant detail that might mislead the reader.\n3. The answer doesn't explicitly suggest that travel cost reimbursement is handled monthly, which is a crucial piece of information.\n\nConsidering these elements, the evaluation is as follows:\n\n**Evaluation:**\nThe essential details related to car travel reimbursement are present, but incorrect email addresses and irrelevant details might mislead or cause inconvenience for employees.\n\nThe final evaluation: **Bad**"</li></ul> |
+
+## Evaluation
+
+### Metrics
+| Label   | Accuracy |
+|:--------|:---------|
+| **all** | 0.6269   |
+
+## Uses
+
+### Direct Use for Inference
+
+First install the SetFit library:
+
+```bash
+pip install setfit
+```
+
+Then you can load this model and run inference.
+
+```python
+from setfit import SetFitModel
+
+# Download from the 🤗 Hub
+model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_cot-few_shot_only_reasoning_1726750220.456809")
+# Run inference
+preds = model("The answer succinctly addresses the question by stating that finance@ORGANIZATION_2.<89312988> should be contacted for questions about travel reimbursement. This is correctly derived from the provided document, which specifies that questions about travel costs and reimbursements should be directed to the finance email.
+
+Final evaluation: Good")
+```
+
+<!--
+### Downstream Use
+
+*List how someone could finetune this model on their own dataset.*
+-->
+
+<!--
+### Out-of-Scope Use
+
+*List how the model may foreseeably be misused and address what users ought not to do with the model.*
+-->
+
+<!--
+## Bias, Risks and Limitations
+
+*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
+-->
+
+<!--
+### Recommendations
+
+*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
+-->
+
+## Training Details
+
+### Training Set Metrics
+| Training set | Min | Median  | Max |
+|:-------------|:----|:--------|:----|
+| Word count   | 30  | 85.7538 | 210 |
+
+| Label | Training Sample Count |
+|:------|:----------------------|
+| 0     | 32                    |
+| 1     | 33                    |
+
+### Training Hyperparameters
+- batch_size: (16, 16)
+- num_epochs: (5, 5)
+- max_steps: -1
+- sampling_strategy: oversampling
+- num_iterations: 20
+- body_learning_rate: (2e-05, 2e-05)
+- head_learning_rate: 2e-05
+- loss: CosineSimilarityLoss
+- distance_metric: cosine_distance
+- margin: 0.25
+- end_to_end: False
+- use_amp: False
+- warmup_proportion: 0.1
+- l2_weight: 0.01
+- seed: 42
+- eval_max_steps: -1
+- load_best_model_at_end: False
+
+### Training Results
+| Epoch  | Step | Training Loss | Validation Loss |
+|:------:|:----:|:-------------:|:---------------:|
+| 0.0061 | 1    | 0.2304        | -               |
+| 0.3067 | 50   | 0.2556        | -               |
+| 0.6135 | 100  | 0.244         | -               |
+| 0.9202 | 150  | 0.1218        | -               |
+| 1.2270 | 200  | 0.0041        | -               |
+| 1.5337 | 250  | 0.0022        | -               |
+| 1.8405 | 300  | 0.0017        | -               |
+| 2.1472 | 350  | 0.0017        | -               |
+| 2.4540 | 400  | 0.0015        | -               |
+| 2.7607 | 450  | 0.0014        | -               |
+| 3.0675 | 500  | 0.0013        | -               |
+| 3.3742 | 550  | 0.0013        | -               |
+| 3.6810 | 600  | 0.0012        | -               |
+| 3.9877 | 650  | 0.0012        | -               |
+| 4.2945 | 700  | 0.0012        | -               |
+| 4.6012 | 750  | 0.0012        | -               |
+| 4.9080 | 800  | 0.0012        | -               |
+
+### Framework Versions
+- Python: 3.10.14
+- SetFit: 1.1.0
+- Sentence Transformers: 3.1.0
+- Transformers: 4.44.0
+- PyTorch: 2.4.1+cu121
+- Datasets: 2.19.2
+- Tokenizers: 0.19.1
+
+## Citation
+
+### BibTeX
+```bibtex
+@article{https://doi.org/10.48550/arxiv.2209.11055,
+    doi = {10.48550/ARXIV.2209.11055},
+    url = {https://arxiv.org/abs/2209.11055},
+    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
+    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
+    title = {Efficient Few-Shot Learning Without Prompts},
+    publisher = {arXiv},
+    year = {2022},
+    copyright = {Creative Commons Attribution 4.0 International}
+}
+```
+
+<!--
+## Glossary
+
+*Clearly define terms in order to be accessible across audiences.*
+-->
+
+<!--
+## Model Card Authors
+
+*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
+-->
+
+<!--
+## Model Card Contact
+
+*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
+-->

config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "BAAI/bge-base-en-v1.5",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "LABEL_0"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "LABEL_0": 0
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.44.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

config_sentence_transformers.json

@@ -0,0 +1,10 @@
+{
+  "__version__": {
+    "sentence_transformers": "3.1.0",
+    "transformers": "4.44.0",
+    "pytorch": "2.4.1+cu121"
+  },
+  "prompts": {},
+  "default_prompt_name": null,
+  "similarity_fn_name": null
+}

config_setfit.json

@@ -0,0 +1,4 @@
+{
+  "normalize_embeddings": false,
+  "labels": null
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2da6e45b31db132d81d4c5e76bd9f9d2f3e811065bba93c41e2b061bdc39a8c5
+size 437951328

model_head.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f3609d95f5b36a966b7dd5e98869dce62e93ac81633130bfb76299c51f945112
+size 7007

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.models.Normalize"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": true
+}

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,57 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}