Add SetFit model

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": true,
+  "pooling_mode_mean_tokens": false,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false,
+  "pooling_mode_weightedmean_tokens": false,
+  "pooling_mode_lasttoken": false,
+  "include_prompt": true
+}

README.md

@@ -0,0 +1,290 @@
+---
+base_model: BAAI/bge-base-en-v1.5
+library_name: setfit
+metrics:
+- accuracy
+pipeline_tag: text-classification
+tags:
+- setfit
+- sentence-transformers
+- text-classification
+- generated_from_setfit_trainer
+widget:
+- text: "Evaluation:\n\n1. **Context Grounding**: The answer references the document,\
+    \ but it consists of multiple points that are not directly related to accessing\
+    \ training resources. While some methods mentioned might facilitate training (e.g.,\
+    \ learning budget), others (like using 1Password, Tresorit) are not directly relevant\
+    \ to accessing training resources.\n   \n2. **Relevance**: The answer partially\
+    \ addresses the question. Points about accessing documents or requesting a learning\
+    \ budget are somewhat related, but the inclusion of security tools and password\
+    \ managers is irrelevant to the question of accessing training resources.\n\n\
+    3. **Conciseness**: The answer includes unnecessary details that do not directly\
+    \ answer the question, making it lengthy and off-point in segments.\n\n4. **Specificity**:\
+    \ The answer is a mix of specific steps and unrelated information. It fails to\
+    \ provide a direct method to access training resources comprehensively.\n\n5.\
+    \ **General Tips**: The answer does provide a step about talking to a lead for\
+    \ a learning budget, which can be considered a relevant tip. However, it's buried\
+    \ in a lot of unrelated content.\n\nOverall, the answer deviates too much from\
+    \ the specific question about accessing training resources and includes information\
+    \ not requested nor directly relevant.\n\nFinal result: **Bad**"
+- text: 'Evaluation:
+
+    The answer provided is concise and directly addresses the question of whom to
+    contact for travel reimbursement questions. It correctly refers to the email address
+    provided in the document. The answer is well-supported by the document and does
+    not deviate into unrelated topics.
+
+
+    The final evaluation: Good'
+- text: 'Reasoning:
+
+
+    1. **Context Grounding**: The answer accurately references content from the provided
+    documents, especially the points related to actively thinking about the possibility
+    of someone leaving, flagging it to HR, analyzing problems, and providing feedback.
+
+
+    2. **Relevance**: The answer directly addresses the question by outlining why
+    it is crucial for team leads to consider the possibility of staff leaving and
+    the steps they can take to mitigate issues early on.
+
+
+    3. **Conciseness**: The answer is relatively concise but gets slightly verbose
+    towards the end. The core points are clearly made without too much unnecessary
+    information.
+
+
+    4. **Specificity**: The answer includes specific reasons like addressing underperformance,
+    lack of growth, disagreement with direction, and maintaining a supportive environment,
+    all of which are well-supported by the documents.
+
+
+    5. **Avoiding Generality**: The answer provides detailed steps and reasons as
+    mentioned in the documents, avoiding overly general statements.
+
+
+    Final Result: **Good**'
+- text: 'Evaluation:
+
+    The answer addresses the question partially by suggesting ways to learn about
+    ORGANIZATION through their website and available job postings, which are referenced
+    in the documents. However, it misses more specific ways to understand their products
+    and challenges; for example, details from the document about stress management,
+    inclusivity issues, and organizational changes could provide better insights into
+    their future and challenges. Additionally, a newsletter about "behind the scenes"
+    is more about updates rather than specific details on products and challenges.
+    Therefore, the answer lacks completeness and specificity.
+
+
+    The final evaluation: Bad'
+- text: "**Evaluation Reasoning:**\n\n1. **Context Grounding:** The answer is well-supported\
+    \ by the provided documents, especially Document 1. It correctly identifies the\
+    \ responsibilities of ORGANIZATION_2, Thomas Barnes, and Charlotte Herrera.\n\
+    \   \n2. **Relevance:** The answer addresses the specific question about the role\
+    \ of ORGANIZATION_2 in the farewell process.\n   \n3. **Conciseness:** The response\
+    \ includes some redundant information, especially the repetition of individuals\
+    \ involved and some operational details not directly related to the extent of\
+    \ ORGANIZATION_2's involvement.\n   \n4. **Specificity:** It clearly outlines\
+    \ the roles and describes the process for different situations, including handling\
+    \ paperwork, process, and tough conversations.\n   \n5. **General vs. Specific\
+    \ Tips:** The answer could have been more concise regarding the involvement extent\
+    \ but remains within bounds.\n\n**Final Evaluation: Good**"
+inference: true
+model-index:
+- name: SetFit with BAAI/bge-base-en-v1.5
+  results:
+  - task:
+      type: text-classification
+      name: Text Classification
+    dataset:
+      name: Unknown
+      type: unknown
+      split: test
+    metrics:
+    - type: accuracy
+      value: 0.6119402985074627
+      name: Accuracy
+---
+
+# SetFit with BAAI/bge-base-en-v1.5
+
+This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.
+
+The model has been trained using an efficient few-shot learning technique that involves:
+
+1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
+2. Training a classification head with features from the fine-tuned Sentence Transformer.
+
+## Model Details
+
+### Model Description
+- **Model Type:** SetFit
+- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
+- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
+- **Maximum Sequence Length:** 512 tokens
+- **Number of Classes:** 2 classes
+<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
+<!-- - **Language:** Unknown -->
+<!-- - **License:** Unknown -->
+
+### Model Sources
+
+- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
+- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
+- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)
+
+### Model Labels
+| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
+|:------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| 0     | <ul><li>"Evaluation:\nThe answer is too general and lacks specificity regarding the organization's guidelines and criteria for spending money wisely. The provided documents are focused on the study and learning budget within the organization, and the answer should have included aspects such as considering whether an expenditure is a good investment, its alignment with personal and organizational goals, and the outlined budget guidelines.\n\n1. **Context Grounding:** The answer does touch upon wise spending and investments, but it lacks direct references to the specific criteria and guidelines outlined in the documents.\n2. **Relevance:** The answer is somewhat relevant but doesn't delve into the detailed criteria provided about spending the budget wisely.\n3. **Conciseness:** The answer is concise but misses key specifics from the document.\n4. **Avoiding unanswerable responses:** The question could be answered based on the document, and the answer does attempt to respond appropriately.\n5. **Specificity and details:** The answer misses details such as the control questions about personal and organizational benefits, the role and responsibility match, and the specific budget thresholds for various spending categories.\n\nFinal evaluation: Bad"</li><li>'Evaluation:\n\n1. **Context Grounding:** The answer is somewhat grounded in the context provided by the documents but not entirely accurate as it contains errors such as repeated names and unclear sentences.\n2. **Relevance:** The answer does address the question by listing pet peeves, yet it includes unnecessary names and errors that detract from its clarity.\n3. **Conciseness:** The response is not concise and includes redundant information, such as repeated names and irrelevant phrases.\n4. **Clarity:** The answer is marred by text errors (e.g., "Cassandra Rivera Heather Nelson"), which obscure its clarity and readability.\n5. **Specificity:** It does list specific pet peeves mentioned in the document but lacks proper structure and clarity.\n\nFinal evaluation: Bad'</li><li>'The provided answer is inadequate for several reasons:\n\n1. **Context Grounding**: The answer makes vague references to multiple unrelated sections, rather than directly addressing how to access training resources. The mentioned documents do not collectively substantiate the response about accessing training resources.\n\n2. **Relevance**: The instructions involve details about reimbursement, security practices, and learning budgets, but they do not respond directly to the query about accessing training resources.\n\n3. **Conciseness**: The answer includes numerous irrelevant points such as the use of 1Password for personal account management and feedback processing, which are not pertinent to accessing training resources.\n\n4. **Specificity**: The answer does not provide specific steps or tips directly related to accessing the training resources of the company. It rather diverts into various other topics.\n\n5. **Following Instructions**: The given documents offer information about other aspects of work at the company but lack direct guidance on accessing training resources. Thus, attempting to answer the question without appropriate grounding in the document misleads the response.\n\nTherefore, the answer fails to meet the necessary criteria and should be evaluated as:\n\n**Bad**'</li></ul> |
+| 1     | <ul><li>'Evaluation:\nThe provided answer addresses the specific question of how feedback should be given according to the provided tips. It covers key points from the document such as giving feedback at the time of the event, making it situational rather than personal, reflecting on the intention of the feedback, being clear and direct, and showing appreciation. Additionally, it touches on tips for receiving feedback, aligning well with the original document.\n\nHowever, the answer includes erroneous and irrelevant information due to text misanalyzation such as "Christopher Estes time" which should not be part of the answer. Furthermore, the mention of "emichelle James Johnson MDamples" appears to be a corrupted text which doesn’t make sense in the context.\n\nFinal evaluation: Bad'</li><li>'Evaluation:\nThe answer accurately addresses the question of why it is important to proactively share information from high-level meetings. It discusses transparency, alignment, sense of purpose, and open communication, which are all points grounded in the provided documents. The answer is relevant, concise, and specific enough, capturing the essence of the reasons mentioned in Document 4. No irrelevant information is included, and the answer contains specific tips related to the importance of sharing high-level meeting details.\n\nFinal evaluation: Good'</li><li>'Evaluation:\nThe provided answer is generally well-grounded in the document, sticking closely to the instructions given for reporting car travel expenses for reimbursement. However, there are a few inaccuracies and incomplete information:\n\n1. **Context Grounding**: The answer is largely based on the content of the document but not entirely accurate. The provided email addresses seem to be incorrectly formatted (e.g., "finance@Dustin [email protected]" should probably be "[email protected]@example.net").\n2. **Relevance**: The response is relevant to the question asked and sticks to the specific procedure for car travel expense reporting.\n3. **Conciseness**: The answer is concise but could be more accurate in presenting the information.\n4. **Specificity**: The document also mentions tracking kilometers and the reimbursement rate, which the answer includes. However, it fails to mention all the specific details accurately.\n\nFinal evaluation: Bad'</li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+
+## Evaluation
+
+### Metrics
+| Label   | Accuracy |
+|:--------|:---------|
+| **all** | 0.6119   |
+
+## Uses
+
+### Direct Use for Inference
+
+First install the SetFit library:
+
+```bash
+pip install setfit
+```
+
+Then you can load this model and run inference.
+
+```python
+from setfit import SetFitModel
+
+# Download from the 🤗 Hub
+model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_cot-few_shot-instructions_only_reasoning_1726750400.408156")
+# Run inference
+preds = model("Evaluation:
+The answer provided is concise and directly addresses the question of whom to contact for travel reimbursement questions. It correctly refers to the email address provided in the document. The answer is well-supported by the document and does not deviate into unrelated topics.
+
+The final evaluation: Good")
+```
+
+<!--
+### Downstream Use
+
+*List how someone could finetune this model on their own dataset.*
+-->
+
+<!--
+### Out-of-Scope Use
+
+*List how the model may foreseeably be misused and address what users ought not to do with the model.*
+-->
+
+<!--
+## Bias, Risks and Limitations
+
+*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
+-->
+
+<!--
+### Recommendations
+
+*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
+-->
+
+## Training Details
+
+### Training Set Metrics
+| Training set | Min | Median   | Max |
+|:-------------|:----|:---------|:----|
+| Word count   | 30  | 106.3538 | 221 |
+
+| Label | Training Sample Count |
+|:------|:----------------------|
+| 0     | 32                    |
+| 1     | 33                    |
+
+### Training Hyperparameters
+- batch_size: (16, 16)
+- num_epochs: (5, 5)
+- max_steps: -1
+- sampling_strategy: oversampling
+- num_iterations: 20
+- body_learning_rate: (2e-05, 2e-05)
+- head_learning_rate: 2e-05
+- loss: CosineSimilarityLoss
+- distance_metric: cosine_distance
+- margin: 0.25
+- end_to_end: False
+- use_amp: False
+- warmup_proportion: 0.1
+- l2_weight: 0.01
+- seed: 42
+- eval_max_steps: -1
+- load_best_model_at_end: False
+
+### Training Results
+| Epoch  | Step | Training Loss | Validation Loss |
+|:------:|:----:|:-------------:|:---------------:|
+| 0.0061 | 1    | 0.2332        | -               |
+| 0.3067 | 50   | 0.2674        | -               |
+| 0.6135 | 100  | 0.2116        | -               |
+| 0.9202 | 150  | 0.0354        | -               |
+| 1.2270 | 200  | 0.0036        | -               |
+| 1.5337 | 250  | 0.0022        | -               |
+| 1.8405 | 300  | 0.0017        | -               |
+| 2.1472 | 350  | 0.0016        | -               |
+| 2.4540 | 400  | 0.0015        | -               |
+| 2.7607 | 450  | 0.0013        | -               |
+| 3.0675 | 500  | 0.0013        | -               |
+| 3.3742 | 550  | 0.0012        | -               |
+| 3.6810 | 600  | 0.0012        | -               |
+| 3.9877 | 650  | 0.0011        | -               |
+| 4.2945 | 700  | 0.0012        | -               |
+| 4.6012 | 750  | 0.0011        | -               |
+| 4.9080 | 800  | 0.0011        | -               |
+
+### Framework Versions
+- Python: 3.10.14
+- SetFit: 1.1.0
+- Sentence Transformers: 3.1.0
+- Transformers: 4.44.0
+- PyTorch: 2.4.1+cu121
+- Datasets: 2.19.2
+- Tokenizers: 0.19.1
+
+## Citation
+
+### BibTeX
+```bibtex
+@article{https://doi.org/10.48550/arxiv.2209.11055,
+    doi = {10.48550/ARXIV.2209.11055},
+    url = {https://arxiv.org/abs/2209.11055},
+    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
+    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
+    title = {Efficient Few-Shot Learning Without Prompts},
+    publisher = {arXiv},
+    year = {2022},
+    copyright = {Creative Commons Attribution 4.0 International}
+}
+```
+
+<!--
+## Glossary
+
+*Clearly define terms in order to be accessible across audiences.*
+-->
+
+<!--
+## Model Card Authors
+
+*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
+-->
+
+<!--
+## Model Card Contact
+
+*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
+-->

config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "BAAI/bge-base-en-v1.5",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "LABEL_0"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "LABEL_0": 0
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.44.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

config_sentence_transformers.json

@@ -0,0 +1,10 @@
+{
+  "__version__": {
+    "sentence_transformers": "3.1.0",
+    "transformers": "4.44.0",
+    "pytorch": "2.4.1+cu121"
+  },
+  "prompts": {},
+  "default_prompt_name": null,
+  "similarity_fn_name": null
+}

config_setfit.json

@@ -0,0 +1,4 @@
+{
+  "normalize_embeddings": false,
+  "labels": null
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6bcb063f17262bad0004e0f65e27e3b8e1a78f692d14323520500efc13cdc329
+size 437951328

model_head.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6e15f1c9007ad31aa3e1b326654090fda1b2bb461006f5640d24ed4d0eb39eb
+size 7007

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.models.Normalize"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": true
+}

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,57 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}