Netta1994/setfit_baai_newrelic_gpt-4o_cot-few_shot-instructions_only_reasoning_1726750400.408156

SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: BAAI/bge-base-en-v1.5
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 512 tokens
• Number of Classes: 2 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
0	"Evaluation:\nThe answer is too general and lacks specificity regarding the organization's guidelines and criteria for spending money wisely. The provided documents are focused on the study and learning budget within the organization, and the answer should have included aspects such as considering whether an expenditure is a good investment, its alignment with personal and organizational goals, and the outlined budget guidelines.\n\n1. Context Grounding: The answer does touch upon wise spending and investments, but it lacks direct references to the specific criteria and guidelines outlined in the documents.\n2. Relevance: The answer is somewhat relevant but doesn't delve into the detailed criteria provided about spending the budget wisely.\n3. Conciseness: The answer is concise but misses key specifics from the document.\n4. Avoiding unanswerable responses: The question could be answered based on the document, and the answer does attempt to respond appropriately.\n5. Specificity and details: The answer misses details such as the control questions about personal and organizational benefits, the role and responsibility match, and the specific budget thresholds for various spending categories.\n\nFinal evaluation: Bad" 'Evaluation:\n\n1. Context Grounding: The answer is somewhat grounded in the context provided by the documents but not entirely accurate as it contains errors such as repeated names and unclear sentences.\n2. Relevance: The answer does address the question by listing pet peeves, yet it includes unnecessary names and errors that detract from its clarity.\n3. Conciseness: The response is not concise and includes redundant information, such as repeated names and irrelevant phrases.\n4. Clarity: The answer is marred by text errors (e.g., "Cassandra Rivera Heather Nelson"), which obscure its clarity and readability.\n5. Specificity: It does list specific pet peeves mentioned in the document but lacks proper structure and clarity.\n\nFinal evaluation: Bad' 'The provided answer is inadequate for several reasons:\n\n1. Context Grounding: The answer makes vague references to multiple unrelated sections, rather than directly addressing how to access training resources. The mentioned documents do not collectively substantiate the response about accessing training resources.\n\n2. Relevance: The instructions involve details about reimbursement, security practices, and learning budgets, but they do not respond directly to the query about accessing training resources.\n\n3. Conciseness: The answer includes numerous irrelevant points such as the use of 1Password for personal account management and feedback processing, which are not pertinent to accessing training resources.\n\n4. Specificity: The answer does not provide specific steps or tips directly related to accessing the training resources of the company. It rather diverts into various other topics.\n\n5. Following Instructions: The given documents offer information about other aspects of work at the company but lack direct guidance on accessing training resources. Thus, attempting to answer the question without appropriate grounding in the document misleads the response.\n\nTherefore, the answer fails to meet the necessary criteria and should be evaluated as:\n\nBad'
1	'Evaluation:\nThe provided answer addresses the specific question of how feedback should be given according to the provided tips. It covers key points from the document such as giving feedback at the time of the event, making it situational rather than personal, reflecting on the intention of the feedback, being clear and direct, and showing appreciation. Additionally, it touches on tips for receiving feedback, aligning well with the original document.\n\nHowever, the answer includes erroneous and irrelevant information due to text misanalyzation such as "Christopher Estes time" which should not be part of the answer. Furthermore, the mention of "emichelle James Johnson MDamples" appears to be a corrupted text which doesn’t make sense in the context.\n\nFinal evaluation: Bad' 'Evaluation:\nThe answer accurately addresses the question of why it is important to proactively share information from high-level meetings. It discusses transparency, alignment, sense of purpose, and open communication, which are all points grounded in the provided documents. The answer is relevant, concise, and specific enough, capturing the essence of the reasons mentioned in Document 4. No irrelevant information is included, and the answer contains specific tips related to the importance of sharing high-level meeting details.\n\nFinal evaluation: Good' 'Evaluation:\nThe provided answer is generally well-grounded in the document, sticking closely to the instructions given for reporting car travel expenses for reimbursement. However, there are a few inaccuracies and incomplete information:\n\n1. Context Grounding: The answer is largely based on the content of the document but not entirely accurate. The provided email addresses seem to be incorrectly formatted (e.g., "finance@Dustin [email protected]" should probably be "[email protected]@example.net").\n2. Relevance: The response is relevant to the question asked and sticks to the specific procedure for car travel expense reporting.\n3. Conciseness: The answer is concise but could be more accurate in presenting the information.\n4. Specificity: The document also mentions tracking kilometers and the reimbursement rate, which the answer includes. However, it fails to mention all the specific details accurately.\n\nFinal evaluation: Bad'

Evaluation

Metrics

Label	Accuracy
all	0.6119

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_cot-few_shot-instructions_only_reasoning_1726750400.408156")
# Run inference
preds = model("Evaluation:
The answer provided is concise and directly addresses the question of whom to contact for travel reimbursement questions. It correctly refers to the email address provided in the document. The answer is well-supported by the document and does not deviate into unrelated topics.

The final evaluation: Good")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	30	106.3538	221

Label	Training Sample Count
0	32
1	33

Training Hyperparameters

• batch_size: (16, 16)
• num_epochs: (5, 5)
• max_steps: -1
• sampling_strategy: oversampling
• num_iterations: 20
• body_learning_rate: (2e-05, 2e-05)
• head_learning_rate: 2e-05
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• l2_weight: 0.01
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0061	1	0.2332	-
0.3067	50	0.2674	-
0.6135	100	0.2116	-
0.9202	150	0.0354	-
1.2270	200	0.0036	-
1.5337	250	0.0022	-
1.8405	300	0.0017	-
2.1472	350	0.0016	-
2.4540	400	0.0015	-
2.7607	450	0.0013	-
3.0675	500	0.0013	-
3.3742	550	0.0012	-
3.6810	600	0.0012	-
3.9877	650	0.0011	-
4.2945	700	0.0012	-
4.6012	750	0.0011	-
4.9080	800	0.0011	-

Framework Versions

• Python: 3.10.14
• SetFit: 1.1.0
• Sentence Transformers: 3.1.0
• Transformers: 4.44.0
• PyTorch: 2.4.1+cu121
• Datasets: 2.19.2
• Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}