SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: BAAI/bge-base-en-v1.5
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
0	"Reasoning: \n\n1. Context Grounding: The answer pulls loosely from the documents provided but does not ground the response specifically to any of the documents' key points about budgeting or investment decisions.\n2. Relevance: While the query is about saving money in general, the answer ambiguously addresses saving money by suggesting wise spending and seeking guidance, which are indirectly related to what's mentioned in the documents.\n3. Conciseness: The answer contains unnecessary details like mentioning health and priorities, which are somewhat related based on Document 3 but are not directly in line with the exact scope of the question.\n4. Too General: The answer is broad and generic. The document contains specific guidelines (such as asking if the investment would be made with personal money and consulting with ORGANIZATION_2 for high-stakes cases), which the response fails to integrate effectively.\n5. Detail Inclusion: The answer needed to incorporate specific strategies mentioned in the documents, like ensuring if an expense aligns with personal and organizational benefits or seeking guidance on higher-budget items.\n\nFinal result: Bad" 'Reasoning:\n\n1. Context Grounding: The answer does mention several pet peeves directly from the provided documents, though it incorrectly repeats the names and has some formatting errors, it aligns generally with the content.\n2. Relevance: The answer focuses on the specific question about pet peeves, summarizing relevant points from the documents.\n3. Conciseness: The answer is cluttered with repeated names and formatting errors that make it unnecessarily lengthy and harder to read.\n4. Avoiding unfounded responses: The answer is founded on the document but suffers from poor editing issues.\n5. Specificity: The answer includes specific pet peeves, but it also includes unnecessary repetition.\n6. Tip Inclusion: The query did not ask for tips, so this criterion is not strictly applicable.\n\nFinal result: Bad\n\nThe repeated names and formatting errors indicate lack of editing and care in the response, making it difficult to read and comprehend fully. It does not meet the required standard for a good, concise, and clear response.' "Reasoning:\n\n1. Context Grounding: The answer only indirectly relates to the question. The provided information gives various ways to interact with the company's systems or tools rather than specifically how to access the training resources.\n2. Relevance: The answer deviates from the main question. It mentions systems for personal documents, expense claims, password management, secure sharing, feedback processes, and learning budgets, none of which directly answer how to access training resources.\n3. Conciseness: While the answer is detailed, it includes unnecessary information that isn't directly related to accessing training resources, making it not concise.\n4. Specificity: The answer lacks details on accessing training materials or platforms where these resources are housed.\n5. Generalization: It provides general guidance on several aspects but fails to narrow down specifically on training resources or offer relevant tips directly tied to them.\n\nFinal result: Bad"
1	'Reasoning:\n\n1. Context Grounding: The answer references various elements from the documents provided, such as giving feedback at the time of the event, focusing on the situation and not the person, and showing appreciation. It generally aligns with the tips described in the documents.\n \n2. Relevance: The answer directly addresses the question of how feedback should be given, touching on many of the specific points mentioned in the documents.\n\n3. Conciseness: The response is overly wordy and includes some repetition (e.g., focusing on the situation not the person and the intention behind giving feedback). It can be more concise while still covering the necessary points.\n\n4. Document Dependency: The response accurately reflects the content of the provided documents. However, it brings in names like "Christopher Estes," "emichelle James Johnson MD," which may be elements of the document's placeholders rather than relevant names in the context of feedback tips. This adds unnecessary confusion to the answer.\n\n5. Specificity: The answer appropriately includes specific advice, though it overdoes some parts (like the role and intentions in giving feedback).\n\n6. Accuracy and Clarity: There are minor grammatical errors and some parts like "emichelle James Johnson MDamples" which are not clear and probably are typos or formatting errors.\n\nGiven the evaluation above:\n\nFinal Result: Bad' 'Reasoning:\n1. Context Grounding: The answer refers to sharing information from high-level meetings which is supported by Document 4. However, the inclusion of "Robin Nichols" appears to be an error or typo, which diminishes the context grounding slightly.\n2. Relevance: The answer is relevant to the question asked; it covers the need for transparency, team alignment, addressing concerns, and fostering an open environment, which is in line with the information in Document 4.\n3. Conciseness: The answer is moderately concise but slightly verbose. It can be trimmed down while still maintaining the key points.\n4. Directness: The answer attempts to respond to the question based on the content provided in Document 4.\n5. Specificity: The answer includes specific points about decisions made and the rationale being shared, aligning the team and fostering a sense of purpose.\n6. Tips: The answer implicitly includes tips such as ensuring transparency and addressing team concerns.\n\nFinal Result: Good' 'Reasoning:\n\n1. Context Grounding: The given answer closely follows the instructions provided in Document 1 regarding car travel expenses. The essential details, such as keeping track of kilometers and submitting an Excel document or email to specified email addresses, are accurately mentioned.\n \n2. Relevance: The answer is pertinent to the question and provides specific steps on how to report car travel expenses, which is exactly what was asked.\n\n3. Conciseness: The response is generally concise, though it includes a minor unnecessary detail about requesting a parking card for a specific date which doesn’t directly pertain to the monthly reporting of car travel expenses.\n\n4. Specificity: The answer effectively covers the key points mentioned in the document for reporting travel expenses, including the specific reimbursement rate.\n\n5. General Tips: The tip about requesting a parking card is accurate but not directly relevant to the primary question of how to report car travel expenses.\n\nGiven the accurate detailing and mostly relevant content, while removing the unnecessary detail about the parking card date would improve the response, the answer is still sufficiently useful.\n\nFinal Result: Good'

Label

Examples

"Reasoning: \n\n1. Context Grounding: The answer pulls loosely from the documents provided but does not ground the response specifically to any of the documents' key points about budgeting or investment decisions.\n2. Relevance: While the query is about saving money in general, the answer ambiguously addresses saving money by suggesting wise spending and seeking guidance, which are indirectly related to what's mentioned in the documents.\n3. Conciseness: The answer contains unnecessary details like mentioning health and priorities, which are somewhat related based on Document 3 but are not directly in line with the exact scope of the question.\n4. Too General: The answer is broad and generic. The document contains specific guidelines (such as asking if the investment would be made with personal money and consulting with ORGANIZATION_2 for high-stakes cases), which the response fails to integrate effectively.\n5. Detail Inclusion: The answer needed to incorporate specific strategies mentioned in the documents, like ensuring if an expense aligns with personal and organizational benefits or seeking guidance on higher-budget items.\n\nFinal result: Bad"
'Reasoning:\n\n1. Context Grounding: The answer does mention several pet peeves directly from the provided documents, though it incorrectly repeats the names and has some formatting errors, it aligns generally with the content.\n2. Relevance: The answer focuses on the specific question about pet peeves, summarizing relevant points from the documents.\n3. Conciseness: The answer is cluttered with repeated names and formatting errors that make it unnecessarily lengthy and harder to read.\n4. Avoiding unfounded responses: The answer is founded on the document but suffers from poor editing issues.\n5. Specificity: The answer includes specific pet peeves, but it also includes unnecessary repetition.\n6. Tip Inclusion: The query did not ask for tips, so this criterion is not strictly applicable.\n\nFinal result: Bad\n\nThe repeated names and formatting errors indicate lack of editing and care in the response, making it difficult to read and comprehend fully. It does not meet the required standard for a good, concise, and clear response.'
"Reasoning:\n\n1. Context Grounding: The answer only indirectly relates to the question. The provided information gives various ways to interact with the company's systems or tools rather than specifically how to access the training resources.\n2. Relevance: The answer deviates from the main question. It mentions systems for personal documents, expense claims, password management, secure sharing, feedback processes, and learning budgets, none of which directly answer how to access training resources.\n3. Conciseness: While the answer is detailed, it includes unnecessary information that isn't directly related to accessing training resources, making it not concise.\n4. Specificity: The answer lacks details on accessing training materials or platforms where these resources are housed.\n5. Generalization: It provides general guidance on several aspects but fails to narrow down specifically on training resources or offer relevant tips directly tied to them.\n\nFinal result: Bad"

'Reasoning:\n\n1. Context Grounding: The answer references various elements from the documents provided, such as giving feedback at the time of the event, focusing on the situation and not the person, and showing appreciation. It generally aligns with the tips described in the documents.\n \n2. Relevance: The answer directly addresses the question of how feedback should be given, touching on many of the specific points mentioned in the documents.\n\n3. Conciseness: The response is overly wordy and includes some repetition (e.g., focusing on the situation not the person and the intention behind giving feedback). It can be more concise while still covering the necessary points.\n\n4. Document Dependency: The response accurately reflects the content of the provided documents. However, it brings in names like "Christopher Estes," "emichelle James Johnson MD," which may be elements of the document's placeholders rather than relevant names in the context of feedback tips. This adds unnecessary confusion to the answer.\n\n5. Specificity: The answer appropriately includes specific advice, though it overdoes some parts (like the role and intentions in giving feedback).\n\n6. Accuracy and Clarity: There are minor grammatical errors and some parts like "emichelle James Johnson MDamples" which are not clear and probably are typos or formatting errors.\n\nGiven the evaluation above:\n\nFinal Result: Bad'
'Reasoning:\n1. Context Grounding: The answer refers to sharing information from high-level meetings which is supported by Document 4. However, the inclusion of "Robin Nichols" appears to be an error or typo, which diminishes the context grounding slightly.\n2. Relevance: The answer is relevant to the question asked; it covers the need for transparency, team alignment, addressing concerns, and fostering an open environment, which is in line with the information in Document 4.\n3. Conciseness: The answer is moderately concise but slightly verbose. It can be trimmed down while still maintaining the key points.\n4. Directness: The answer attempts to respond to the question based on the content provided in Document 4.\n5. Specificity: The answer includes specific points about decisions made and the rationale being shared, aligning the team and fostering a sense of purpose.\n6. Tips: The answer implicitly includes tips such as ensuring transparency and addressing team concerns.\n\nFinal Result: Good'
'Reasoning:\n\n1. Context Grounding: The given answer closely follows the instructions provided in Document 1 regarding car travel expenses. The essential details, such as keeping track of kilometers and submitting an Excel document or email to specified email addresses, are accurately mentioned.\n \n2. Relevance: The answer is pertinent to the question and provides specific steps on how to report car travel expenses, which is exactly what was asked.\n\n3. Conciseness: The response is generally concise, though it includes a minor unnecessary detail about requesting a parking card for a specific date which doesn’t directly pertain to the monthly reporting of car travel expenses.\n\n4. Specificity: The answer effectively covers the key points mentioned in the document for reporting travel expenses, including the specific reimbursement rate.\n\n5. General Tips: The tip about requesting a parking card is accurate but not directly relevant to the primary question of how to report car travel expenses.\n\nGiven the accurate detailing and mostly relevant content, while removing the unnecessary detail about the parking card date would improve the response, the answer is still sufficiently useful.\n\nFinal Result: Good'

Evaluation

Metrics

Label	Accuracy
all	0.6716

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_cot-instructions_only_reasoning_1726750606.384621")
# Run inference
preds = model("Reasoning:
1. Context Grounding: The response draws from the documents providing relevant sources such as the organization's website, job ads, and newsletter link.
2. Relevance: The answer is directly related to the question about understanding the organization's products, challenges, and future.
3. Conciseness: The answer is clear and to the point.
4. Does not attempt to respond when the document lacks information: It addresses the question appropriately with the available information.
5. Specificity: The answer is specific and provides concrete steps to follow.
6. Relevant tips: The answer includes actionable steps like visiting the website, viewing job ads, and signing up for a newsletter, which are relevant.

The answer precisely matches all the criteria set for evaluation.

Final Result: Good")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	87	141.3077	245

Label	Training Sample Count
0	32
1	33

Training Hyperparameters

batch_size: (16, 16)
num_epochs: (5, 5)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (2e-05, 2e-05)
head_learning_rate: 2e-05
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.01
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0061	1	0.2339	-
0.3067	50	0.2693	-
0.6135	100	0.2364	-
0.9202	150	0.0942	-
1.2270	200	0.0031	-
1.5337	250	0.0019	-
1.8405	300	0.0016	-
2.1472	350	0.0016	-
2.4540	400	0.0015	-
2.7607	450	0.0013	-
3.0675	500	0.0013	-
3.3742	550	0.0012	-
3.6810	600	0.0012	-
3.9877	650	0.0012	-
4.2945	700	0.0012	-
4.6012	750	0.0011	-
4.9080	800	0.0011	-

Framework Versions

Python: 3.10.14
SetFit: 1.1.0
Sentence Transformers: 3.1.0
Transformers: 4.44.0
PyTorch: 2.4.1+cu121
Datasets: 2.19.2
Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}

Netta1994
/

setfit_baai_newrelic_gpt-4o_cot-instructions_only_reasoning_1726750606.384621