README.md

metadata

base_model: Snowflake/snowflake-arctic-embed-m
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
  - dot_accuracy@1
  - dot_accuracy@3
  - dot_accuracy@5
  - dot_accuracy@10
  - dot_precision@1
  - dot_precision@3
  - dot_precision@5
  - dot_precision@10
  - dot_recall@1
  - dot_recall@3
  - dot_recall@5
  - dot_recall@10
  - dot_ndcg@10
  - dot_mrr@10
  - dot_map@100
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:522
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
widget:
  - source_sentence: >-
      How did the hiring tool's design contribute to the rejection of women
      applicants?
    sentences:
      - >-
        legal protections. Throughout this framework the term “algorithmic
        discrimination” takes this meaning (and 

        not a technical understanding of discrimination as distinguishing
        between items). 

        AUTOMATED SYSTEM: An "automated system" is any system, software, or
        process that uses computation as 

        whole or part of a system to determine outcomes, make or aid decisions,
        inform policy implementation, collect 

        data or observations, or otherwise interact with individuals and/or
        communities. Automated systems 

        include, but are not limited to, systems derived from machine learning,
        statistics, or other data processing 

        or artificial intelligence techniques, and exclude passive computing
        infrastructure. “Passive computing
      - >-
        communities. 

        • An automated system using nontraditional factors such as educational
        attainment and employment history as

        part of its loan underwriting and pricing model was found to be much
        more likely to charge an applicant whoattended a Historically Black
        College or University (HBCU) higher loan prices for refinancing a
        student loanthan an applicant who did not attend an HBCU. This was found
        to be true even when controlling for

        other credit-related factors.32

        •A hiring tool that learned the features of a company's employees
        (predominantly men) rejected women appli -

        cants for spurious and discriminatory reasons; resumes with the word
        “women’s,” such as “women’s

        chess club captain,” were penalized in the candidate ranking.33
      - >-
        dures before deploying the system, as well as responsibility of specific
        individuals or entities to oversee ongoing assessment and mitigation.
        Organizational stakeholders including those with oversight of the
        business process or operation being automated, as well as other
        organizational divisions that may be affected due to the use of the
        system, should be involved in establishing governance procedures.
        Responsibility should rest high enough in the organization that
        decisions about resources, mitigation, incident response, and potential
        rollback can be made promptly, with sufficient weight given to risk
        mitigation objectives against competing concerns. Those holding this
        responsibility should be made aware of any use cases with the
  - source_sentence: >-
      How are companies using individual profiles based on tracked behavior to
      impact the American public?
    sentences:
      - >-
        requests should be used so that users understand for what use contexts,
        time span, and entities they are providing data and metadata consent.
        User experience research should be performed to ensure these consent
        requests meet performance standards for readability and comprehension.
        This includes ensuring that consent requests are accessible to users
        with disabilities and are available in the language(s) and reading level
        appro

        -

        priate for the audience.  User experience design choices that
        intentionally obfuscate or manipulate user choice (i.e., “dark
        patterns”) should be not be used. 

        34
              DATA PRIVACY 
        WHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS
      - >-
        with more and more companies tracking the behavior of the American
        public, building individual profiles based on this data, and using this
        granular-level information as input into automated systems that further
        track, profile, and impact the American public. Government agencies,
        particularly law enforcement agencies, also use and help develop a
        variety of technologies that enhance and expand surveillance
        capabilities, which similarly collect data used as input into other
        automated systems that directly impact people’s lives. Federal law has
        not grown to address the expanding scale of private data collection, or
        of the ability of governments at all levels to access that data and
        leverage the means of private collection.
      - >-
        ways that threaten the rights of the American public. Too often, these
        tools are used to limit our opportunities and 

        prevent our access to critical resources or services. These problems are
        well documented. In America and around 

        the world, systems supposed to help with patient care have proven
        unsafe, ineffective, or biased. Algorithms used 

        in hiring and credit decisions have been found to reflect and reproduce
        existing unwanted inequities or embed 

        new harmful bias and discrimination. Unchecked social media data
        collection has been used to threaten people’s 

        opportunities, undermine their privac y, or pervasively track their
        activity—often without their knowledge or 

        consent.
  - source_sentence: >-
      What should entities developing technologies related to sensitive data
      regularly report on?
    sentences:
      - >-
        concerns that may limit their effectiveness. The results of assessments
        of the efficacy and potential bias of such human-based systems should be
        overseen by governance structures that have the potential to update the
        operation of the human-based system in order to mitigate these effects. 

        50
              
         HUMAN ALTERNATIVES, 
        CONSIDERATION, AND 

        FALLBACK 

        WHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS

        The expectations for automated systems are meant to serve as a blueprint
        for the development of additional 

        technical standards and practices that are tailored for particular
        sectors and contexts. 

        Implement additional human oversight and safeguards for automated
        systems related to 

        sensitive domains
      - >-
        performance testing including, but not limited to, accuracy,
        differential demographic impact, resulting 

        error rates (overall and per demographic group), and comparisons to
        previously deployed systems; 

        ongoing monitoring procedures and regular performance testing reports,
        including monitoring frequency, 

        results, and actions taken; and the procedures for and results from
        independent evaluations. Reporting 

        should be provided in a plain language and machine-readable manner. 

        20
               
         
         
         
         
         
          SAFE AND EFFECTIVE 
        SYSTEMS 

        HOW THESE PRINCIPLES CAN MOVE INTO PRACTICE

        Real-life examples of how these principles can become reality, through
        laws, policies, and practical
      - >-
        those who are less proximate do not (e.g., a teacher has access to their
        students’ daily progress data while a 

        superintendent does not). 

        Reporting.  In addition to the reporting on data privacy (as listed
        above for non-sensitive data), entities devel-

        oping technologies related to a sensitive domain and those collecting,
        using, storing, or sharing sensitive data 

        should, whenever appropriate, regularly provide public reports
        describing: any data security lapses or breaches 

        that resulted in sensitive data leaks; the numbe r, type, and outcomes
        of ethical pre-reviews undertaken; a 

        description of any data sold, shared, or made public, and how that data
        was assessed to determine it did not pres-
  - source_sentence: >-
      What are the expectations for automated systems intended to serve as a
      blueprint for?
    sentences:
      - >-
        Clear organizational oversight. Entities responsible for the development
        or use of automated systems should lay out clear governance structures
        and procedures.  This includes clearly-stated governance proce

        -
      - >-
        critical resources or services. These rights, opportunities, and access
        to critical resources of services should 

        be enjoyed equally and be fully protected, regardless of the changing
        role that automated systems may play in 

        our lives. 

        This framework describes protections that should be applied with respect
        to all automated systems that 

        have the potential to meaningfully impact individuals' or communities'
        exercise of: 

        RIGHTS, OPPORTUNITIES, OR ACCESS

        Civil rights, civil liberties, and privacy, including freedom of speech,
        voting, and protections from discrimi -

        nation, excessive punishment, unlawful surveillance, and violations of
        privacy and other freedoms in both 

        public and private sector contexts;
      - >-
        19
               
         
          SAFE AND EFFECTIVE 
        SYSTEMS 

        WHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS

        The expectations for automated systems are meant to serve as a blueprint
        for the development of additional 

        technical standards and practices that are tailored for particular
        sectors and contexts. 

        Derived data sources tracked and reviewed carefully. Data that is
        derived from other data through 

        the use of algorithms, such as data derived or inferred from prior model
        outputs, should be identified and tracked, e.g., via a specialized type
        in a data schema. Derived data should be viewed as potentially high-risk
        inputs that may lead to feedback loops, compounded harm, or inaccurate
        results. Such sources should be care

        -
  - source_sentence: >-
      What types of systems are considered time-critical according to the
      context?
    sentences:
      - >-
        Equity includes a commitment from the agencies that oversee mortgage
        lending to include a 

        nondiscrimination standard in the proposed rules for Automated Valuation
        Models.52

        The Equal Employment Opportunity Commission and the Department of
        Justice have clearly 

        laid out how employers’ use of AI and other automated systems can result
        in discrimination 

        against job applicants and employees with disabilities.53 The documents
        explain 

        how employers’ use of software that relies on algorithmic
        decision-making may violate existing requirements 

        under Title I of the Americans with Disabilities Act (“ADA”). This
        technical assistance also provides practical
      - >-
        Discrimination 

        Protections  
              
         WHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS
        The expectations for automated systems are meant to serve as a blueprint
        for the development of additional 

        technical standards and practices that are tailored for particular
        sectors and contexts. 

        Demonstrate that the system protects against algorithmic discrimination 

        Independent evaluation. As described in the section on Safe and
        Effective Systems, entities should allow 

        independent evaluation of potential algorithmic discrimination caused by
        automated systems they use or
      - >-
        where possible, available before the harm occurs. Time-critical systems
        include, but are not limited to, 

        voting-related systems, automated building access and other access
        systems, systems that form a critical 

        component of healthcare, and systems that have the ability to withhold
        wages or otherwise cause 

        immediate financial penalties. 

        Effective. The organizational structure surrounding processes for
        consideration and fallback should 

        be designed so that if the human decision-maker charged with reassessing
        a decision determines that it 

        should be overruled, the new decision will be effectively enacted. This
        includes ensuring that the new
model-index:
  - name: SentenceTransformer based on Snowflake/snowflake-arctic-embed-m
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: Unknown
          type: unknown
        metrics:
          - type: cosine_accuracy@1
            value: 0.8448275862068966
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.9482758620689655
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9770114942528736
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9942528735632183
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.8448275862068966
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.3160919540229885
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19540229885057464
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09942528735632182
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8448275862068966
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.9482758620689655
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9770114942528736
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9942528735632183
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.924865695917767
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.901963601532567
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.9021617783062492
            name: Cosine Map@100
          - type: dot_accuracy@1
            value: 0.8448275862068966
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.9482758620689655
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.9770114942528736
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.9942528735632183
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.8448275862068966
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.3160919540229885
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.19540229885057464
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.09942528735632182
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.8448275862068966
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.9482758620689655
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.9770114942528736
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.9942528735632183
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.924865695917767
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.901963601532567
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.9021617783062492
            name: Dot Map@100

SentenceTransformer based on Snowflake/snowflake-arctic-embed-m

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-m
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'What types of systems are considered time-critical according to the context?',
    'where possible, available before the harm occurs. Time-critical systems include, but are not limited to, \nvoting-related systems, automated building access and other access systems, systems that form a critical \ncomponent of healthcare, and systems that have the ability to withhold wages or otherwise cause \nimmediate financial penalties. \nEffective. The organizational structure surrounding processes for consideration and fallback should \nbe designed so that if the human decision-maker charged with reassessing a decision determines that it \nshould be overruled, the new decision will be effectively enacted. This includes ensuring that the new',
    'Discrimination \nProtections  \n      \n WHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nDemonstrate that the system protects against algorithmic discrimination \nIndependent evaluation. As described in the section on Safe and Effective Systems, entities should allow \nindependent evaluation of potential algorithmic discrimination caused by automated systems they use or',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8448
cosine_accuracy@3	0.9483
cosine_accuracy@5	0.977
cosine_accuracy@10	0.9943
cosine_precision@1	0.8448
cosine_precision@3	0.3161
cosine_precision@5	0.1954
cosine_precision@10	0.0994
cosine_recall@1	0.8448
cosine_recall@3	0.9483
cosine_recall@5	0.977
cosine_recall@10	0.9943
cosine_ndcg@10	0.9249
cosine_mrr@10	0.902
cosine_map@100	0.9022
dot_accuracy@1	0.8448
dot_accuracy@3	0.9483
dot_accuracy@5	0.977
dot_accuracy@10	0.9943
dot_precision@1	0.8448
dot_precision@3	0.3161
dot_precision@5	0.1954
dot_precision@10	0.0994
dot_recall@1	0.8448
dot_recall@3	0.9483
dot_recall@5	0.977
dot_recall@10	0.9943
dot_ndcg@10	0.9249
dot_mrr@10	0.902
dot_map@100	0.9022

Training Details

Training Dataset

Unnamed Dataset

• Size: 522 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 522 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 11 tokens mean: 19.05 tokens max: 35 tokens	min: 10 tokens mean: 116.38 tokens max: 161 tokens

• Samples:

  sentence_0	sentence_1
  What is the purpose of the AI Bill of Rights mentioned in the context?	BLUEPRINT FOR AN AI B ILL OF RIGHTS MAKING AUTOMATED SYSTEMS WORK FOR THE AMERICAN PEOPLE OCTOBER 2022
  When was the Blueprint for an AI Bill of Rights published?	BLUEPRINT FOR AN AI B ILL OF RIGHTS MAKING AUTOMATED SYSTEMS WORK FOR THE AMERICAN PEOPLE OCTOBER 2022
  What is the purpose of the Blueprint for an AI Bill of Rights published by the White House Office of Science and Technology Policy?	About this Document The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People was published by the White House Office of Science and Technology Policy in October 2022. This framework was released one year after OSTP announced the launch of a process to develop “a bill of rights for an AI-powered world.” Its release follows a year of public engagement to inform this initiative. The framework is available online at: https://www.whitehouse.gov/ostp/ai-bill-of-rights About the Office of Science and Technology Policy The Office of Science and Technology Policy (OSTP) was established by the National Science and Technology

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 20
• per_device_eval_batch_size: 20
• num_train_epochs: 5
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 20
• per_device_eval_batch_size: 20
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	cosine_map@100
1.0	27	0.8792
1.8519	50	0.8950
2.0	54	0.9011
3.0	81	0.9022

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.1.1
• Transformers: 4.44.2
• PyTorch: 2.4.1+cu121
• Accelerate: 0.34.2
• Datasets: 2.19.2
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}