policy_gte_large_2 / README.md
lw2134's picture
Add new SentenceTransformer model.
9006307 verified
metadata
base_model: Alibaba-NLP/gte-large-en-v1.5
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
  - dot_accuracy@1
  - dot_accuracy@3
  - dot_accuracy@5
  - dot_accuracy@10
  - dot_precision@1
  - dot_precision@3
  - dot_precision@5
  - dot_precision@10
  - dot_recall@1
  - dot_recall@3
  - dot_recall@5
  - dot_recall@10
  - dot_ndcg@10
  - dot_mrr@10
  - dot_map@100
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:224
  - loss:MultipleNegativesRankingLoss
widget:
  - source_sentence: >-
      What are some of the mental health impacts associated with the increased
      use of surveillance technologies in schools and workplaces, as mentioned
      in the context information?
    sentences:
      - >-
        15 GV-1.3-004 Obtain input from stakeholder communities to identify
        unacceptable use , in 

        accordance with activities in the AI RMF Map function . CBRN Information
        or Capabilities ; 

        Obscene, Degrading, and/or 

        Abusive Content ; Harmful Bias 

        and Homogenization ; Dangerous, 

        Violent, or Hateful Content  

        GV-1.3-005 Maintain an updated hierarch y of identified and expected GAI
        risks connected to 

        contexts of GAI model advancement and use, potentially including
        specialized risk 

        levels for GAI systems that address issues such as model collapse and
        algorithmic 

        monoculture.  Harmful Bias and Homogenization  

        GV-1.3-006 Reevaluate organizational risk tolerances to account for
        unacceptable negative risk 

        (such as where significant negative impacts are imminent, severe harms
        are actually occurring, or large -scale risks could occur); and broad
        GAI negative risks, 

        including: Immature safety or risk cultures related to AI and GAI
        design, development and deployment, public information integrity risks,
        including impacts on democratic processes, unknown long -term
        performance characteristics of GAI.  Information Integrity ; Dangerous
        , 

        Violent, or Hateful Content ; CBRN 

        Information or Capabilities  

        GV-1.3-007 Devise a plan to halt development or deployment of a GAI
        system that poses unacceptable negative risk.  CBRN Information and
        Capability ; 

        Information Security ; Information 

        Integrity  

        AI Actor Tasks: Governance and Oversight  
         
        GOVERN 1.4: The risk management process and its outcomes are established
        through transparent policies, procedures, and other 

        controls based on organizational risk priorities.  

        Action ID  Suggested Action  GAI Risks  

        GV-1.4-001 Establish policies and mechanisms to prevent GAI systems from
        generating 

        CSAM, NCII or content that violates the law.   Obscene, Degrading,
        and/or 

        Abusive Content ; Harmful Bias 

        and Homogenization ; 

        Dangerous, Violent, or Hateful Content
         
        GV-1.4-002 Establish transparent acceptable use policies for GAI that
        address illegal use or 

        applications of GAI.  CBRN Information or 

        Capabilities ; Obscene, 

        Degrading, and/or Abusive Content ; Data Privacy ; Civil 

        Rights violations
         
        AI Actor Tasks: AI Development, AI Deployment, Governance and Oversight
      - >-
        DATA PRIVACY 

        WHY THIS PRINCIPLE IS IMPORTANT

        This section provides a brief summary of the problems which the
        principle seeks to address and protect 

        against, including illustrative examples. 

        Data privacy is a foundational and cross-cutting principle required for
        achieving all others in this framework. Surveil -

        lance and data collection, sharing, use, and reuse now sit at the
        foundation of business models across many industries, 

        with more and more companies tracking the behavior of the American
        public, building individual profiles based on this data, and using this
        granular-level information as input into automated systems that further
        track, profile, and impact the American public. Government agencies,
        particularly law enforcement agencies, also use and help develop a
        variety of technologies that enhance and expand surveillance
        capabilities, which similarly collect data used as input into other
        automated systems that directly impact people’s lives. Federal law has
        not grown to address the expanding scale of private data collection, or
        of the ability of governments at all levels to access that data and
        leverage the means of private collection.  

        Meanwhile, members of the American public are often unable to access
        their personal data or make critical decisions about its collection and
        use. Data brokers frequently collect consumer data from numerous sources
        without consumers’ permission or 

        knowledge.60 Moreover, there is a risk that inaccurate and faulty data
        can be used to 

        make decisions about their lives, such as whether they will qualify for
        a loan or get a job. Use of surveillance 

        technologies has increased in schools and workplaces, and, when coupled
        with consequential management and 

        evaluation decisions, it is leading to mental health harms such as
        lowered self-confidence, anxiet y, depression, and 

        a reduced ability to use analytical reasoning.61 Documented patterns
        show that personal data is being aggregated by 

        data brokers to profile communities in harmful ways.62 The impact of all
        this data harvesting is corrosive, 

        breeding distrust, anxiety, and other mental health problems; chilling
        speech, protest, and worker organizing; and 

        threatening our democratic process.63 The American public should be
        protected from these growing risks. 

        Increasingl y, some companies are taking these concerns seriously and
        integrating mechanisms to protect consumer 

        privacy into their products by design and by default, including by
        minimizing the data they collect, communicating collection and use
        clearl y, and improving security practices. Federal government
        surveillance and other collection and 

        use of data is governed by legal protections that help to protect civil
        liberties and provide for limits on data retention in some cases. Many
        states have also enacted consumer data privacy protection regimes to
        address some of these harms. 

        Howeve r, these are not yet standard practices, and the United States
        lacks a comprehensive statutory or regulatory 

        framework governing the rights of the public when it comes to personal
        data. While a patchwork of laws exists to guide the collection and use
        of personal data in specific contexts, including health, employment,
        education, and credit, it can be unclear how these laws apply in other
        contexts and in an increasingly automated societ y. Additional protec

        -

        tions would assure the American public that the automated systems they
        use are not monitoring their activities, collecting information on their
        lives, or otherwise surveilling them without context-specific consent or
        legal authori

        -

        ty. 

        31
      - >-
        Applying The Blueprint for an AI Bill of Rights 

        SENSITIVE DATA: Data and metadata are sensitive if they pertain to an
        individual in a sensitive domain 

        (defined below); are generated by technologies used in a sensitive
        domain; can be used to infer data from a 

        sensitive domain or sensitive data about an individual (such as
        disability-related data, genomic data, biometric data, behavioral data,
        geolocation data, data related to interaction with the criminal justice
        system, relationship history and legal status such as custody and
        divorce information, and home, work, or school environmental data); or
        have the reasonable potential to be used in ways that are likely to
        expose individuals to meaningful harm, such as a loss of privacy or
        financial harm due to identity theft. Data and metadata generated by or
        about those who are not yet legal adults is also sensitive, even if not
        related to a sensitive domain. Such data includes, but is not limited
        to, numerical, text, image, audio, or video data. 

        SENSITIVE DOMAINS: “Sensitive domains” are those in which activities
        being conducted can cause material 

        harms, including significant adverse effects on human rights such as
        autonomy and dignit y, as well as civil liber-

        ties and civil rights. Domains that have historically been singled out
        as deserving of enhanced data protections 

        or where such enhanced protections are reasonably expected by the public
        include, but are not limited to, health, family planning and care,
        employment, education, criminal justice, and personal finance. In the
        context of this framework, such domains are considered sensitive whether
        or not the specifics of a system context would necessitate coverage
        under existing la w, and domains and data that are considered sensitive
        are under-

        stood to change over time based on societal norms and context. 

        SURVEILLANCE TECHNOLOGY : “Surveillance technology” refers to products
        or services marketed for 

        or that can be lawfully used to detect, monitor, intercept, collect,
        exploit, preserve, protect, transmit, and/or 

        retain data, identifying information, or communications concerning
        individuals or groups. This framework 

        limits its focus to both government and commercial use of surveillance
        technologies when juxtaposed with 

        real-time or subsequent automated analysis and when such systems have a
        potential for meaningful impact 

        on individuals’ or communities’ rights, opportunities, or access.
        UNDERSERVED COMMUNITIES: The term “underserved communities” refers to
        communities that have 

        been systematically denied a full opportunity to participate in aspects
        of economic, social, and civic life, as 

        exemplified by the list in the preceding definition of “equit y.” 

        11
  - source_sentence: >-
      Discuss the implications of automatic signature verification software on
      voter disenfranchisement in the United States, as highlighted in the
      article by Kyle Wiggers. What are the potential risks associated with this
      technology?
    sentences:
      - >-
        ENDNOTES

        96. National Science Foundation. NSF Program on Fairness in Artificial
        Intelligence in Collaboration

        with Amazon (FAI). Accessed July 20, 2022.

        https://www.nsf.gov/pubs/2021/nsf21585/nsf21585.htm

        97. Kyle Wiggers. Automatic signature verification software threatens to
        disenfranchise U.S. voters.

        VentureBeat. Oct. 25, 2020.

        https://venturebeat.com/2020/10/25/automatic-signature-verification-software-threatens-to-disenfranchise-u-s-voters/

        98. Ballotpedia. Cure period for absentee and mail-in ballots. Article
        retrieved Apr 18, 2022.

        https://ballotpedia.org/Cure_period_for_absentee_and_mail-in_ballots

        99. Larry Buchanan and Alicia Parlapiano. Two of these Mail Ballot
        Signatures are by the Same Person.

        Which Ones? New York Times. Oct. 7, 2020.

        https://www.nytimes.com/interactive/2020/10/07/upshot/mail-voting-ballots-signature-

        matching.html

        100. Rachel Orey and Owen Bacskai. The Low Down on Ballot Curing. Nov.
        04, 2020.

        https://bipartisanpolicy.org/blog/the-low-down-on-ballot-curing/101.
        Andrew Kenney. 'I'm shocked that they need to have a smartphone': System
        for unemployment

        benefits exposes digital divide. USA Today. May 2, 2021.

        https://www.usatoday.com/story/tech/news/2021/05/02/unemployment-benefits-system-leaving-

        people-behind/4915248001/

        102. Allie Gross. UIA lawsuit shows how the state criminalizes the
        unemployed . Detroit Metro-Times.

        Sep. 18, 2015.

        https://www.metrotimes.com/news/uia-lawsuit-shows-how-the-state-criminalizes-the-unemployed-2369412

        103. Maia Szalavitz. The Pain Was Unbearable. So Why Did Doctors Turn
        Her Away? Wired. Aug. 11,

        2021.
        https://www.wired.com/story/opioid-drug-addiction-algorithm-chronic-pain/

        104. Spencer Soper. Fired by Bot at Amazon: "It's You Against the
        Machine" . Bloomberg, Jun. 28, 2021.

        https://www.bloomberg.com/news/features/2021-06-28/fired-by-bot-amazon-turns-to-machine-

        managers-and-workers-are-losing-out

        105. Definitions of ‘equity’ and ‘underserved communities’ can be found
        in the Definitions section of

        this document as well as in Executive Order on Advancing Racial Equity
        and Support for Underserved

        Communities Through the Federal
        Government:https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/20/executive-order-advancing-racial-equity-and-support-for-underserved-communities-through-the-federal-government/

        106. HealthCare.gov. Navigator - HealthCare.gov Glossary. Accessed May
        2, 2022.

        https://www.healthcare.gov/glossary/navigator/

        72
      - >-
        SAFE AND EFFECTIVE 

        SYSTEMS 

        WHY THIS PRINCIPLE IS IMPORTANT

        This section provides a brief summary of the problems which the
        principle seeks to address and protect 

        against, including illustrative examples. 

         AI-enabled “nudification” technology that creates images where people
        appear to be nude—including apps that

        enable non-technical users to create or alter images of individuals
        without their consent—has proliferated at an

        alarming rate. Such technology is becoming a common form of image-based
        abuse that disproportionately

        impacts women. As these tools become more sophisticated, they are
        producing altered images that are increasing -

        ly realistic and are difficult for both humans and AI to detect as
        inauthentic. Regardless of authenticit y, the expe -

        rience of harm to victims of non-consensual intimate images can be
        devastatingly real—affecting their personal

        and professional lives, and impacting their mental and physical
        health.10

         A company installed AI-powered cameras in its delivery vans in order
        to evaluate the road safety habits of its driv -

        ers, but the system incorrectly penalized drivers when other cars cut
        them off or when other events beyond

        their control took place on the road. As a result, drivers were
        incorrectly ineligible to receive a bonus.11

        17
      - >-
        NOTICE & 

        EXPLANATION 

        WHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS

        The expectations for automated systems are meant to serve as a blueprint
        for the development of additional 

        technical standards and practices that are tailored for particular
        sectors and contexts. 

        Tailored to the level of risk. An assessment should be done to determine
        the level of risk of the auto -

        mated system. In settings where the consequences are high as determined
        by a risk assessment, or extensive 

        oversight is expected (e.g., in criminal justice or some public sector
        settings), explanatory mechanisms should be built into the system design
        so that the system’s full behavior can be explained in advance (i.e.,
        only fully transparent models should be used), rather than as an
        after-the-decision interpretation. In other settings, the extent of
        explanation provided should be tailored to the risk level. 

        Valid. The explanation provided by a system should accurately reflect
        the factors and the influences that led 

        to a particular decision, and should be meaningful for the particular
        customization based on purpose, target, and level of risk. While
        approximation and simplification may be necessary for the system to
        succeed based on the explanatory purpose and target of the explanation,
        or to account for the risk of fraud or other concerns related to
        revealing decision-making information, such simplifications should be
        done in a scientifically supportable way. Where appropriate based on the
        explanatory system, error ranges for the explanation should be
        calculated and included in the explanation, with the choice of
        presentation of such information balanced with usability and overall
        interface complexity concerns. 

        Demonstrate protections for notice and explanation 

        Reporting. Summary reporting should document the determinations made
        based on the above consider -

        ations, including: the responsible entities for accountability purposes;
        the goal and use cases for the system, identified users, and impacted
        populations; the assessment of notice clarity and timeliness; the
        assessment of the explanation's validity and accessibility; the
        assessment of the level of risk; and the account and assessment of how
        explanations are tailored, including to the purpose, the recipient of
        the explanation, and the level of risk. Individualized profile
        information should be made readily available to the greatest extent
        possible that includes explanations for any system impacts or
        inferences. Reporting should be provided in a clear plain language and
        machine-readable manner. 

        44
  - source_sentence: >-
      How does the document aim to bridge the gap between theoretical principles
      and practical applications in the context of AI rights?
    sentences:
      - |-
        FROM 
        PRINCIPLES 
        TO PRACTICE 
        A T ECHINCAL COMPANION TO
        THE Blueprint for an 
        AI B ILL OF RIGHTS
        12
      - >-
        3 the abuse, misuse, and unsafe repurposing by humans (adversarial or
        not ), and  others result 

        from interactions between a human and an AI system.   

         Time  scale: GAI risks  may materialize  abruptly or  across extended
        periods . Example s include 

        immediate  (and/or prolonged) emotional harm  and potential risks to
        physical safety  due to the 

        distribution of harmful deepfake images , or the lo ng-term effect of
        disinformation on soci etal 

        trust in public  institutions . 

        The presence of risks  and where they fall along the dimensions above
        will vary depending on the 

        characteristics of the GAI model , system, or use case at hand. These
        characteristics include but are not 

        limited to GAI model or system architecture, training mechanisms  and
        libraries , data types  used for 

        training or fine -tuning , levels of model access or availability of
        model weights,  and application or use 

        case context. 

        Organizations may choose to tailor how they measure  GAI risks  based
        on  these characteristics . They may 

        additionally wish to  allocate risk management resources relative to the
        severity and likelihood of 

        negative impact s, including where and how these risks manifest , and
        their direct and material impacts  

        harms in the context of GAI use. Mitigations for model or system  level
        risks may differ from mitigations  

        for use-case or ecosystem level risks.  

        Importantly, some GAI risks are un known , and are therefore difficult to
        properly scope or evaluate given  

        the uncertaint y about potential GAI scale, complexity, and
        capabilities. Other risks may be known but  

        difficult to estimate  given the wide range of GAI stakeholders, uses,
        inputs, and outputs . Challenges  with 

        risk estimation are aggravated by a lack of visibility into GAI training
        data, and the generally immature 

        state of the science of AI measurement and safety  today . This document
        focuses on risks for which there 

        is an existing empirical evidence base at the time this profile was
        written ; for example,  speculative risks 

        that may potentially arise in more advanced, future GAI systems are not
        considered . Future updates may 

        incorporate additional risks or provide further details on the risks
        identified below.  

        To guide organizations in identifying and managing GAI risks, a set of
        risks unique to or exacerbated by 

        the development and use of GAI are defined below.5 Each risk is  labeled
        according to  the outcome , 

        object,  or source of the risk  (i.e., some are risks “to  a subject 
        or domain  and others are  risks  “of” or 

        “from” an issue or  theme ). These risks provide a lens  through which
        organizations can frame and execute 

        risk management efforts.  To help streamline risk management efforts, each
        risk is mapped in Section 3  

        (as well as in tables in Appendix B)  to relevant Trustworthy AI
        Characteristics  identified in the AI RMF .  
         
         
        5 These risks can be further categorized by organizations depending on
        their unique approaches to risk definition 

        and management. One possible way to further categorize these risks,
        derived in part from the UK’s International 

        Scientific Report on the Safety of Advanced AI , could be: 1 ) Technical
        / Model risks (or risk from malfunction): 

        Confabulation; Dangerous or Violent Recommendations; Data Privacy; Value
        Chain and Component Integration; 

        Harmful Bias, and Homogenization ; 2) Misuse by humans (or malicious
        use):  CBRN Information  or Capabilities ; 

        Data Privacy; Human -AI Configuration; Obscene, Degrading, and/or Abusive
        Content; Information Integrity;  

        Information Security; 3) Ecosystem / societal risks (or systemic risks)
        : Data Privacy; Environmental; Intellectual 

        Property . We also note that some risks are cross -cutting between these
        categories.
      - >-
        5 operations , or other cyberattacks ; increas ed attack surface  for
        targeted cyberattacks , which may 

        compromise a system’s availability or the confidentiality or integrity of
        training data, code, or 

        model weights.   

        10. Intellectual Property:  Eased production or replication  of alleged 
        copyrighted, trademarked, or 

        licensed content without authorization  (possibly in situations which do
        not fall under fair use ); 

        eased exposure of trade secrets;  or plagiari sm or illegal replication
        .  

        11. Obscen e, Degrading, and/or A busive Content : Eased production of
        and access to obscene , 

        degrading, and/or abusive  imagery  which can cause harm , including
        synthetic child sexual abuse 

        material (CSAM) , and nonconsensual intimate images (NCII) of adults . 

        12. Value Chain and Component Integration : Non-transparent or
        untraceable  integration of 

        upstream third- party components, including data that has been
        improperly obtained or not 

        processed and cleaned due to increased automation from GAI; improper
        supplier vetting across 

        the AI lifecycle ; or other issues that diminish transparency or
        accountability for downstream 

        users.  

        2.1. CBRN Information or Capabilities  

        In the future, GAI may enable malicious actors to more easily  access
        CBRN  weapons and/or relevant 

        knowledge, information , materials, tools, or technologies that could be
        misused  to assist in the design, 

        development, production, or use of CBRN weapons or other dangerous
        materials  or agents . While 

        relevant biological and chemical threat knowledge and information is
        often  publicly accessible , LLMs  

        could facilitate its analysis or synthesis , particularly by
        individuals  without formal scientific training  or 

        expertise.  

        Recent research on this topic found that LLM outputs regarding
        biological threat creation  and attack 

        planning  pr ovided minima l assistance  beyond traditional search
        engine queries, suggesting that state -of-

        the-art LLMs at the time these studies were conducted do not
        substantially  increase the operational 

        likelihood of such an attack.  The physical synthesis development,
        production, and use of chemical or 

        biological agents will continue to require both applicable expertise and
        supporting materials and 

        infrastructure . The impact of GAI on chemical or biological agent
        misuse will depend on  what the  key 

        barriers  for malicious actors  are (e.g., whether information access is
        one such barrier ), and how well GAI 

        can help actors  address those barriers .  

        Furthermore , chemical and biological design tools  (BDTs)   highly
        specialized AI systems  trained on 

        scientific data that aid in chemical and biological design  may augment 
        design capabilities in chemistry 

        and biology beyond what text -based LLMs are able to provide . As these
        models become more 

        efficacious , including for beneficial uses, it will be important to assess
        their potential to be used for 

        harm, such as the ideation and design of novel harmful chemical or
        biological agents .  

        While some of these described capabilities  lie beyond the reach  of
        existing GAI tools, ongoing 

        assessments of this risk would be enhanced by monitoring both  the
        ability of  AI tools  to facilitate CBRN 

        weapons planning and GAI systems’  connection or access to relevant data
        and tools . 

        Trustworthy AI Characteristic : Safe , Explainable and Interpretable
  - source_sentence: >-
      What are the key components that should be included in the ongoing
      monitoring procedures for automated systems to ensure their performance
      remains acceptable over time?
    sentences:
      - >-
        AI B ILL OF RIGHTS

        FFECTIVE  SYSTEMS

        ineffective systems. Automated systems should be 

        communities, stakeholders, and domain experts to identify 

        Systems should undergo pre-deployment testing, risk 

        that demonstrate they are safe and effective based on 

        including those beyond the intended use, and adherence to 

        protective measures should include the possibility of not 

        Automated systems should not be designed with an intent 

        reasonably foreseeable possibility of endangering your safety or the
        safety of your communit y. They should 

        stemming from unintended, yet foreseeable, uses or 
           
         
          
         
         SECTION  TITLE
        BLUEPRINT FOR AN

        SAFE AND E 

        You should be protected from unsafe or 

        developed with consultation from diverse 

        concerns, risks, and potential impacts of the system. 

        identification and mitigation, and ongoing monitoring 

        their intended use, mitigation of unsafe outcomes 

        domain-specific standards. Outcomes of these 

        deploying the system or removing a system from use. 

        or 

        be designed to proactively protect you from harms 

        impacts of automated systems. You should be protected from inappropriate
        or irrelevant data use in the 

        design, development, and deployment of automated systems, and from the
        compounded harm of its reuse. 

        Independent evaluation and reporting that confirms that the system is
        safe and effective, including reporting of 

        steps taken to mitigate potential harms, should be performed and the
        results made public whenever possible. 

        ALGORITHMIC DISCRIMINATION P ROTECTIONS

        You should not face discrimination by algorithms and systems should be
        used and designed in 

        an equitable way.  Algorithmic  discrimination occurs when automated
        systems contribute to unjustified 

        different treatment or impacts disfavoring people based on their race,
        color, ethnicity, sex (including 

        pregnancy, childbirth, and related medical conditions, gender identity,
        intersex status, and sexual 

        orientation), religion, age, national origin, disability, veteran
        status, genetic information, or any other 

        classification protected by law. Depending on the specific
        circumstances, such algorithmic  discrimination 

        may violate legal protections. Designers, developers, and deployers of
        automated systems should take 

        proactive and continuous measures to protect individuals and communities
        from algorithmic 

        discrimination and to use and design systems in an equitable way. This
        protection should include proactive 

        equity assessments as part of the system design, use of representative
        data and protection against proxies 

        for demographic features, ensuring accessibility for people with
        disabilities in design and development, 

        pre-deployment and ongoing disparity testing and mitigation, and clear
        organizational oversight. Independent 

        evaluation and plain language reporting in the form of an algorithmic
        impact assessment, including 

        disparity testing results and mitigation information, should be
        performed and made public whenever 

        possible to confirm these protections. 

        5
      - >-
        DATA PRIVACY 

        WHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS

        The expectations for automated systems are meant to serve as a blueprint
        for the development of additional 

        technical standards and practices that are tailored for particular
        sectors and contexts. 

        In addition to the privacy expectations above for general non-sensitive
        data, any system collecting, using, shar-

        ing, or storing sensitive data should meet the expectations belo w.
        Depending on the technological use case and 

        based on an ethical assessment, consent for sensitive data may need to
        be acquired from a guardian and/or child. 

        Provide enhanced protections for data related to sensitive domains 

        Necessar y function s only . Sensitive data should only be used for
        functions strictly necessary for that 

        domain or for functions that are required for administrative reasons
        (e.g., school attendance records), unless 

        consent is acquired, if appropriate, and the additional expectations in
        this section are met. Consent for non-

        necessary functions should be optional, i.e., should not be required,
        incentivized, or coerced in order to 

        receive opportunities or access to services. In cases where data is
        provided to an entity (e.g., health insurance 

        company) in order to facilitate payment for such a need, that data
        should only be used for that purpose. 

        Ethical review and use prohibitions. Any use of sensitive data or
        decision process based in part on sensi-

        tive data that might limit rights, opportunities, or access, whether the
        decision is automated or not, should go 

        through a thorough ethical review and monitoring, both in advance and by
        periodic review (e.g., via an indepen-

        dent ethics committee or similarly robust process). In some cases, this
        ethical review may determine that data 

        should not be used or shared for specific uses even with consent. Some
        novel uses of automated systems in this 

        context, where the algorithm is dynamically developing and where the
        science behind the use case is not well 

        established, may also count as human subject experimentation, and
        require special review under organizational 

        compliance bodies applying medical, scientific, and academic human
        subject experimentation ethics rules and 

        governance procedures. 

        Data  quality. In sensitive domains, entities should be especially
        careful to maintain the quality of data to 

        avoid adverse consequences arising from decision-making based on flawed
        or inaccurate data. Such care is 

        necessary in a fragmented, complex data ecosystem and for datasets that
        have limited access such as for fraud 

        prevention and law enforcement. It should be not left solely to
        individuals to carry the burden of reviewing and 

        correcting data. Entities should conduct regula r, independent audits
        and take prompt corrective measures to 

        maintain accurate, timel y, and complete data. 

        Limit access  to sensitive  data  and derived data. Sensitive data and
        derived data should not be sold, 

        shared, or made public as part of data brokerage or other agreements.
        Sensitive data includes data that can be 

        used to infer sensitive information; even systems that are not directly
        marketed as sensitive domain technologies 

        are expected to keep sensitive data private. Access to such data should
        be limited based on necessity and based 

        on a principle of local control, such that those individuals closest to
        the data subject have more access while 

        those who are less proximate do not (e.g., a teacher has access to their
        students’ daily progress data while a 

        superintendent does not). 

        Reporting.  In addition to the reporting on data privacy (as listed
        above for non-sensitive data), entities devel-

        oping technologies related to a sensitive domain and those collecting,
        using, storing, or sharing sensitive data 

        should, whenever appropriate, regularly provide public reports
        describing: any data security lapses or breaches 

        that resulted in sensitive data leaks; the numbe r, type, and outcomes
        of ethical pre-reviews undertaken; a 

        description of any data sold, shared, or made public, and how that data
        was assessed to determine it did not pres-

        ent a sensitive data risk; and ongoing risk identification and
        management procedures, and any mitigation added 

        based on these procedures. Reporting should be provided in a clear and
        machine-readable manne r. 

        38
      - >-
        SAFE AND EFFECTIVE 

        SYSTEMS 

        WHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS

        The expectations for automated systems are meant to serve as a blueprint
        for the development of additional 

        technical standards and practices that are tailored for particular
        sectors and contexts. 

        Ongoing monitoring. Automated systems should have ongoing monitoring
        procedures, including recalibra -

        tion procedures, in place to ensure that their performance does not fall
        below an acceptable level over time, 

        based on changing real-world conditions or deployment contexts,
        post-deployment modification, or unexpect -

        ed conditions. This ongoing monitoring should include continuous
        evaluation of performance metrics and harm assessments, updates of any
        systems, and retraining of any machine learning models as necessary, as
        well as ensuring that fallback mechanisms are in place to allow
        reversion to a previously working system. Monitor

        -

        ing should take into account the performance of both technical system
        components (the algorithm as well as any hardware components, data
        inputs, etc.) and human operators. It should include mechanisms for
        testing the actual accuracy of any predictions or recommendations
        generated by a system, not just a human operator’s determination of
        their accuracy. Ongoing monitoring procedures should include manual,
        human-led monitor

        -

        ing as a check in the event there are shortcomings in automated
        monitoring systems. These monitoring proce -

        dures should be in place for the lifespan of the deployed automated
        system. 

        Clear organizational oversight. Entities responsible for the development
        or use of automated systems should lay out clear governance structures
        and procedures.  This includes clearly-stated governance proce

        -

        dures before deploying the system, as well as responsibility of specific
        individuals or entities to oversee ongoing assessment and mitigation.
        Organizational stakeholders including those with oversight of the
        business process or operation being automated, as well as other
        organizational divisions that may be affected due to the use of the
        system, should be involved in establishing governance procedures.
        Responsibility should rest high enough in the organization that
        decisions about resources, mitigation, incident response, and potential
        rollback can be made promptly, with sufficient weight given to risk
        mitigation objectives against competing concerns. Those holding this
        responsibility should be made aware of any use cases with the potential
        for meaningful impact on people’s rights, opportunities, or access as
        determined based on risk identification procedures.  In some cases, it
        may be appropriate for an independent ethics review to be conducted
        before deployment. 

        Avoid inappropriate, low-quality, or irrelevant data use and the
        compounded harm of its reuse 

        Relevant and high-quality data. Data used as part of any automated
        system’s creation, evaluation, or 

        deployment should be relevant, of high quality, and tailored to the task
        at hand. Relevancy should be 

        established based on research-backed demonstration of the causal
        influence of the data to the specific use case 

        or justified more generally based on a reasonable expectation of
        usefulness in the domain and/or for the 

        system design or ongoing development. Relevance of data should not be
        established solely by appealing to 

        its historical connection to the outcome. High quality and tailored data
        should be representative of the task at 

        hand and errors from data entry or other sources should be measured and
        limited. Any data used as the target 

        of a prediction process should receive particular attention to the
        quality and validity of the predicted outcome 

        or label to ensure the goal of the automated system is appropriately
        identified and measured. Additionally , 

        justification should be documented for each data attribute and source to
        explain why it is appropriate to use 

        that data to inform the results of the automated system and why such use
        will not violate any applicable laws. 

        In cases of high-dimensional and/or derived attributes, such
        justifications can be provided as overall 

        descriptions of the attribute generation process and appropriateness. 

        19
  - source_sentence: >-
      What are the key principles and frameworks mentioned in the white paper
      that govern the implementation of AI in national security and defense
      activities?
    sentences:
      - >-
        APPENDIX

         OSTP conducted meetings with a variety of stakeholders in the private
        sector and civil society. Some of these

        meetings were specifically focused on providing ideas related to the
        development of the Blueprint for an AI

        Bill of Rights while others provided useful general context on the
        positive use cases, potential harms, and/or

        oversight possibilities for these technologies. Participants in these
        conversations from the private sector and

        civil society included:

        Adobe 

        American Civil Liberties Union (ACLU) The Aspen Commission on
        Information Disorder The Awood Center The Australian Human Rights
        Commission Biometrics Institute The Brookings Institute BSA | The
        Software Alliance Cantellus Group Center for American Progress Center
        for Democracy and Technology Center on Privacy and Technology at
        Georgetown Law Christiana Care Color of Change Coworker Data Robot Data
        Trust Alliance Data and Society Research Institute Deepmind EdSAFE AI
        Alliance Electronic Privacy Information Center (EPIC) Encode Justice
        Equal AI Google Hitachi's AI Policy Committee The Innocence Project
        Institute of Electrical and Electronics Engineers (IEEE) Intuit Lawyers
        Committee for Civil Rights Under Law Legal Aid Society The Leadership
        Conference on Civil and Human Rights Meta Microsoft The MIT AI Policy
        Forum Movement Alliance Project The National Association of Criminal
        Defense Lawyers O’Neil Risk Consulting & Algorithmic Auditing The
        Partnership on AI Pinterest The Plaintext Group pymetrics SAP The
        Security Industry Association Software and Information Industry
        Association (SIIA) Special Competitive Studies Project Thorn United for
        Respect University of California at Berkeley Citris Policy Lab
        University of California at Berkeley Labor Center Unfinished/Project
        Liberty Upturn US Chamber of Commerce US Chamber of Commerce Technology
        Engagement Center 

        A.I. Working Group

        Vibrent HealthWarehouse Worker ResourceCenterWaymap

        62
      - >-
        This white paper recognizes that national security (which includes
        certain law enforcement and 

        homeland security activities) and defense activities are of increased
        sensitivity and interest to our nation’s 

        adversaries and are often subject to special requirements, such as those
        governing classified information and 

        other protected data. Such activities require alternative, compatible
        safeguards through existing policies that 

        govern automated systems and AI, such as the Department of Defense (DOD)
        AI Ethical Principles and 

        Responsible AI Implementation Pathway and the Intelligence Community
        (IC) AI Ethics Principles and 

        Framework. The implementation of these policies to national security and
        defense activities can be informed by 

        the Blueprint for an AI Bill of Rights where feasible. 

        The Blueprint for an AI Bill of Rights is not intended to, and does not,
        create any legal right, benefit, or 

        defense, substantive or procedural, enforceable at law or in equity by
        any party against the United States, its 

        departments, agencies, or entities, its officers, employees, or agents,
        or any other person, nor does it constitute a 

        waiver of sovereign immunity. 

        Copyright Information 

        This document is a work of the United States Government and is in the
        public domain (see 17 U.S.C. §105). 

        2
      - >-
        This white paper recognizes that national security (which includes
        certain law enforcement and 

        homeland security activities) and defense activities are of increased
        sensitivity and interest to our nation’s 

        adversaries and are often subject to special requirements, such as those
        governing classified information and 

        other protected data. Such activities require alternative, compatible
        safeguards through existing policies that 

        govern automated systems and AI, such as the Department of Defense (DOD)
        AI Ethical Principles and 

        Responsible AI Implementation Pathway and the Intelligence Community
        (IC) AI Ethics Principles and 

        Framework. The implementation of these policies to national security and
        defense activities can be informed by 

        the Blueprint for an AI Bill of Rights where feasible. 

        The Blueprint for an AI Bill of Rights is not intended to, and does not,
        create any legal right, benefit, or 

        defense, substantive or procedural, enforceable at law or in equity by
        any party against the United States, its 

        departments, agencies, or entities, its officers, employees, or agents,
        or any other person, nor does it constitute a 

        waiver of sovereign immunity. 

        Copyright Information 

        This document is a work of the United States Government and is in the
        public domain (see 17 U.S.C. §105). 

        2
model-index:
  - name: SentenceTransformer based on Alibaba-NLP/gte-large-en-v1.5
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: Unknown
          type: unknown
        metrics:
          - type: cosine_accuracy@1
            value: 0.7222222222222222
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.9444444444444444
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 1
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 1
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7222222222222222
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.31481481481481477
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19999999999999993
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09999999999999996
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.7222222222222222
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.9444444444444444
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 1
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 1
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.87665680931096
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.8348765432098766
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.8348765432098766
            name: Cosine Map@100
          - type: dot_accuracy@1
            value: 0.7222222222222222
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.9444444444444444
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.9814814814814815
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 1
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.7222222222222222
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.31481481481481477
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.1962962962962962
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.09999999999999996
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.7222222222222222
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.9444444444444444
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.9814814814814815
            name: Dot Recall@5
          - type: dot_recall@10
            value: 1
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.8752777468856755
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.8333333333333334
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.8333333333333334
            name: Dot Map@100

SentenceTransformer based on Alibaba-NLP/gte-large-en-v1.5

This is a sentence-transformers model finetuned from Alibaba-NLP/gte-large-en-v1.5. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: Alibaba-NLP/gte-large-en-v1.5
  • Maximum Sequence Length: 8192 tokens
  • Output Dimensionality: 1024 tokens
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: NewModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'What are the key principles and frameworks mentioned in the white paper that govern the implementation of AI in national security and defense activities?',
    'This white paper recognizes that national security (which includes certain law enforcement and \nhomeland security activities) and defense activities are of increased sensitivity and interest to our nation’s \nadversaries and are often subject to special requirements, such as those governing classified information and \nother protected data. Such activities require alternative, compatible safeguards through existing policies that \ngovern automated systems and AI, such as the Department of Defense (DOD) AI Ethical Principles and \nResponsible AI Implementation Pathway and the Intelligence Community (IC) AI Ethics Principles and \nFramework. The implementation of these policies to national security and defense activities can be informed by \nthe Blueprint for an AI Bill of Rights where feasible. \nThe Blueprint for an AI Bill of Rights is not intended to, and does not, create any legal right, benefit, or \ndefense, substantive or procedural, enforceable at law or in equity by any party against the United States, its \ndepartments, agencies, or entities, its officers, employees, or agents, or any other person, nor does it constitute a \nwaiver of sovereign immunity. \nCopyright Information \nThis document is a work of the United States Government and is in the public domain (see 17 U.S.C. §105). \n2',
    "APPENDIX\n• OSTP conducted meetings with a variety of stakeholders in the private sector and civil society. Some of these\nmeetings were specifically focused on providing ideas related to the development of the Blueprint for an AI\nBill of Rights while others provided useful general context on the positive use cases, potential harms, and/or\noversight possibilities for these technologies. Participants in these conversations from the private sector and\ncivil society included:\nAdobe \nAmerican Civil Liberties Union (ACLU) The Aspen Commission on Information Disorder The Awood Center The Australian Human Rights Commission Biometrics Institute The Brookings Institute BSA | The Software Alliance Cantellus Group Center for American Progress Center for Democracy and Technology Center on Privacy and Technology at Georgetown Law Christiana Care Color of Change Coworker Data Robot Data Trust Alliance Data and Society Research Institute Deepmind EdSAFE AI Alliance Electronic Privacy Information Center (EPIC) Encode Justice Equal AI Google Hitachi's AI Policy Committee The Innocence Project Institute of Electrical and Electronics Engineers (IEEE) Intuit Lawyers Committee for Civil Rights Under Law Legal Aid Society The Leadership Conference on Civil and Human Rights Meta Microsoft The MIT AI Policy Forum Movement Alliance Project The National Association of Criminal Defense Lawyers O’Neil Risk Consulting & Algorithmic Auditing The Partnership on AI Pinterest The Plaintext Group pymetrics SAP The Security Industry Association Software and Information Industry Association (SIIA) Special Competitive Studies Project Thorn United for Respect University of California at Berkeley Citris Policy Lab University of California at Berkeley Labor Center Unfinished/Project Liberty Upturn US Chamber of Commerce US Chamber of Commerce Technology Engagement Center \nA.I. Working Group\nVibrent HealthWarehouse Worker ResourceCenterWaymap\n62",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Metric Value
cosine_accuracy@1 0.7222
cosine_accuracy@3 0.9444
cosine_accuracy@5 1.0
cosine_accuracy@10 1.0
cosine_precision@1 0.7222
cosine_precision@3 0.3148
cosine_precision@5 0.2
cosine_precision@10 0.1
cosine_recall@1 0.7222
cosine_recall@3 0.9444
cosine_recall@5 1.0
cosine_recall@10 1.0
cosine_ndcg@10 0.8767
cosine_mrr@10 0.8349
cosine_map@100 0.8349
dot_accuracy@1 0.7222
dot_accuracy@3 0.9444
dot_accuracy@5 0.9815
dot_accuracy@10 1.0
dot_precision@1 0.7222
dot_precision@3 0.3148
dot_precision@5 0.1963
dot_precision@10 0.1
dot_recall@1 0.7222
dot_recall@3 0.9444
dot_recall@5 0.9815
dot_recall@10 1.0
dot_ndcg@10 0.8753
dot_mrr@10 0.8333
dot_map@100 0.8333

Training Details

Training Dataset

Unnamed Dataset

  • Size: 224 training samples
  • Columns: sentence_0 and sentence_1
  • Approximate statistics based on the first 224 samples:
    sentence_0 sentence_1
    type string string
    details
    • min: 23 tokens
    • mean: 36.01 tokens
    • max: 55 tokens
    • min: 22 tokens
    • mean: 569.67 tokens
    • max: 1018 tokens
  • Samples:
    sentence_0 sentence_1
    What are the primary objectives outlined in the "Blueprint for an AI Bill of Rights" as it pertains to the American people? BLUEPRINT FOR AN
    AI B ILL OF
    RIGHTS
    MAKING AUTOMATED
    SYSTEMS WORK FOR
    THE AMERICAN PEOPLE
    OCTOBER 2022
    In what ways does the document propose to ensure that automated systems are designed to work effectively for the benefit of society? BLUEPRINT FOR AN
    AI B ILL OF
    RIGHTS
    MAKING AUTOMATED
    SYSTEMS WORK FOR
    THE AMERICAN PEOPLE
    OCTOBER 2022
    What is the primary purpose of the Blueprint for an AI Bill of Rights as outlined by the White House Office of Science and Technology Policy? About this Document
    The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People was
    published by the White House Office of Science and Technology Policy in October 2022. This framework was
    released one year after OSTP announced the launch of a process to develop “a bill of rights for an AI-powered
    world.” Its release follows a year of public engagement to inform this initiative. The framework is available
    online at: https://www.whitehouse.gov/ostp/ai-bill-of-rights
    About the Office of Science and Technology Policy
    The Office of Science and Technology Policy (OSTP) was established by the National Science and Technology
    Policy, Organization, and Priorities Act of 1976 to provide the President and others within the Executive Office
    of the President with advice on the scientific, engineering, and technological aspects of the economy, national
    security, health, foreign relations, the environment, and the technological recovery and use of resources, among
    other topics. OSTP leads interagency science and technology policy coordination efforts, assists the Office of
    Management and Budget (OMB) with an annual review and analysis of Federal research and development in
    budgets, and serves as a source of scientific and technological analysis and judgment for the President with
    respect to major policies, plans, and programs of the Federal Government.
    Legal Disclaimer
    The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People is a white paper
    published by the White House Office of Science and Technology Policy. It is intended to support the
    development of policies and practices that protect civil rights and promote democratic values in the building,
    deployment, and governance of automated systems.
    The Blueprint for an AI Bill of Rights is non-binding and does not constitute U.S. government policy. It
    does not supersede, modify, or direct an interpretation of any existing statute, regulation, policy, or
    international instrument. It does not constitute binding guidance for the public or Federal agencies and
    therefore does not require compliance with the principles described herein. It also is not determinative of what
    the U.S. government’s position will be in any international negotiation. Adoption of these principles may not
    meet the requirements of existing statutes, regulations, policies, or international instruments, or the
    requirements of the Federal agencies that enforce them. These principles are not intended to, and do not,
    prohibit or limit any lawful activity of a government agency, including law enforcement, national security, or
    intelligence activities.
    The appropriate application of the principles set forth in this white paper depends significantly on the
    context in which automated systems are being utilized. In some circumstances, application of these principles
    in whole or in part may not be appropriate given the intended use of automated systems to achieve government
    agency missions. Future sector-specific guidance will likely be necessary and important for guiding the use of
    automated systems in certain settings such as AI systems used as part of school building security or automated
    health diagnostic systems.
    The Blueprint for an AI Bill of Rights recognizes that law enforcement activities require a balancing of
    equities, for example, between the protection of sensitive law enforcement information and the principle of
    notice; as such, notice may not be appropriate, or may need to be adjusted to protect sources, methods, and
    other law enforcement equities. Even in contexts where these principles may not apply in whole or in part,
    federal departments and agencies remain subject to judicial, privacy, and civil liberties oversight as well as
    existing policies and safeguards that govern automated systems, including, for example, Executive Order 13960,
    Promoting the Use of Trustworthy Artificial Intelligence in the Federal Government (December 2020).
    This white paper recognizes that national security (which includes certain law enforcement and
    homeland security activities) and defense activities are of increased sensitivity and interest to our nation’s
    adversaries and are often subject to special requirements, such as those governing classified information and
    other protected data. Such activities require alternative, compatible safeguards through existing policies that
    govern automated systems and AI, such as the Department of Defense (DOD) AI Ethical Principles and
    Responsible AI Implementation Pathway and the Intelligence Community (IC) AI Ethics Principles and
    Framework. The implementation of these policies to national security and defense activities can be informed by
    the Blueprint for an AI Bill of Rights where feasible.
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim"
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 5
  • per_device_eval_batch_size: 5
  • num_train_epochs: 2
  • multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 5
  • per_device_eval_batch_size: 5
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1
  • num_train_epochs: 2
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.0
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • eval_use_gather_object: False
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: round_robin

Training Logs

Epoch Step cosine_map@100
1.0 45 0.8179
1.1111 50 0.8318
2.0 90 0.8349

Framework Versions

  • Python: 3.10.12
  • Sentence Transformers: 3.1.1
  • Transformers: 4.44.2
  • PyTorch: 2.4.1+cu121
  • Accelerate: 0.34.2
  • Datasets: 3.0.1
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}