Spaces:
Sleeping
Sleeping
#!/usr/bin/env python3 | |
# -*- coding: utf-8 -*- | |
""" | |
Created on Fri Oct 11 18:53:59 2024 | |
@author: legalchain | |
""" | |
default_prompt = """ | |
Evaluate the given dataset card based on these mandatory criteria: | |
1. Purpose and Applications: Clearly articulates the dataset's purpose and potential real-world applications. | |
2. Unique Value: Explicitly states what makes this dataset unique or valuable compared to others. | |
3. Content Description: Provides detailed information about the actual data content, not just structure. | |
4. Completeness: Covers dataset size, data distribution, collection methodology, and any preprocessing steps. | |
5. Limitations and Biases: Discusses any known limitations, biases, or ethical considerations. | |
6. Human Curation: Shows evidence of human-added insights, context, or valuable information beyond auto-generated content. | |
Rate the card from 1 to 5, with half-points allowed (e.g., 2.5): | |
Dataset Card to Evaluate: {readme_content} | |
""" | |
prompt_enhanced = """Evaluate the given dataset card based on these mandatory criteria: | |
# Steps | |
1. **Data Origin**: Evaluate the description of where the data originates from. Check if it includes details like data source, collection methods, and any preprocessing steps. | |
2. **Usage**: Assess how the model usage is outlined in the card. Consider the intended applications, recommended scenarios, and potential misuse cases. | |
3. **Biases**: Review the section addressing biases. Determine if it identifies possible biases in the data or the model's output and how these are mitigated or acknowledged. | |
4. **Performance**: Check for clear information on the model's performance, including metrics, benchmarks, and testing conditions. | |
5. **Limitations**: Look for a discussion of the model's limitations, noting any restrictions on the types of data or contexts it should be used in. | |
6. **Ethics and Safety**: Evaluate any ethical considerations and safety precautions mentioned, focusing on steps taken to ensure responsible use of the model. | |
7. **Transparency and Explainability**: Determine if the card includes details that enhance transparency, such as architecture, algorithms used, and explainability measures. | |
# Output Format | |
Provide a detailed paragraph for each aspect (data origin, usage, biases, etc.) specifying the strengths and weaknesses observed in the model card. Summarize with a concluding statement about the overall quality and completeness of the model card. | |
# Examples | |
- **Data Origin**: The model card should comprehensively detail the origin of data, including specific datasets used, any licensing or ethical considerations in data collection, and a summary of preprocessing steps (e.g., "The data originates from public health records collected between 2010-2020, after anonymization and normalization processes were applied"). | |
- **Biases**: An exemplary model card would identify potential biases by describing the demographic distribution of the training data (e.g., "The data set has a skew towards urban populations, which may affect the model's accuracy in rural settings"). | |
# Notes | |
- Rate the card from 1 to 5, with half-points allowed (e.g., 2.5): | |
- Create a table with marddown with items name and score at the beginning of your response | |
- Pay special attention to how each section contributes to understanding the model's behavior and reliability. | |
- Consider the completeness and clarity of the information presented. | |
- Document any missing elements or recommendations for improvement. | |
Dataset Card to Evaluate: {readme_content} | |
""" |