Upload 3 files

about.md

@@ -47,17 +47,4 @@ We see HAL being useful for four categories of users:
 1. Downstream users and procurers of agents: Customers looking to deploy agents can get visibility into existing benchmarks that resemble tasks of interest to them, get to know who are the developers building useful agents (and see agent demos), and identify where the state of the art is for both cost and accuracy for the tasks they are looking to solve.
 2. Agent benchmark developers: Reporting results on a centralized leaderboard could allow improved visibility into agent benchmarks that measure real-world utility.
 3. Agent developers: HAL allows for easy reproduction of past agents, clear comparison with past baselines, and a straightforward way to compete on a leaderboard.
-4. Safety researchers: Understanding the capabilities of agents on real-world safety threats, as well as the cost required to carry them out, is important for safety research. For example, evaluations on Cybench could give a sense of how well agents perform (accuracy) and which adversaries can afford such agents (cost).
-
-## Platform demo
-
-The platform features a user-friendly frontend for accessing and interacting with the evaluation results generated with the evaluation harness. 
-
-- Public leaderboards: The public leaderboard displays agent rankings for each supported benchmark. While some metrics are benchmark-specific, others, for example, cost, are reported for each. 
-- Automatic Pareto frontiers: We automatically determine the convex hull of agents for each benchmark and visualize it in a scatter plot.
-- Verified Results: We will prominently show which results we verified (by re-running them). We plan to periodically re-run the top 5 agents for each benchmark in order to always have a verified SOTA agent.
-- Task completion heatmap: Of all tasks in the benchmark, how many were solved by which agent? How many were solved by at least one?
-- Qualitative Failure Mode Analysis: In addition to the raw predictions, we also provide an LLM-based analysis of the recurring failure modes of each agent and plot the number of affected tasks for each identified failure mode. 
-- Agent Monitor: To aid visibility into which steps an agent took and compare the approaches of agents on the same task, we provide a visual overview of the steps an agent took. For each step, we include an LLM-generated summary of the action in light of the overall goal of a task.
-- Raw Traces: All uploaded results are publicly accessible, and detailed information about each evaluation, including API parameters, token usage, and I/O, is made available. 
-- Submission Interface: We provide a submission interface for researchers to upload their evaluation results in a standardized way across all benchmarks supported by the platform to support easy integration of benchmark results for any given agent.
+4. Safety researchers: Understanding the capabilities of agents on real-world safety threats, as well as the cost required to carry them out, is important for safety research. For example, evaluations on Cybench could give a sense of how well agents perform (accuracy) and which adversaries can afford such agents (cost).

app.py

@@ -391,6 +391,7 @@ with gr.Blocks(theme=my_theme, css='css.css', title="HAL: Holistic Agent Leaderb
     .user-type-links a {
         display: inline-block;
         padding: 5px 12px;
+        margin-bottom: 5px;
         background-color: #f0f4f8;
         color: #2c3e50 !important; /* Force the color change */
         text-decoration: none !important; /* Force remove underline */
@@ -1124,422 +1125,17 @@ with gr.Blocks(theme=my_theme, css='css.css', title="HAL: Holistic Agent Leaderb
                                         outputs=[raw_step_dropdown, raw_call_details])
                 raw_step_dropdown.change(update_raw_call_details, 
                                         inputs=[raw_agent_dropdown, raw_task_dropdown, raw_step_dropdown], 
-                                        outputs=[raw_call_details])
-        
-        
-        # with gr.Tab("SWE-Bench Verified"):
-        #     gr.Markdown("""SWE-bench is a dataset that tests systems' ability to solve GitHub issues automatically. Verified is a human-validated subset of 500 problems reviewed by software engineers. The  We are currently actively developing this platform and this benchmark is not fully implemented yet.""")
-        #     with gr.Row():
-        #         with gr.Column(scale=2):
-        #             Leaderboard(
-        #                 value=parse_json_files(os.path.join(abs_path, "evals_live"), 'swebench_verified'),
-        #                 select_columns=SelectColumns(
-        #                     default_selection=config.SWEBENCH_ON_LOAD_COLUMNS + ["Verified"],
-        #                     cant_deselect=["Agent Name"],
-        #                     label="Select Columns to Display:",
-        #                 ),
-        #                 hide_columns=config.SWEBENCH_HIDE_COLUMNS,
-        #                 search_columns=config.SWEBENCH_SEARCH_COLUMNS
-        #             )
-        #             gr.Markdown("""*95% CIs calculated using Student's t-distribution.*""", elem_classes=["text-right"])
-        #     with gr.Row():
-        #         scatter_plot = gr.Plot(create_scatter_plot(parse_json_files(os.path.join(abs_path, "evals_live"), 'swebench_verified', aggregate=False), "Total Cost", "Accuracy", "Total Cost (in USD)", "Accuracy", ["Agent Name"]))
-            
-        #     gr.Markdown("")
-        #     gr.Markdown("")
-        #     gr.Markdown("## Task success heatmap")
-        #     with gr.Row():
-        #         task_success_heatmap = gr.Plot()
-        #     demo.load(
-        #     lambda: create_task_success_heatmap(
-        #         preprocessor.get_task_success_data('swebench_verified'),
-        #         'SWEBench Verified'
-        #     ),
-        #     outputs=[task_success_heatmap]
-        #     )
-            
-        #     gr.Markdown("")
-        #     gr.Markdown("")
-        #     gr.Markdown("## Failure report for each agent")
-        #     with gr.Row():
-        #         with gr.Column(scale=1):
-        #             failure_report_agent_dropdown = gr.Dropdown(label="Select Agent for Failure Report")
-        #     with gr.Row():
-        #         with gr.Column(scale=1):
-        #             failure_categories_overview = gr.Markdown()
-            
-        #         with gr.Column(scale=1):
-        #             failure_categories_chart = gr.Plot()
-
-        #     # Initialize the failure report agent dropdown with all agents
-        #     demo.load(update_agent_dropdown, 
-        #             inputs=[gr.Textbox(value="swebench_verified", visible=False), gr.Textbox(value="Accuracy", visible=False)], 
-        #             outputs=[failure_report_agent_dropdown])
-            
-        #     # Update failure report when agent is selected
-        #     failure_report_agent_dropdown.change(update_failure_report,
-        #                                         inputs=[failure_report_agent_dropdown, gr.Textbox(value="swebench_verified", visible=False)],
-        #                                         outputs=[failure_categories_overview, failure_categories_chart])
-            
-        #     gr.Markdown("")
-        #     gr.Markdown("")
-        #     gr.Markdown("## Agent monitor")
-        #     with gr.Row():
-        #         with gr.Column(scale=1):
-        #             agent_dropdown = gr.Dropdown(label="Select Agent")
-        #         with gr.Column(scale=1):
-        #             task_dropdown = gr.Dropdown(label="Select SWE-Bench Task")
-        #     with gr.Row():
-        #         task_overview = gr.Markdown()
-        #     with gr.Row():
-        #         flow_chart = gr.Plot(label="Task Flow")
-
-        #     # Initialize the agent dropdown with the best agent
-        #     demo.load(update_agent_dropdown, inputs=[gr.Textbox(value="swebench_verified", visible=False), gr.Textbox(value="Accuracy", visible=False)], outputs=[agent_dropdown])
-        #     demo.load(update_task_analysis, inputs=[gr.Textbox(value="swebench_verified", visible=False), agent_dropdown], outputs=[task_overview, flow_chart, task_dropdown, gr.Textbox(visible=False)])
-
-        #     agent_dropdown.change(update_task_analysis, 
-        #                           inputs=[gr.Textbox(value="swebench_verified", visible=False), agent_dropdown],
-        #                           outputs=[task_overview, flow_chart, task_dropdown, gr.Textbox(visible=False)])
-        #     task_dropdown.change(update_task_details,
-        #                          inputs=[gr.Textbox(value="swebench_verified", visible=False), agent_dropdown, task_dropdown],
-        #                          outputs=[task_overview, flow_chart, gr.Textbox(visible=False)])
-            
-        #     gr.Markdown("## Raw predictions")
-        #     with gr.Row():
-        #         with gr.Column(scale=1):
-        #             raw_agent_dropdown = gr.Dropdown(label="Select Agent")
-        #         with gr.Column(scale=1):
-        #             raw_task_dropdown = gr.Dropdown(label="Select Task")
-        #         with gr.Column(scale=1):
-        #             raw_step_dropdown = gr.Dropdown(label="Select Step")
-            
-        #     with gr.Row():
-        #         raw_call_details = gr.HTML()
-            
-        #     def update_raw_task_dropdown(agent_name):
-        #         analyzed_traces = get_analyzed_traces(agent_name, "swebench_verified")
-        #         if not analyzed_traces:
-        #             return gr.Dropdown(choices=[], label="Select Task"), gr.Dropdown(choices=[], label="Select Step"), f"No raw predictions data available for agent: {agent_name}."
-        #         task_ids = list(analyzed_traces.keys())
-        #         steps = analyzed_traces[task_ids[0]]['steps']
-        #         return gr.Dropdown(choices=task_ids, label="Select Task", value=task_ids[0]), gr.Dropdown(choices=[(f"Step {i+1}", i) for i in range(len(steps))], label="Select Step", value=0), update_raw_call_details(agent_name, task_ids[0], 0)
-
-        #     def update_raw_step_dropdown(agent_name, task_id):
-        #         analyzed_traces = get_analyzed_traces(agent_name, "swebench_verified")
-        #         if not analyzed_traces or task_id not in analyzed_traces:
-        #             return gr.Dropdown(choices=[], label="Select Step", value="No data available.")
-        #         steps = analyzed_traces[task_id]['steps']
-        #         return gr.Dropdown(choices=[(f"Step {i+1}", i) for i in range(len(steps))], label="Select Step", value=0), format_call_info(steps[0], 0)
-
-        #     def update_raw_call_details(agent_name, task_id, step_index):
-        #         analyzed_traces = get_analyzed_traces(agent_name, "swebench_verified")
-        #         if not analyzed_traces or task_id not in analyzed_traces:
-        #             return "No data available for this selection."
-        #         steps = analyzed_traces[task_id]['steps']
-        #         if step_index is None:
-        #             return "Invalid step selection."
-        #         step = steps[step_index]
-        #         return format_call_info(step, step_index)
-
-        #     # Initialize the raw agent dropdown with all agents
-        #     demo.load(update_agent_dropdown, 
-        #             inputs=[gr.Textbox(value="swebench_verified", visible=False), gr.Textbox(value="Accuracy", visible=False)], 
-        #             outputs=[raw_agent_dropdown])
-        #     demo.load(update_raw_task_dropdown,
-        #             inputs=[raw_agent_dropdown],
-        #             outputs=[raw_task_dropdown, raw_step_dropdown])
-        #     demo.load(update_raw_call_details,
-        #             inputs=[raw_agent_dropdown, raw_task_dropdown, raw_step_dropdown],
-        #             outputs=[raw_call_details])
-
-        #     raw_agent_dropdown.change(update_raw_task_dropdown, 
-        #                             inputs=[raw_agent_dropdown], 
-        #                             outputs=[raw_task_dropdown, raw_step_dropdown, raw_call_details])
-        #     raw_task_dropdown.change(update_raw_step_dropdown, 
-        #                             inputs=[raw_agent_dropdown, raw_task_dropdown], 
-        #                             outputs=[raw_step_dropdown, raw_call_details])
-        #     raw_step_dropdown.change(update_raw_call_details, 
-        #                             inputs=[raw_agent_dropdown, raw_task_dropdown, raw_step_dropdown], 
-        #                             outputs=[raw_call_details])
-            
-        # with gr.Tab("SWE-Bench Lite"):
-        #     gr.Markdown("""SWE-bench is a dataset that tests systems' ability to solve GitHub issues automatically. Lite is a subset of 300 tasks of the original SWE-bench. We are currently actively developing this platform and this benchmark is not fully implemented yet.""")
-        #     with gr.Row():
-        #         with gr.Column(scale=2):
-        #             Leaderboard(
-        #                 value=create_leaderboard(parse_json_files(os.path.join(abs_path, "evals_live"), 'swebench_lite'), ci_metrics=['Accuracy', 'Total Cost']),
-        #                 select_columns=SelectColumns(
-        #                     default_selection=config.SWEBENCH_ON_LOAD_COLUMNS + ["Verified"],
-        #                     cant_deselect=["Agent Name"],
-        #                     label="Select Columns to Display:",
-        #                 ),
-        #                 search_columns=config.SWEBENCH_SEARCH_COLUMNS,
-        #                 hide_columns=config.SWEBENCH_HIDE_COLUMNS
-        #             )
-        #             # make right aligned markdown
-        #             gr.Markdown("""*95% CIs calculated using Student's t-distribution.*""", elem_classes=["text-right"])
-        #     with gr.Row():
-        #         scatter_plot = gr.Plot(create_scatter_plot(parse_json_files(os.path.join(abs_path, "evals_live"), 'swebench_lite', aggregate=True), "Total Cost", "Accuracy", "Total Cost (in USD)", "Accuracy", ["Agent Name"]))
-
-        #     gr.Markdown("")
-        #     gr.Markdown("")
-        #     gr.Markdown("## Task success heatmap")
-        #     with gr.Row():
-        #         task_success_heatmap = gr.Plot()
-        #     demo.load(
-        #     lambda: create_task_success_heatmap(
-        #         preprocessor.get_task_success_data('swebench_lite'),
-        #         'SWEBench Lite'
-        #     ),
-        #     outputs=[task_success_heatmap]
-        #     )
-            
-        #     gr.Markdown("")
-        #     gr.Markdown("")
-        #     gr.Markdown("## Failure report for each agent")
-        #     with gr.Row():
-        #         with gr.Column(scale=1):
-        #             failure_report_agent_dropdown = gr.Dropdown(label="Select Agent for Failure Report")
-        #     with gr.Row():
-        #         with gr.Column(scale=1):
-        #             failure_categories_overview = gr.Markdown()
-            
-        #         with gr.Column(scale=1):
-        #             failure_categories_chart = gr.Plot()
-
-        #     # Initialize the failure report agent dropdown with all agents
-        #     demo.load(update_agent_dropdown, 
-        #             inputs=[gr.Textbox(value="swebench_lite", visible=False), gr.Textbox(value="Accuracy", visible=False)], 
-        #             outputs=[failure_report_agent_dropdown])
-            
-        #     # Update failure report when agent is selected
-        #     failure_report_agent_dropdown.change(update_failure_report,
-        #                                         inputs=[failure_report_agent_dropdown, gr.Textbox(value="swebench_lite", visible=False)],
-        #                                         outputs=[failure_categories_overview, failure_categories_chart])
-            
-        #     gr.Markdown("")
-        #     gr.Markdown("")
-        #     gr.Markdown("## Agent monitor")
-        #     with gr.Row():
-        #         with gr.Column(scale=1):
-        #             agent_dropdown = gr.Dropdown(label="Select Agent")
-        #         with gr.Column(scale=1):
-        #             task_dropdown = gr.Dropdown(label="Select SWE-Bench Task")
-        #     with gr.Row():
-        #         task_overview = gr.Markdown()
-        #     with gr.Row():
-        #         flow_chart = gr.Plot(label="Task Flow")
-
-        #     # Initialize the agent dropdown with the best agent
-        #     demo.load(update_agent_dropdown, inputs=[gr.Textbox(value="swebench_lite", visible=False), gr.Textbox(value="Accuracy", visible=False)], outputs=[agent_dropdown])
-        #     demo.load(update_task_analysis, inputs=[gr.Textbox(value="swebench_lite", visible=False), agent_dropdown], outputs=[task_overview, flow_chart, task_dropdown, gr.Textbox(visible=False)])
-
-        #     agent_dropdown.change(update_task_analysis, 
-        #                           inputs=[gr.Textbox(value="swebench_lite", visible=False), agent_dropdown],
-        #                           outputs=[task_overview, flow_chart, task_dropdown, gr.Textbox(visible=False)])
-        #     task_dropdown.change(update_task_details,
-        #                          inputs=[gr.Textbox(value="swebench_lite", visible=False), agent_dropdown, task_dropdown],
-        #                          outputs=[task_overview, flow_chart, gr.Textbox(visible=False)])
-
-           
-        #     gr.Markdown("## Raw predictions")
-        #     with gr.Row():
-        #         with gr.Column(scale=1):
-        #             raw_agent_dropdown = gr.Dropdown(label="Select Agent")
-        #         with gr.Column(scale=1):
-        #             raw_task_dropdown = gr.Dropdown(label="Select Task")
-        #         with gr.Column(scale=1):
-        #             raw_step_dropdown = gr.Dropdown(label="Select Step")
-            
-        #     with gr.Row():
-        #         raw_call_details = gr.HTML()
-            
-        #     def update_raw_task_dropdown(agent_name):
-        #         analyzed_traces = get_analyzed_traces(agent_name, "swebench_lite")
-        #         if not analyzed_traces:
-        #             return gr.Dropdown(choices=[], label="Select Task"), gr.Dropdown(choices=[], label="Select Step"), f"No raw predictions data available for agent: {agent_name}."
-        #         task_ids = list(analyzed_traces.keys())
-        #         steps = analyzed_traces[task_ids[0]]['steps']
-        #         return gr.Dropdown(choices=task_ids, label="Select Task", value=task_ids[0]), gr.Dropdown(choices=[(f"Step {i+1}", i) for i in range(len(steps))], label="Select Step", value=0), update_raw_call_details(agent_name, task_ids[0], 0)
-
-        #     def update_raw_step_dropdown(agent_name, task_id):
-        #         analyzed_traces = get_analyzed_traces(agent_name, "swebench_lite")
-        #         if not analyzed_traces or task_id not in analyzed_traces:
-        #             return gr.Dropdown(choices=[], label="Select Step", value="No data available.")
-        #         steps = analyzed_traces[task_id]['steps']
-        #         return gr.Dropdown(choices=[(f"Step {i+1}", i) for i in range(len(steps))], label="Select Step", value=0), format_call_info(steps[0], 0)
-
-        #     def update_raw_call_details(agent_name, task_id, step_index):
-        #         analyzed_traces = get_analyzed_traces(agent_name, "swebench_lite")
-        #         if not analyzed_traces or task_id not in analyzed_traces:
-        #             return "No data available for this selection."
-        #         steps = analyzed_traces[task_id]['steps']
-        #         if step_index is None:
-        #             return "Invalid step selection."
-        #         step = steps[step_index]
-        #         return format_call_info(step, step_index)
-
-        #     # Initialize the raw agent dropdown with all agents
-        #     demo.load(update_agent_dropdown, 
-        #             inputs=[gr.Textbox(value="swebench_lite", visible=False), gr.Textbox(value="Accuracy", visible=False)], 
-        #             outputs=[raw_agent_dropdown])
-        #     demo.load(update_raw_task_dropdown,
-        #             inputs=[raw_agent_dropdown],
-        #             outputs=[raw_task_dropdown, raw_step_dropdown])
-        #     demo.load(update_raw_call_details,
-        #             inputs=[raw_agent_dropdown, raw_task_dropdown, raw_step_dropdown],
-        #             outputs=[raw_call_details])
-
-        #     raw_agent_dropdown.change(update_raw_task_dropdown, 
-        #                             inputs=[raw_agent_dropdown], 
-        #                             outputs=[raw_task_dropdown, raw_step_dropdown, raw_call_details])
-        #     raw_task_dropdown.change(update_raw_step_dropdown, 
-        #                             inputs=[raw_agent_dropdown, raw_task_dropdown], 
-        #                             outputs=[raw_step_dropdown, raw_call_details])
-        #     raw_step_dropdown.change(update_raw_call_details, 
-        #                             inputs=[raw_agent_dropdown, raw_task_dropdown, raw_step_dropdown], 
-        #                             outputs=[raw_call_details])
-            
-        
-        
-        # with gr.Tab("MLAgentBench"):
-        #     gr.Markdown("""MLAgentBench is a suite of end-to-end Machine Learning (ML) experimentation tasks, where the agent aims to take a given dataset and a machine learning task description and autonomously develop or improve an ML model. We are currently actively developing this platform and this benchmark is not fully implemented yet. In particular, we only include one agent and a subset of tasks for this benchmark.""")
-        #     with gr.Row():
-        #         with gr.Column(scale=2):
-        #             Leaderboard(
-        #                 value=parse_json_files(os.path.join(abs_path, "evals_live"), 'mlagentbench'),
-        #                 select_columns=SelectColumns(
-        #                     default_selection=config.MLAGENTBENCH_ON_LOAD_COLUMNS + ["Verified"],
-        #                     cant_deselect=["Agent Name"],
-        #                     label="Select Columns to Display:",
-        #                 ),
-        #                 search_columns=config.MLAGENTBENCH_SEARCH_COLUMNS,
-        #                 hide_columns=config.MLAGENTBENCH_HIDE_COLUMNS,
-        #             )
-        #             gr.Markdown("""*95% CIs calculated using Student's t-distribution.*""", elem_classes=["text-right"])
-        #     with gr.Row():
-        #         scatter_plot = gr.Plot(create_scatter_plot(parse_json_files(os.path.join(abs_path, "evals_live"), 'mlagentbench', aggregate=False), "Total Cost", "Overall Score", "Total Cost (in USD)", "Overall Score", ["Agent Name"]))
-
-        #     gr.Markdown("")
-        #     gr.Markdown("")
-        #     gr.Markdown("## Failure report for each agent")
-        #     with gr.Row():
-        #         with gr.Column(scale=1):
-        #             failure_report_agent_dropdown = gr.Dropdown(label="Select Agent for Failure Report")
-        #     with gr.Row():
-        #         with gr.Column(scale=1):
-        #             failure_categories_overview = gr.Markdown()
-            
-        #         with gr.Column(scale=1):
-        #             failure_categories_chart = gr.Plot()
-
-        #     # Initialize the failure report agent dropdown with all agents
-        #     demo.load(update_agent_dropdown, 
-        #             inputs=[gr.Textbox(value="mlagentbench", visible=False), gr.Textbox(value="Overall Score", visible=False)], 
-        #             outputs=[failure_report_agent_dropdown])
-            
-        #     # Update failure report when agent is selected
-        #     failure_report_agent_dropdown.change(update_failure_report,
-        #                                         inputs=[failure_report_agent_dropdown, gr.Textbox(value="mlagentbench", visible=False)],
-        #                                         outputs=[failure_categories_overview, failure_categories_chart])
-            
-        #     gr.Markdown("")
-        #     gr.Markdown("")
-        #     gr.Markdown("## Agent monitor")
-        #     with gr.Row():
-        #         with gr.Column(scale=1):
-        #             agent_dropdown = gr.Dropdown(label="Select Agent")
-        #         with gr.Column(scale=1):
-        #             task_dropdown = gr.Dropdown(label="Select SWE-Bench Task")
-        #     with gr.Row():
-        #         task_overview = gr.Markdown()
-        #     with gr.Row():
-        #         flow_chart = gr.Plot(label="Task Flow")
-
-        #     # Initialize the agent dropdown with the best agent
-        #     demo.load(update_agent_dropdown, inputs=[gr.Textbox(value="mlagentbench", visible=False), gr.Textbox(value="Overall Score", visible=False)], outputs=[agent_dropdown])
-        #     demo.load(update_task_analysis, inputs=[gr.Textbox(value="mlagentbench", visible=False), agent_dropdown], outputs=[task_overview, flow_chart, task_dropdown, gr.Textbox(visible=False)])
-
-        #     agent_dropdown.change(update_task_analysis, 
-        #                           inputs=[gr.Textbox(value="mlagentbench", visible=False), agent_dropdown],
-        #                           outputs=[task_overview, flow_chart, task_dropdown, gr.Textbox(visible=False)])
-        #     task_dropdown.change(update_task_details,
-        #                          inputs=[gr.Textbox(value="mlagentbench", visible=False), agent_dropdown, task_dropdown],
-        #                          outputs=[task_overview, flow_chart, gr.Textbox(visible=False)])
-
-        
-        #     gr.Markdown("## Raw predictions")
-        #     with gr.Row():
-        #         with gr.Column(scale=1):
-        #             raw_agent_dropdown = gr.Dropdown(label="Select Agent")
-        #         with gr.Column(scale=1):
-        #             raw_task_dropdown = gr.Dropdown(label="Select Task")
-        #         with gr.Column(scale=1):
-        #             raw_step_dropdown = gr.Dropdown(label="Select Step")
-            
-        #     with gr.Row():
-        #         raw_call_details = gr.HTML()
-            
-        #     def update_raw_task_dropdown(agent_name):
-        #         analyzed_traces = get_analyzed_traces(agent_name, "mlagentbench")
-        #         if not analyzed_traces:
-        #             return gr.Dropdown(choices=[], label="Select Task"), gr.Dropdown(choices=[], label="Select Step"), f"No raw predictions data available for agent: {agent_name}."
-        #         task_ids = list(analyzed_traces.keys())
-        #         steps = analyzed_traces[task_ids[0]]['steps']
-        #         return gr.Dropdown(choices=task_ids, label="Select Task", value=task_ids[0]), gr.Dropdown(choices=[(f"Step {i+1}", i) for i in range(len(steps))], label="Select Step", value=0), update_raw_call_details(agent_name, task_ids[0], 0)
-
-        #     def update_raw_step_dropdown(agent_name, task_id):
-        #         analyzed_traces = get_analyzed_traces(agent_name, "mlagentbench")
-        #         if not analyzed_traces or task_id not in analyzed_traces:
-        #             return gr.Dropdown(choices=[], label="Select Step", value="No data available.")
-        #         steps = analyzed_traces[task_id]['steps']
-        #         return gr.Dropdown(choices=[(f"Step {i+1}", i) for i in range(len(steps))], label="Select Step", value=0), format_call_info(steps[0], 0)
-
-        #     def update_raw_call_details(agent_name, task_id, step_index):
-        #         analyzed_traces = get_analyzed_traces(agent_name, "mlagentbench")
-        #         if not analyzed_traces or task_id not in analyzed_traces:
-        #             return "No data available for this selection."
-        #         steps = analyzed_traces[task_id]['steps']
-        #         if step_index is None:
-        #             return "Invalid step selection."
-        #         step = steps[step_index]
-        #         return format_call_info(step, step_index)
-
-        #     # Initialize the raw agent dropdown with all agents
-        #     demo.load(update_agent_dropdown, 
-        #             inputs=[gr.Textbox(value="mlagentbench", visible=False), gr.Textbox(value="Overall Score", visible=False)], 
-        #             outputs=[raw_agent_dropdown])
-        #     demo.load(update_raw_task_dropdown,
-        #             inputs=[raw_agent_dropdown],
-        #             outputs=[raw_task_dropdown, raw_step_dropdown])
-        #     demo.load(update_raw_call_details,
-        #             inputs=[raw_agent_dropdown, raw_task_dropdown, raw_step_dropdown],
-        #             outputs=[raw_call_details])
-
-        #     raw_agent_dropdown.change(update_raw_task_dropdown, 
-        #                             inputs=[raw_agent_dropdown], 
-        #                             outputs=[raw_task_dropdown, raw_step_dropdown, raw_call_details])
-        #     raw_task_dropdown.change(update_raw_step_dropdown, 
-        #                             inputs=[raw_agent_dropdown, raw_task_dropdown], 
-        #                             outputs=[raw_step_dropdown, raw_call_details])
-        #     raw_step_dropdown.change(update_raw_call_details, 
-        #                             inputs=[raw_agent_dropdown, raw_task_dropdown, raw_step_dropdown], 
-        #                             outputs=[raw_call_details])
-            
+                                        outputs=[raw_call_details])   
         
         with gr.Tab("About"):
             gr.Markdown((Path(__file__).parent / "about.md").read_text())
 
 
-    gr.HTML("""<h2 class="section-heading" id="agent-submission">How to add an agent?</h2>
-<p>Below we provide a guide on how to add an agent to the leaderboard:</p>""")
+    gr.HTML("""<h2 class="section-heading" id="agent-submission">How to add an agent?</h2>""")
     gr.Markdown((Path(__file__).parent / "agent_submission.md").read_text())
-    gr.HTML("""<h2 class="section-heading" id="benchmark-submission">How to add a benchmark?</h2>
-<p>Below we provide a guide on how to add a benchmark to the leaderboard:</p>""")
+    gr.HTML("""<h2 class="section-heading" id="benchmark-submission">How to add a benchmark?</h2>""")
     gr.Markdown((Path(__file__).parent / "benchmark_submission.md").read_text())
-    gr.HTML("""<h2 class="section-heading" id="reproduction-guide">How can I run evaluations?</h2>
-<p>Below we provide a guide on how to reproduce evaluations:</p>""")
+    gr.HTML("""<h2 class="section-heading" id="reproduction-guide">How can I run evaluations?</h2>""")
     gr.Markdown("""Coming soon...""")
 
 

benchmark_submission.md

@@ -1,4 +1,4 @@
-To submit **a new benchmark **to the library:
+To submit **a new benchmark** to the library:
 
 1. Implement a new benchmark using some standard format (such as the [METR Task Standard](https://github.com/METR/task-standard)). This includes specifying the exact instructions for each tasks as well as the task environment that is provided inside the container the agent is run in.