Hugging Face's logo

Spaces:
SE-Arena
/
Software-Engineering-Arena
Running

App Files Community
Software-Engineering-Arena / README.md
zhiminy's picture
zhiminy
Update README.md
db88383 unverified
preview code
|
raw
history blame contribute delete
3.63 kB

A newer version of the Gradio SDK is available: 5.20.0

Upgrade
metadata
title: SE-Arena
emoji: 🛠️
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 5.8.0
app_file: app.py
hf_oauth: true
pinned: false
short_description: The chatbot arena for software engineering

SE Arena: Evaluate Best SE Chatbots with Long-Context Interactions

Welcome to SE Arena, an open-source platform designed for evaluating software engineering-focused chatbots. SE Arena benchmarks foundation models (FMs), such as large language models (LLMs), in iterative, context-rich workflows that are characteristic of software engineering (SE) tasks.

Key Features

  • Advanced Pairwise Comparisons: Assess chatbots using Elo score, PageRank, and Newman modularity to understand both global performance and task-specific strengths.
  • Interactive Evaluation: Test chatbots in multi-round conversations tailored for SE tasks like debugging, code generation, and requirement refinement.
  • Open-Source: Built on Hugging Face Spaces, enabling transparency and fostering community-driven innovation.
  • Transparent Leaderboard: View real-time model rankings across diverse SE workflows, updated using advanced evaluation metrics.

Why SE Arena?

Existing evaluation frameworks often do not address the complex, iterative nature of SE tasks. SE Arena fills this gap by:

  • Supporting long-context, multi-turn evaluations to capture iterative workflows.
  • Allowing anonymous model comparisons to prevent bias.
  • Providing rich, multidimensional metrics for more nuanced model evaluations.

How It Works

  1. Submit a Prompt: Sign in and input your SE-related task (e.g., debugging, code reviews).
  2. Compare Responses: Two anonymous chatbots provide responses to your query.
  3. Vote: Choose the better response, mark as tied, or select "Can't Decide."
  4. Iterative Testing: Continue the conversation with follow-up prompts to test contextual understanding over multiple rounds.

Getting Started

Prerequisites

  • A Hugging Face account.
  • Basic understanding of software engineering workflows.

Usage

  1. Navigate to the SE Arena platform.
  2. Sign in with your Hugging Face account.
  3. Enter your SE task prompt and start evaluating model responses.
  4. Vote on the better response or continue multi-round interactions to test contextual understanding.

Contributing

We welcome contributions from the community! Here's how you can help:

  1. Submit Prompts: Share your SE-related tasks to enrich our evaluation dataset.
  2. Report Issues: Found a bug or have a feature request? Open an issue in this repository.
  3. Enhance the Codebase: Fork the repository, make your changes, and submit a pull request.

Privacy Policy

Your interactions are anonymized and used solely for improving SE Arena and FM benchmarking. By using SE Arena, you agree to our Terms of Service.

Future Plans

  • Enhanced Metrics: Add round-wise analysis and context-aware evaluation metrics.
  • Domain-Specific Sub-Leaderboards: Rankings focused on tasks like debugging, requirement refinement, etc.
  • Advanced Context Compression: Techniques like LongRope and SelfExtend to manage long-term memory.
  • Support for Multimodal Models: Evaluate models that integrate text, code, and other modalities.

Contact

For inquiries or feedback, please open an issue in this repository. We welcome your contributions and suggestions!