This comprehensive benchmark evaluates language models' ability to effectively utilize tools and functions in complex scenarios.
Our evaluation process follows a systematic approach to ensure comprehensive and fair assessment of AI agents. We evaluate language models' ability to effectively use tools in single and multi-turn conversations. Our evaluation focuses on both basic functionality and edge cases that challenge real-world applicability.
| Dataset | Domains | Link |
|---|---|---|
| BFCL | Mathematics, Entertainment, Education, and Academic Domains | View Dataset |
| Ο-bench | Retail and Airline Industry Scenarios | View Dataset |
| xLAM | Cross-domain Data Generation (21 Domains) | View Dataset |
| ToolACE | API Interactions across 390 Domains | View Dataset |
| Category | Finding |
|---|---|
| Performance Champion | Gemini-2.0-flash dominates with 0.938 score at a very affordable cost, excelling in both complex tasks and safety features. |
| Price-Performance Paradox | Top 3 models span 10x price difference yet only 4% performance gap, challenging pricing assumptions |
| Open Vs Closed Source | The new Mistral-small leads in open source models and performs similar to GPT-4o-mini at 0.83, signaling OSS maturity in tool calling |
| Reasoning Models | Although being great for reasoning, o1 and o3-mini are far from perfect scoring 0.87 and 0.84 respectively. DeepSeek V3 and R1 were excluded from rankings due to limited function support |
| Tool Miss Detection | Low dataset averages of 0.60(tool_miss) and 0.73(miss_func) reveal fundamental challenges in handling edge cases and maintaining context, even as models excel at basic tasks |
| Architecture Trade-offs | Long context vs parallel execution shows architectural limits: O1 leads context (0.98) but fails parallel tasks (0.43), while GPT-4o shows opposite pattern |
| Area | Recommendation |
|---|---|
| Task Complexity | Simple tasks work with most models. Complex workflows requiring multiple tools need models with 0.85+ scores in composite tests |
| Error Handling | Models with low tool selection scores need guardrails. Add validation layers and structured error recovery, especially for parameter collection |
| Context Management | Long conversations require either models strong in context retention or external context storage systems |
| Reasoning Models | While o1 and o3-mini excelled in function calling, DeepSeek V3 and R1 were excluded from rankings due to limited function support |
| Safety Controls | Add strict tool access controls for models weak in irrelevance detection. Include validation layers for inconsistent performers |
| Open Vs Closed Source | Private models lead in complex tasks, but open-source options work well for basic operations. Choose based on your scaling needs |
When an agent encounters a query, it must first determine if tool usage is warranted. Information may already exist in the conversation history, making tool calls redundant. Alternatively, available tools might be insufficient or irrelevant to the task, requiring the agent to acknowledge limitations rather than force inappropriate tool usage.
Tool selection isn't binaryβit involves both precision and recall. An agent might correctly identify one necessary tool while missing others (recall issue) or select appropriate tools alongside unnecessary ones (precision issue). While suboptimal, these scenarios represent different severity levels of selection errors.
Even with correct tool selection, argument handling introduces additional complexity. Agents must:
Multi-step tasks require agents to:
We developed the Tool Selection Quality metric to assess agents' tool call performance, evaluating tool selection accuracy and effectiveness of parameter usage. This is an example code for evaluating an LLM with a dataset with Galileo's Tool Selection Quality.
import promptquality as pq
df = pd.read_parquet(file_path)
chainpoll_tool_selection_scorer = pq.CustomizedChainPollScorer(
scorer_name=pq.CustomizedScorerName.tool_selection_quality,
model_alias=pq.Models.gpt_4o,
)
evaluate_handler = pq.GalileoPromptCallback(
project_name=project_name,
run_name=run_name,
scorers=[chainpoll_tool_selection_scorer],
)
llm = llm_handler.get_llm(model, temperature=0.0, max_tokens=4000) # llm_handler is a custom handler for LLMs
system_msg = {
"role": "system",
"content": 'Your job is to use the given tools to answer the query of human. If there is no relevant tool then reply with "I cannot answer the question with given tools". If tool is available but sufficient information is not available, then ask human to get the same. You can call as many tools as you want. Use multiple tools if needed. If the tools need to be called in a sequence then just call the first tool.',
}
for row in df.itertuples():
chain = llm.bind_tools(tools) # attach the tools
outputs.append(
chain.invoke(
[system_msg, *row.conversation],
config=dict(callbacks=[evaluate_handler])
)
)
evaluate_handler.finish()
| Type | Samples | Category | Dataset Name | Purpose |
|---|---|---|---|---|
| Single-Turn | 100 + 100 | Single Function Call | xlam_single_tool_single_call, xlam_multiple_tool_single_call | Evaluates basic ability to read documentation and make single function calls |
| 200 + 50 | Multiple Function Call | xlam_multiple_tool_multiple_call, xlam_single_tool_multiple_call | Tests parallel execution and result aggregation capabilities | |
| 100 | Irrelevant Query | BFCL_v3_irrelevance | Tests ability to recognize when available tools don't match user needs | |
| 100 | Long Context | tau_long_context | Assesses handling of extended interactions and complex instructions | |
| 100 | Missing Function | xlam_tool_miss | Tests graceful handling of unavailable tools | |
| Multi-Turn | 50 + 30 | Single Function Call | BFCL_v3_multi_turn_base_single_func_call, toolscs_single_func_call | Tests basic conversational function calling abilities |
| 50 | Multiple Function Call | BFCL_v3_multi_turn_base_multi_func_call | Evaluates handling of multiple function calls in conversation | |
| 100 | Missing Function | BFCL_v3_multi_turn_miss_func | Tests graceful handling of unavailable tools | |
| 100 | Missing Parameters | BFCL_v3_multi_turn_miss_param | Assesses parameter collection and handling incomplete information | |
| 100 | Composite | BFCL_v3_multi_turn_composite | Tests overall robustness in complex scenarios |