SWE-bench, OSWorld, GAIA, AgentBench — four distinct measuring sticks for AI agents, each encoding a different theory of what "intelligence" means in practice.
The analogy first: imagine you want to know how well a new surgeon performs. You could test their medical knowledge with a written exam (traditional ML benchmark) — or you could put them in an operating room with a real patient and measure outcomes (agent benchmark). The surgeon might ace the written test and still be terrible in practice. Agent benchmarks force the agent to actually do the work, inside real environments, with real consequences for errors.
Traditional ML benchmarks — like ImageNet or MMLU — measure what a model knows: can it pick the right answer from multiple choices? Agent benchmarks measure what a model can accomplish: given a real codebase and a bug report, can it ship a working fix? Given a real computer, can it find a file and email it? This is a categorically harder problem.
Every serious agent benchmark shares one architectural property: execution-based evaluation. Rather than checking model outputs against a reference answer (string matching, BLEU score), the benchmark actually runs the agent's output in an environment and checks what happens.
For SWE-bench: the agent writes a code patch → the patch is applied to the real repository → the full test suite runs → pass/fail is determined by whether the previously-failing tests now pass. No partial credit for "mostly right." No human judging quality. The environment is the judge.
This matters enormously because execution-based evaluation is both more honest and harder to game. A model that memorized the patch can pass, but a model that writes plausible-looking wrong code fails cleanly. The tradeoff: evaluation is expensive, slow, and environment-dependent, which is why contamination and scaffold design matter so much.
Task: Resolve real GitHub issues in Python repos
Signal: % of issues where unit tests pass after patch
Size: 2,294 tasks (Verified: 500 human-curated)
SOTA ~2026: ~81% on Verified (Claude Opus 4.5)
Task: Complete real GUI tasks on Ubuntu/Windows/macOS
Signal: % tasks completed correctly vs. human baseline
Size: 369 diverse real-world tasks
Human baseline: 72.36% (now matched by top agents)
Task: Multi-step real-world questions requiring web, tools, reasoning
Signal: % correct answers on 466 questions (300 public)
Size: 3 difficulty levels; humans score ~92%
SOTA ~2026: ~90% (top agents)
Task: 8 environments: OS, DB, KG, WebShop, web browsing, games
Signal: Composite score across all 8 task types
Size: 1,091 total tasks across environments
Key finding: Large gap between proprietary and open-source LLMs
Every public benchmark eventually becomes a training target. Once SWE-bench tasks are published, they can be included in future model training data — either intentionally or via web scraping. When this happens, benchmark scores measure memorization, not generalization. This is called data contamination.
The field's response has been multi-pronged: SWE-bench Verified uses human engineers to confirm tasks are genuinely solvable; SWE-bench Live (Microsoft, NeurIPS 2025) continuously sources new GitHub issues from after each model's training cutoff; and private test splits are now increasingly the norm for credible lab claims. OpenAI stopped self-reporting SWE-bench Verified scores in 2025 after finding contamination evidence across frontier models.
Based on the official leaderboard update run in February 2026, here are scores from the independent evaluation — notable because these results were not self-reported by the labs:
The authors collected 2,294 tasks from 12 real Python open-source repositories (Django, scikit-learn, pytest, etc.) by pulling GitHub issues alongside the merged pull request that fixed them. The "oracle patch" (the actual fix) is withheld; the agent receives only the issue text and the repository. Success is binary: do the previously-failing unit tests pass after the agent's patch is applied?
The benchmark was deliberately calibrated against real developer work rather than synthetic problems. The average fix touches 1.7 files, 3.0 functions, and 32.8 lines — genuinely representative of day-to-day software engineering. Early models scored under 5%; current best approaches exceed 80% on the Verified subset.
The team also released SWE-bench Verified: 500 tasks reviewed by Upwork software engineers who confirmed each task is (a) uniquely described in the issue, (b) solvable without inside knowledge, and (c) has tests that actually validate the fix. This became the community's preferred evaluation split.
OSWorld created the first scalable, cross-platform benchmark running inside actual virtual machines (Ubuntu, Windows, macOS). Each of 369 tasks starts from a defined computer state (specific files, apps open, settings configured), gives the agent a natural language instruction, and evaluates completion via automated scripts that check filesystem state, screen content, or application output — not just whether the agent tried.
The benchmark captures six task categories: file management, web browsing, coding, calculator/spreadsheet work, multi-app workflows, and system configuration. Human performance sits at 72.36%. When the paper was released in April 2024, the best model scored 12.24% — revealing a massive gap. By 2026, top computer-use agents have closed that gap to near-human parity on this benchmark.
Technically, the evaluation harness provisions a VM, injects the initial state, runs the agent's action stream (mouse clicks, keystrokes, screenshot observations), and then executes a custom verifier script for each task. This makes evaluation reproducible but slow (~minutes per task) and expensive at scale.
AgentBench evaluates LLMs across 8 distinct interactive environments: Operating System (bash commands), Database (SQL queries), Knowledge Graph (SPARQL), Digital Card Game, Lateral Thinking Puzzles, House Holding (household simulation), WebShop (online shopping), and Web Browsing. Each environment tests a different combination of reasoning, tool use, decision-making, and memory.
The key finding: GPT-4 dramatically outperformed all other models tested, including Claude-1 and open-source models up to 70B parameters. The paper identified three root causes of failure: poor long-term reasoning across many steps, inability to follow complex multi-part instructions, and weak decision-making under uncertainty. Open-source models scored near 0 on several environments where GPT-4 scored 40–60%.
By designing 8 environments with vastly different task structures, AgentBench revealed that model rankings are not universal — a model that excels at database queries may struggle with household simulation. This challenged the assumption of a single "agent capability" axis and pushed toward domain-specific evaluation.
NeurIPS 2024
NeurIPS 2024
ICLR'24
| Platform | SWE-bench Approach | Primary Eval Platform | Notable Stance |
|---|---|---|---|
| Claude (Anthropic) | ~80.9% Verified (Opus 4.5, Feb 2026). Claude Code is primary agent scaffold for code evals. | Internal + third-party swebench.com runs | Participates in independent runs; leads current Verified leaderboard. Constitutional AI and safety constraints affect some eval categories. |
| OpenAI (GPT-5.x) | Stopped self-reporting Verified scores (2025) after contamination concerns. Continues reporting on other splits. | Internal evals; MMAU benchmark | Most transparent about contamination risk. Operator (computer-use agent) uses internal evals rather than OSWorld as primary signal. |
| Google Gemini | Gemini 3 Flash in top tier (~80%+ on Verified). Strong on GAIA (multi-modal + web reasoning). | Vertex AI Model Eval + third-party | GAIA performance reflects Gemini's multi-modal search capabilities. OSWorld scores benefit from Gemini's vision quality. |
| Open Source (DeepSeek, Kimi, GLM) | Chinese lab models now occupy 3–4 top-10 SWE-bench Verified slots as of Feb 2026. | Community-run, academic labs | Open weights enable independent scaffolding. DeepSeek V3.2 and Kimi K2.5 at competitive performance with major frontier models, at fraction of cost. |
As of early 2026, a standard enterprise eval stack has emerged: LangSmith (or similar) for trace collection and human annotation, public benchmarks for cross-vendor comparison and press releases, and domain-specific evals as the actual procurement signal. The public benchmarks are the menu — the custom evals are the meal.
Trajectory evaluation is the next frontier: rather than just checking final output, evaluating the quality of the agent's reasoning path, tool selection, and resource efficiency during execution. This is where tau-bench's multi-turn conversation evaluation and LangSmith's trace analysis converge.