SWE-bench is a benchmark that gives a coding agent a real, historical GitHub issue from a set of open-source Python repositories and checks whether the agent's generated patch makes the repository's existing test suite pass, including tests specifically written to verify that issue's fix.

Does a high SWE-bench score mean an AI is good at all programming languages?

No. The benchmark's tasks are drawn from a small set of Python repositories, so a score measures performance on that specific distribution of issues, codebases and language — not a general claim about ability in other languages, frameworks, or the kind of greenfield feature work that never appears as a bug-report-with-a-fix.

Does SWE-bench measure code quality, not just whether tests pass?

No. Success is a binary pass/fail on the test suite; a patch that makes tests pass through an ugly, overly narrow, or maintainability-hostile change scores identically to a clean, well-generalized fix, because the benchmark has no mechanism for judging code quality beyond the tests it runs.

Can a model's SWE-bench score be inflated by training data contamination?

It's a documented general risk for any benchmark built from public GitHub data — a model trained on data that includes the benchmark's source repositories, including commits made after the issues were resolved, could have effectively seen the answer during training. Benchmark maintainers take steps to mitigate this, but it remains a standing methodological concern for any public-code-derived benchmark, not unique to SWE-bench.

What SWE-bench Actually Measures (and What It Doesn't)

SWE-bench gives a coding agent a real, historical GitHub issue from a fixed set of open-source Python repositories and scores success as pass/fail on whether the agent's patch makes the repository's test suite pass. It measures issue-resolution ability on that specific task distribution — not general coding ability or code quality.

Key takeaways

The task shape is narrow and specific: an existing, real GitHub issue in one of a fixed set of Python repositories, with a known fix and known tests that verify it.
Success is binary pass/fail on tests, not a quality judgment — an ugly, narrowly-targeted patch that happens to pass scores the same as a clean, well-generalized one.
The language and codebase distribution is fixed, so a score says little about performance in other languages, frameworks, or unfamiliar codebases outside that set.
Bug-fixing and feature-building are different tasks. SWE-bench measures the former specifically; it says nothing directly about designing and building new functionality from a specification.
Training data contamination is a standing methodological concern for any benchmark built from public GitHub data, not a flaw unique to SWE-bench, but a real reason to read scores with some caution.

What the benchmark's task actually looks like

Each SWE-bench task starts from a real, previously-resolved GitHub issue: a bug report or feature request against one of a curated set of popular open-source Python repositories, paired with the pull request that actually fixed it in the repository's real history. The agent under test receives the issue and the codebase at the commit before the fix, and has to produce a patch. Success is determined by running the repository's real test suite, including tests specifically added by the historical fix to verify that exact bug is resolved — a mechanical, reproducible pass/fail, not a human judgment call.

What SWE-bench's setup captures vs. what it structurally can't.
Property	Does SWE-bench measure it?	Why
Resolving a real, existing bug report with a known fix	Yes	Exactly the task shape the benchmark is built from
Working correctly across many programming languages	No	Tasks are drawn from a fixed set of Python repositories
Code quality, maintainability, or style of the fix	No	Scoring is binary pass/fail on tests, with no quality dimension
Designing and building a new feature from a spec	No	Every task starts from an existing, already-diagnosed issue with a known correct fix
Performance on an unfamiliar, out-of-distribution codebase	No	The repository set is fixed and, for widely-used models, plausibly present in training data

A passing patch isn't necessarily a good patch. A fix that hard-codes the specific test case's expected value, or narrowly patches around the symptom without addressing the underlying cause, can pass the benchmark's test suite identically to a properly generalized fix. The benchmark has no mechanism to distinguish the two, because its only signal is whether the tests pass.

The training-data contamination question

SWE-bench tasks are built from public GitHub repositories and their real commit histories — the same kind of data large language models are commonly trained on. A model trained on data that includes these repositories, potentially including the actual fix commits made after the issues in the benchmark were originally resolved, could have effectively seen the answer during training rather than solving the problem fresh. Benchmark maintainers take steps to reduce this risk, but it remains a standing, general methodological concern for any benchmark constructed from public code — not a flaw specific to SWE-bench, and not fully solvable without a genuinely held-out task set that never touches public data.

Documented vs. inferred vs. unsupported

What's actually established about SWE-bench and benchmark scores generally.
Claim	Status	Basis
SWE-bench scores success as pass/fail against a real repository's test suite	Documented	Directly described in the benchmark's published methodology
Training data contamination is a general risk for public-code-derived benchmarks	Documented	Widely discussed methodological concern across the benchmark research community, not specific to any one benchmark
A high SWE-bench score means a model is generally good at programming	Unsupported	The benchmark's task distribution (issue-resolution, fixed Python repos) is narrower than "programming ability" as commonly implied when scores are cited
SWE-bench scores are comparable across model releases without checking methodology changes	Inferred	Benchmark variants and evaluation harnesses have changed over time; comparing raw numbers across different setups without checking they're the same variant risks an apples-to-oranges comparison

Curious about a different AI-generated-code failure mode? The confident-wrong-output piece covers why passing a benchmark's tests and being actually correct aren't automatically the same thing, from a different angle.

Badri Dutta

Software engineer · 15 years building for the web

Fifteen years building for the web, now including daily work with AI coding agents. This piece exists because a leaderboard percentage gets repeated as a general ability claim far more often than the underlying task definition actually supports.

Full background →

What SWE-bench actually measures (and what it doesn't)

Key takeaways

What the benchmark's task actually looks like

The training-data contamination question

Documented vs. inferred vs. unsupported

Badri Dutta

Questions about SWE-bench

Continue reading