Reference
SWE-bench Verified: checklist assessment
How SWE-bench Verified measures up to the Agentic Benchmark Checklist, item by item, as the maintainers assessed it. 500 real GitHub issues from Python projects, each judged by the project's own tests.
Reviewed 2026-10-29 by the maintainers, against the checklist in the dataset checklist. Source: laude-institute/harbor-datasets datasets/swebench-verified at 86723674f04e4209ac479d0fb75d9d9f44b4377e, public, no licence file in the repository. 500 real GitHub issues from Python projects, the human-validated subset of SWE-bench, each graded by the project's own test suite: the tests the fix must make pass and the tests it must keep passing. Difficulty, as the validators estimated the fix: under 15 minutes 194, 15 minutes to 1 hour 261, 1 to 4 hours 42, over 4 hours 3. Named subset sample-50. Trials get 4 GB. Run with hbench 0.3.9.
Task validity
| Item | Answer | Evidence |
|---|---|---|
| T.4 | yes | A fresh container per trial (hbench). |
| T.5 | yes | The test patch is applied after the agent stops; the gold patch is never in the container (hbench). |
| T.6 | yes | Tasks pinned by digest and checksum in the task catalog. |
| T.7 | yes | The Verified subset is the 500 issues human annotators judged well-specified with fair tests. |
| T.8 | not assessed | Environments are upstream's images; not rebuilt by the maintainers. |
| T.9 | yes | Every task ships the gold patch as its solution; hbench's oracle agent runs it. |
| T.10 | not assessed | No pilot-run outlier review by the maintainers. |
Score: 5 of 7.
Outcome validity
| Item | Answer | Evidence |
|---|---|---|
| O.b.3 | yes | A fix has to make named tests pass and keep others passing. |
| O.d.1 | yes | The annotators checked that each task's tests fairly judge a fix. |
| O.d.2 | not assessed | No coverage or similar metric. |
| O.f.2 | not assessed | Test suites of real projects; flaky tests haven't been measured here. |
Score: 2 of 4.
Benchmark reporting
| Item | Answer | Evidence |
|---|---|---|
| R.1 | yes | Tasks public; hbench Apache-2.0. |
| R.3 | no | Public since 2023; the fixes are in the projects' histories. On allowlist boards the agent can't reach PyPI, so it can't download a release that holds the fix. |
| R.5 | yes | The benchmark page says what it measures: resolving a real issue against the project's tests. |
| R.7 | yes | The Opus board's note says scores this high on a public set may reflect contamination; the methodology's "what we can't check". |
| R.9 | no | No measurement of contamination's effect here. |
| R.10 | yes | 95% interval per row, paired test between rows. |
| R.11 | yes | The methodology. |
| R.12 | no | Reference harness mini-swe-agent is planned; Δ is against the leader today. |
Score: 5 of 8.
Reading a board on it
A board on all 500 tasks runs the same tasks as published full-set numbers, but its limits, attempts, retries and hbench version can differ, so compare with the official leaderboard as a rough guide only. The 50-task board isn't comparable with full-set numbers at all.
Reviewed on October 29, 2026.