SWE-bench Verified on Claude Opus 5.5
500 tasks, 1 attempt, allowlisted network. Updated 19 Oct 2026.
Real GitHub issues, resolved against the projects' own test suites. Comparable with published full-set numbers, but scores this high on a public set may reflect contamination: read them as harness differences on one model, not as the model's skill. About 5-7 hours and about $80 per run. Bring your own Anthropic key.
Value
What a task cost each configuration. Claude Code used 7% more tokens than DeepSeek Harness, and 20% less time per task.
$ per solved task
Lower is cheaper. Dollars per task divided by the pass rate.
Tokens per task
Lower is leaner. Input plus output, cached input included.
Agent time per task
Lower is faster. Agent start to finish; setup and tests left out.
Pass rate against tokens per task
Up and to the left is better. Not offered: dollars, which fewer than two rows report. Pass-rate axis starts at 85%.
- On the frontier: no other row has a higher pass rate for less tokens
- Off the frontier
- 95% interval of the pass rate