Terminal-Bench 2.0 on GPT-5.5
89 tasks, 5 attempts, open network. Updated 25 Oct 2026.
Compare
Pick two harnesses to compare them task by task.
Capy against clnkr 0.3.11, task by task
Over the 89 tasks both ran, 5 attempts each
Capy minus clnkr 0.3.11, pass rate
+17.1 points
95% interval +10.1 to +24.5
The interval leaves out zero, so the paired test tells the two apart.
Where the tasks landed
- 19Only Capy solved
- 57Both solved
- 1Only clnkr 0.3.11 solved
- 12Neither solved
Of 89 tasks, 19 only Capy solved, 1 only clnkr 0.3.11 solved, 57 both solved and 12 neither did.
| Task | Reward | Tokens | Time | |||
|---|---|---|---|---|---|---|
| Capy | clnkr 0.3.11 | Capy | clnkr 0.3.11 | Capy | clnkr 0.3.11 | |
custom-memory-heap-crash | 1 | 0 | — | — | 4.0 min | 19.2 min |
gcode-to-text | 1 | 0 | — | — | 8.1 min | 7.5 min |
gpt2-codegolf | 1 | 0 | — | — | 10.3 min | 5.2 min |
mteb-retrieve | 1 | 0 | — | — | 5.4 min | 1.2 min |
polyglot-c-py | 1 | 0 | — | — | 4.7 min | 57 s |
protein-assembly | 1 | 0 | — | — | 8.4 min | 6.1 min |
pypi-server | 0 | 1 | — | — | 2.3 min | 1.0 min |
adaptive-rejection-sampler | 0.8 | 0 | — | — | 8.3 min | 15.0 min |
overfull-hbox | 1 | 0.2 | — | — | 5.7 min | 1.4 min |
query-optimize | 1 | 0.2 | — | — | 14.4 min | 10.0 min |
build-pov-ray | 1 | 0.4 | — | — | 4.7 min | 6.9 min |
dna-insert | 0.6 | 0 | — | — | 5.2 min | 2.2 min |
path-tracing-reverse | 1 | 0.4 | — | — | 5.3 min | 6.5 min |
pytorch-model-recovery | 1 | 0.4 | — | — | 9.3 min | 1.4 min |
regex-chess | 1 | 0.4 | — | — | 19.2 min | 9.9 min |
break-filter-js-from-html | 0.8 | 0.4 | — | — | 9.1 min | 1.9 min |
compile-compcert | 1 | 0.6 | — | — | 23.1 min | 35.7 min |
db-wal-recovery | 0.4 | 0 | — | — | 5.8 min | 4.1 min |
make-mips-interpreter | 0.8 | 0.4 | — | — | 10.2 min | 6.2 min |
polyglot-rust-c | 0.4 | 0 | — | — | 5.7 min | 3.2 min |
tune-mjcf | 1 | 0.6 | — | — | 9.5 min | 10.1 min |
video-processing | 0 | 0.4 | — | — | 8.1 min | 2.6 min |
extract-moves-from-video | 0.6 | 0.2 | — | — | 28.9 min | 30.0 min |
model-extraction-relu-logits | 0.6 | 0.2 | — | — | 6.2 min | 1.7 min |
chess-best-move | 0.8 | 0.6 | — | — | 5.0 min | 5.0 min |
torch-pipeline-parallelism | 0.8 | 0.6 | — | — | 13.3 min | 1.7 min |
dna-assembly | 0.2 | 0 | — | — | 7.8 min | 10.1 min |
install-windows-3.11 | 0.2 | 0 | — | — | 28.6 min | 15.8 min |
raman-fitting | 0.2 | 0 | — | — | 7.9 min | 6.2 min |
configure-git-webserver | 0.8 | 1 | — | — | 5.1 min | 8.2 min |
extract-elf | 0.6 | 0.4 | — | — | 5.3 min | 1.7 min |
hf-model-inference | 1 | 0.8 | — | — | 3.3 min | 5.4 min |
log-summary-date-ranges | 1 | 0.8 | — | — | 1.4 min | 34 s |
mailman | 1 | 0.8 | — | — | 9.0 min | 6.1 min |
mcmc-sampling-stan | 0.8 | 1 | — | — | 19.7 min | 15.8 min |
password-recovery | 1 | 0.8 | — | — | 2.9 min | 3.9 min |
path-tracing | 1 | 0.8 | — | — | 8.4 min | 22.4 min |
portfolio-optimization | 1 | 0.8 | — | — | 7.5 min | 2.3 min |
qemu-alpine-ssh | 1 | 0.8 | — | — | 11.3 min | 7.0 min |
sanitize-git-repo | 0.8 | 1 | — | — | 4.7 min | 1.6 min |
winning-avg-corewars | 0.8 | 1 | — | — | 30.0 min | 6.6 min |
write-compressor | 1 | 0.8 | — | — | 4.9 min | 7.9 min |