Terminal-Bench 2.0 on GPT-5.5
89 tasks, 5 attempts, open network. Updated 25 Oct 2026.
Compare
Pick two harnesses to compare them task by task.
clnkr 0.3.11 against Capy, task by task
Over the 89 tasks both ran, 5 attempts each
clnkr 0.3.11 minus Capy, pass rate
−17.1 points
95% interval −24.5 to −10.1
The interval leaves out zero, so the paired test tells the two apart.
Where the tasks landed
- 1Only clnkr 0.3.11 solved
- 57Both solved
- 19Only Capy solved
- 12Neither solved
Of 89 tasks, 1 only clnkr 0.3.11 solved, 19 only Capy solved, 57 both solved and 12 neither did.
| Task | Reward | Tokens | Time | |||
|---|---|---|---|---|---|---|
| clnkr 0.3.11 | Capy | clnkr 0.3.11 | Capy | clnkr 0.3.11 | Capy | |
custom-memory-heap-crash | 0 | 1 | — | — | 19.2 min | 4.0 min |
gcode-to-text | 0 | 1 | — | — | 7.5 min | 8.1 min |
gpt2-codegolf | 0 | 1 | — | — | 5.2 min | 10.3 min |
mteb-retrieve | 0 | 1 | — | — | 1.2 min | 5.4 min |
polyglot-c-py | 0 | 1 | — | — | 57 s | 4.7 min |
protein-assembly | 0 | 1 | — | — | 6.1 min | 8.4 min |
pypi-server | 1 | 0 | — | — | 1.0 min | 2.3 min |
adaptive-rejection-sampler | 0 | 0.8 | — | — | 15.0 min | 8.3 min |
overfull-hbox | 0.2 | 1 | — | — | 1.4 min | 5.7 min |
query-optimize | 0.2 | 1 | — | — | 10.0 min | 14.4 min |
build-pov-ray | 0.4 | 1 | — | — | 6.9 min | 4.7 min |
dna-insert | 0 | 0.6 | — | — | 2.2 min | 5.2 min |
path-tracing-reverse | 0.4 | 1 | — | — | 6.5 min | 5.3 min |
pytorch-model-recovery | 0.4 | 1 | — | — | 1.4 min | 9.3 min |
regex-chess | 0.4 | 1 | — | — | 9.9 min | 19.2 min |
break-filter-js-from-html | 0.4 | 0.8 | — | — | 1.9 min | 9.1 min |
compile-compcert | 0.6 | 1 | — | — | 35.7 min | 23.1 min |
db-wal-recovery | 0 | 0.4 | — | — | 4.1 min | 5.8 min |
make-mips-interpreter | 0.4 | 0.8 | — | — | 6.2 min | 10.2 min |
polyglot-rust-c | 0 | 0.4 | — | — | 3.2 min | 5.7 min |
tune-mjcf | 0.6 | 1 | — | — | 10.1 min | 9.5 min |
video-processing | 0.4 | 0 | — | — | 2.6 min | 8.1 min |
extract-moves-from-video | 0.2 | 0.6 | — | — | 30.0 min | 28.9 min |
model-extraction-relu-logits | 0.2 | 0.6 | — | — | 1.7 min | 6.2 min |
chess-best-move | 0.6 | 0.8 | — | — | 5.0 min | 5.0 min |
torch-pipeline-parallelism | 0.6 | 0.8 | — | — | 1.7 min | 13.3 min |
dna-assembly | 0 | 0.2 | — | — | 10.1 min | 7.8 min |
install-windows-3.11 | 0 | 0.2 | — | — | 15.8 min | 28.6 min |
raman-fitting | 0 | 0.2 | — | — | 6.2 min | 7.9 min |
configure-git-webserver | 1 | 0.8 | — | — | 8.2 min | 5.1 min |
extract-elf | 0.4 | 0.6 | — | — | 1.7 min | 5.3 min |
hf-model-inference | 0.8 | 1 | — | — | 5.4 min | 3.3 min |
log-summary-date-ranges | 0.8 | 1 | — | — | 34 s | 1.4 min |
mailman | 0.8 | 1 | — | — | 6.1 min | 9.0 min |
mcmc-sampling-stan | 1 | 0.8 | — | — | 15.8 min | 19.7 min |
password-recovery | 0.8 | 1 | — | — | 3.9 min | 2.9 min |
path-tracing | 0.8 | 1 | — | — | 22.4 min | 8.4 min |
portfolio-optimization | 0.8 | 1 | — | — | 2.3 min | 7.5 min |
qemu-alpine-ssh | 0.8 | 1 | — | — | 7.0 min | 11.3 min |
sanitize-git-repo | 1 | 0.8 | — | — | 1.6 min | 4.7 min |
winning-avg-corewars | 1 | 0.8 | — | — | 6.6 min | 30.0 min |
write-compressor | 0.8 | 1 | — | — | 7.9 min | 4.9 min |