Reference
Terminal-Bench 2.0: checklist assessment
How Terminal-Bench 2.0 measures up to the Agentic Benchmark Checklist, item by item, as the maintainers assessed it. 89 tasks an agent carries out in a Linux container, each checked by its own tests.
Reviewed 2026-10-29 by the maintainers, against the checklist in the dataset checklist. Source: laude-institute/terminal-bench-2 at 69671fbaac6d67a7ef0dfec016cc38a64ef7a77c, Apache-2.0; issued as terminal-bench@2.0.1 from oci://ghcr.io/node-q/hbench-datasets, the same tasks with their images pinned by digest. 89 tasks an agent carries out in a Linux container, each with its own tests, time limit and memory. Categories: software engineering 26, system administration 9, scientific computing 8, security 8, data science 8, debugging 5, file operations 5, model training 4, other 16. Difficulty, as upstream labels it: easy 4, medium 55, hard 30. Run with hbench 0.3.9.
Task validity
| Item | Answer | Evidence |
|---|---|---|
| T.4 | yes | A fresh container per trial (hbench). |
| T.5 | yes | Tests and solution/ removed before the agent starts, tests copied in after (hbench). |
| T.6 | yes | Tasks pinned by digest and checksum in the task catalog; images by digest in 2.0.1. |
| T.7 | not assessed | Upstream reviews each task; the maintainers haven't repeated it. |
| T.8 | not assessed | Two tasks need 4 GB (caffe-cifar-10, mteb-retrieve), set in policy.yaml; the rest run at 2 GB unchecked. |
| T.9 | yes | Every task ships a solution; hbench's oracle agent runs it. |
| T.10 | not assessed | No pilot-run outlier review by the maintainers. |
Score: 4 of 7.
Outcome validity
| Item | Answer | Evidence |
|---|---|---|
| O.b.3 | not assessed | Tasks are graded on the container's final state; whether any accepts a guessed value hasn't been checked. |
| O.d.1 | not assessed | Tests are upstream's; not read task by task by the maintainers. |
| O.d.2 | not assessed | No coverage or similar metric. |
| O.f.2 | not assessed | Some tasks grade services the agent started or depend on timing; the site reports each row's spread instead. |
Score: 0 of 4.
Benchmark reporting
| Item | Answer | Evidence |
|---|---|---|
| R.1 | yes | Tasks public under Apache-2.0; hbench Apache-2.0. |
| R.3 | no | Public since 2025; no held-out tasks. |
| R.5 | yes | The benchmark page says what it measures: carrying out work in a terminal, graded by tests. |
| R.7 | yes | The methodology's "what we can't check"; the 7B board's page says its model rarely solves the tasks. |
| R.9 | no | No measurement of how much contamination or task bugs move scores here. |
| R.10 | yes | 95% interval per row, paired test between rows. |
| R.11 | yes | The methodology. |
| R.12 | no | Reference harness terminus-2 is planned; Δ is against the leader today. |
Score: 5 of 8.
Reading a board on it
Scores compare harnesses on one model. A board on this dataset says nothing about Terminal-Bench's own leaderboard, which runs other conditions. On the free 7B model almost no task is solved, so use that board to check that a harness runs from start to finish.
Reviewed on October 31, 2026.