Reference
The dataset checklist
What makes a dataset reviewed, the Agentic Benchmark Checklist items each assessment answers, how they are scored, and what hbench and the site answer for every dataset.
A dataset is reviewed when the task catalog lists it and this folder holds its assessment against the Agentic Benchmark Checklist (ABC; Zhu et al., 2025, arXiv 2507.02825). The repository's checks refuse a catalog entry without one. Each benchmark page links its assessment, and a dataset outside the catalog shows as unreviewed.
How an assessment is written
One page per dataset, at /docs/reference/datasets/<benchmark key>, the key its benchmark page uses, such as Terminal-Bench 2.0. It states what was reviewed (source repository and commit, task count, hbench version) and answers the checklist items below, one line of evidence each. Answers are yes, no or not assessed; the paper scores an item 1 or 0, so not assessed counts as 0. Each part's score is yes over applicable items. An item that doesn't apply says n/a and is left out of the score.
The items, numbered as the paper's figures number them, kept to those that apply to tasks graded by tests in a container:
| Part | Item | Question |
|---|---|---|
| Task validity | T.4 | Is every task's state clean before it starts? |
| T.5 | Is the agent isolated from the ground truth (tests, reference solution)? | |
| T.6 | Is the environment frozen at release and reproducible? | |
| T.7 | Were the ground-truth annotations verified? | |
| T.8 | Was the task setup verified? | |
| T.9 | Is there an automatic oracle solver? | |
| T.10 | Were outliers in pilot runs inspected for task bugs? | |
| Outcome validity | O.b.3 | Does the grader prevent success by guessing? |
| O.d.1 | Were the test cases checked by hand for correctness and quality? | |
| O.d.2 | Were the tests measured with an objective quality metric, such as coverage? | |
| O.f.2 | Is grading free of non-determinism, so a result repeats? | |
| Benchmark reporting | R.1 | Are the dataset and the harness open source? |
| R.3 | Are there measures against training-data contamination? | |
| R.5 | Does the benchmark say which capability it measures? | |
| R.7 | Are the limits of its validity documented? | |
| R.9 | Is there evidence of how much those limits matter? | |
| R.10 | Are results reported with statistical significance? | |
| R.11 | Are there guidelines for reading the results? | |
| R.12 | Is there a baseline to compare against? |
T.1 to T.3 (external tool APIs) and the information-acquisition, state-space and reasoning items (O.a, O.c, O.e, O.g, O.h, O.i) don't apply to these datasets and aren't listed. Where an answer is the same for every dataset because hbench or the site provides it, the assessment says so.
What hbench and the site provide for every dataset
T.4, T.5, T.6, T.9. hbench runs each trial in a fresh container, removes the tests and the reference solution before the agent starts and copies the tests in after it stops, and every task ships a reference solution that hbench's
oracleagent runs (the site refusesoraclejobs). The task catalog pins each task by digest and checksum, and the site checks every trial of a known dataset against them.O.f.2. The site measures what non-determinism is left: each row's spread over its runs, and a 95% bootstrap interval over the tasks.
R.1, R.10, R.11. hbench is Apache-2.0. Every score carries its interval, two rows are told apart by a paired test, and the methodology says how to read a board.
R.3 is answered per dataset: the site takes no measure against contamination, and the methodology says every public task set may be in a model's training data.
R.12. A reference harness per benchmark is planned; until it lands, Δ is against the leader.
Assessed by the maintainers. When a dataset's version changes, its assessment is reviewed again and its date updated.
Reviewed on October 29, 2026.