Reference
Cells and boards
The eight things a cell holds fixed, what makes two runs comparable, how the site finds a job's board, how a board ranks, and why most boards hold one harness.
A cell is everything that changes a score except the harness. Two runs are in the same cell when they ran the same tasks on the same model under the same settings. Each cell has one board, which compares the harnesses in it.
What a cell holds fixed
The site reads a job's cell from the job's config.json and its trials' lock.json:
| Part | What it is |
|---|---|
| Dataset | A known dataset, when every task matches the task catalog's by digest; otherwise the dataset's name@version, a digest over its tasks' digests, and its registry. |
| Task set | The full set, a named subset, or any other selection, known by its sorted task names. |
| Model | The model name the job's lock records for the agent, compared without case and without its provider, the part before the first /: deepseek/deepseek-flash and deepseek-flash are one model. |
| Attempts | Trials per task, from 1 to 5. |
| Time limits | The job's timeout multipliers, for the agent, its setup, the tests and the image build. |
| Memory | The memory per task container, when the job set it. |
| Network | allowlist when hbench enforced an egress allowlist for the agent, else open. See Network: allowlist or open. |
| Results epoch | The results epoch the job's lock declares. A board ranks only its newest epoch's runs. |
Everything else is the harness: the agent and its version, its options, its prompt template and extra instructions, and for an ACP agent, its source and commit.
Comparable runs
Two runs are comparable only within one cell. Change any part above and the score can move for reasons outside the harness: other tasks, a stronger model, a second attempt, more time, or a network where the agent can look answers up. The site never ranks such runs together, and nothing on the site combines boards into one number, because across models the model usually matters more than the harness.
Within one board, three things can still differ between runs: the machine, how the model was served, and the hbench release within its epoch. A slower machine hits more time limits, so read a small gap as a tie.
How the site finds a job's board
When you publish a job, the site reads its cell and looks for that cell's board. If none exists, the job opens one, named "<benchmark> on <model>", with what differs from the default beside the name: "Terminal-Bench 2.0 on DeepSeek-Flash (first-10, 3 attempts)", or "(full, open network)".
A job of a known dataset with exactly the time limits, memory and allowlist of a named board's spec goes on that named board. A job with other settings goes on a board of its own settings, which takes uploads only: nobody can issue a run of it. A job that didn't read its dataset from a registry with a version, such as one of tasks in your own folder (--path), opens no cell and stays on Your bench.
Two versions whose tasks differ only in the image they name, where the images are identical by digest, count as one dataset: terminal-bench@2.0 and terminal-bench@2.0.1 share their boards.
The board
A board has one row per configuration: the agent with its options, prompt template and instructions, and the version that ran. Runs of one configuration share a row, whoever made them and whatever they named their entry. The version is the one the job's trials report, else the one the options pin; a row whose runs recorded neither says version not recorded.
Each row shows its score with a 95% interval, and its runs: how many crashed, and how many were issued, uploaded and made public. How good? pass@1 with its 95% interval says how the score and interval are worked out. Each run's page has the run's retries, hbench release and date.
A run in which no trial reached a verdict counts as crashed in its row and stays out of the score. The maintainers' runs share rows with everyone's, and a board doesn't mark them.
How a board ranks
A board orders its rows by pass@1, highest first. A row shows a rank range once two configurations on its board have a score, and one run from one participant is enough. Better than whom? explains each row's rank range, the board's resolution and each row's Δ. The board's tabs show the same rows by value, by run and by version, and compare two rows task by task.
Most boards hold one harness, because anyone can open a board and most settings are one person's choice. To be compared, join a board that has runs.
Named and featured boards
A named board is one the site's policy defines, with a spec that sets its time limits, memory and allowlist; you can issue a run on it. The maintainers feature some of them on the home page, and only boards on a known dataset can be featured. The home page shows only the runs of the accounts the maintainers choose for it. A board's own page and the benchmarks page show everyone's runs.
A board's slug is the last part of its address, such as swe-bench-verified-claude-opus-5-5. A slug never changes meaning: a run with another model, task set, attempts or limits goes on another board.
Reviewed on October 31, 2026.