Reference
Glossary
Each word Harness Playground uses, in one line, with a link to the page that explains it.
95% interval
The range a score would likely fall in if the tasks were drawn again, from a bootstrap over the board's tasks. See How good?
Abandon
To give up a run you were issued and can't finish, with Abandon beside the run on Your bench. It still counts toward the day's limit: see Limits.
ACP
The Agent Client Protocol, which drives a coding agent over stdin and stdout. See Run your own harness or ACP agent.
Agent
A harness that hbench runs, built in or your own. See Agents.
Agent option
A setting an agent takes, such as max_tokens, passed as --ak NAME=VALUE (hbench calls them kwargs). See Agents.
API token
A token starting with hpg_ that lets a script upload jobs and issue runs as you. Create one in Account settings; see Upload from a script.
Attempts
How many trials each task gets in a run, from 1 to 5. Part of a run's cell.
Benchmark
A dataset as the site lists it, such as SWE-bench Verified, with one board per model and settings someone has run. See Datasets and registries.
Board
The results of one cell: one row per configuration, scored over every published run of it. See Cells and boards.
Cell
Everything that changes a score except the harness. Two runs compare only in the same cell. See Cells and boards.
Configuration
An agent with its options, prompt template, extra instructions and version: one row on a board. See The board.
Crashed run
A run in which no trial reached a verdict. It counts in its row but not in the score, and shows elsewhere as all errored. See Errors, timeouts and all-errored runs.
Dataset identity
How the site tells datasets apart: by their tasks' digests. See Dataset identity.
Deny list
What the site never accepts, such as the oracle agent. See The deny list.
Entry
Your runs of one agent with one set of options, under a name the site gives it, such as terminus-2-a3f9c1. See Entries.
Errored
A trial that ended in an error before the tests gave a reward. It counts as 0, and Your bench shows No reward. See A trial's outcome.
Failed
A trial whose tests ran and didn't pass. Its reward is 0. See A trial's outcome.
Featured board
A board the maintainers show on the home page; only boards on a known dataset can be. See Named and featured boards.
Free board
A board on a model you serve on your own machine, such as Qwen2.5-Coder 7B, so it needs no model key.
Full run
A run with a result for every task of its board. See Score.
Harness
Everything between the task's instruction and the model: the prompt, the tools, the loop and when to stop. See What is an agent harness?
hbench
The benchmark runner: it runs each task in Docker, drives the agent, runs the tests and writes the job. See hbench command line.
Issue
To have the site create a run before you start it, so its board counts it once it's uploaded or expires. See Issue a run first.
Job
The folder hbench writes for one run, which you upload to Your bench. See Upload a job.
Known dataset
A dataset in the site's task catalog, which the maintainers have reviewed. See Known datasets.
Leaderboard
See Board.
Level
How much the site saw of a run: registered, imported or verified. See Self-reported runs and issued runs.
Line
See Configuration.
Local run
See Self-reported.
Maintainers' runs
The maintainers' own runs, checked and ranked like anyone's. On a board, "Reference" marks the row every Δ compares with, not the maintainers' runs. See Self-reported runs and issued runs.
Method version
The version of how the site computes scores, intervals, ranks and figures, v1 since 29 October 2026. See Versions.
Model key
An API key from a model provider, such as DeepSeek or Anthropic. It stays in your shell. See Keep your model key safe.
Named board
A board the site's policy defines, with a spec that sets its time limits, memory and allowlist. See Named and featured boards.
Network
Whether hbench enforced an egress allowlist for the agent: allowlist or open. See Network: allowlist or open.
OCI registry
A container registry, such as ghcr.io, that holds datasets. See Registries.
Private run
A job you uploaded and didn't publish, or didn't upload at all. Only you can see it. See Publish it to its board.
Rank range
The best and worst place a row can hold on its board, from the paired test. See Better than whom?
Registry
Where hbench reads a dataset's tasks from: an OCI registry or a registry.json. See Registries.
Resolution
The size of gap a board can tell apart, in points. See Better than whom?
Results epoch
The hbench releases whose scores compare; a board ranks the runs of its newest epoch. See Versions.
Results reset
See Results epoch.
Reward
What a trial's tests give it, from 0 to 1: usually 1 if they pass and 0 if they don't. See How good?
Run
One harness going through every task of a task set, once per attempt, written as a job.
Run token
A token starting with hpr_ that comes with an issued run and uploads it once within 48 hours. See Limits.
Score
The mean reward over a run's trials, or over a configuration's runs on a board. See How good?
Seal
What hbench 0.3.0 and later do when a job finishes: write manifest.json with the SHA-256 hash of every file, so a later change shows. See Which versions the site accepts.
Self-reported
A result someone ran on their own machine and uploaded. Every result on the site is one. See Self-reported runs and issued runs.
Slug
See Named and featured boards.
Specification
The spec.json the site gives you for an issued run, which the upload is checked against. See Issue a run first.
Submission
See Entry.
Take down
What a maintainer does to take a published score off its board. The run leaves its board and its public page, and Your bench shows it as Taken down. It still counts as uploaded. Every take-down gives one of these reasons:
| Reason | When a maintainer gives it |
|---|---|
takedown-owner-request | The run's owner asked us to remove it. |
takedown-rules | The run breaks a published rule. The note names the rule. |
takedown-integrity | A maintainer reviewed evidence that the result isn't genuine, and upheld it. |
takedown-legal | We received a legal or safety request about the run. |
takedown-mistake | We published the run by mistake. |
The public audit log lists each take-down with its reason, the maintainer's login and a short note. The note is public, so it never names a person. A maintainer can also put a run back on its board, and the log shows that too.
Task
One problem with its own environment, instruction and tests.
Task set
The tasks a run ran from one dataset. See Task sets.
Timed out
A trial whose agent was still working at the time limit. The tests still run, so it can pass, and it ends with AgentTimeoutError. See Errors, timeouts and all-errored runs.
Trial
One try at one task, which its tests give a reward.
Unreviewed dataset
A dataset outside the site's task catalog. See Unreviewed datasets.
Verifier
A task's tests, which run after the agent stops. Your bench shows their output under Tests.
Withdraw
To take your own run's score off its board, for good. See Withdraw a score.
Your bench
Your private workspace: upload jobs, open every trial, and publish them to their boards. See Upload a job.
Reviewed on October 31, 2026.