Reference
hbench command line
The hbench commands a participant uses, the flags that decide a job's board, and every flag of hbench run, read from hbench's own help.
These are the hbench commands and flags you need to run a benchmark and upload it. Every command prints its own help with --help, such as hbench run --help.
Commands you'll use
| Command | What it does |
|---|---|
hbench --version | Prints the installed version. |
hbench run | Runs a job: every task, once per attempt. The same as hbench jobs start. Without --config, --path or --dataset, in a terminal, it asks for the tasks, the agent and the model, then prints the same run as a command. |
hbench jobs resume -p <job-folder> | Continues a job that stopped. Finished trials are kept; cancelled ones run again. |
hbench jobs summary -p <job-folder> | Prints a job's statistics. |
hbench jobs verify -p <job-folder> | Checks a job's files against its manifest.json, and lists any that were changed, removed or added since the run. |
hbench jobs upload --url <url> --token-env <VAR> <job-folder> | Packs a finished job, checks it against its manifest.json and sends it to the site, with the token in <VAR> as a bearer token. -H 'Name: value' adds a header, such as the run token. See Upload from a script. |
hbench view [<folder>] | Serves a results viewer for a folder of jobs, jobs unless you name another, at 127.0.0.1 on the first free port from 8080 to 8089. Open the address it prints in your browser. |
hbench agents list | Lists the built-in agents. |
hbench agents schema <agent> | Prints the options an agent takes, with their types and defaults. Add --json for the JSON Schema. |
hbench datasets list | Lists the datasets in hbench's default registry, oci://ghcr.io/node-q/hbench-datasets. --registry, --registry-url or --registry-path lists another registry's. |
hbench cache clean | Removes hbench's cache in ~/.cache/hbench and the Docker images it built. --dry shows what it would remove; without -f, it asks first. |
hbench update | Installs the latest release with the install script from nodeq.tech, into ~/.local/bin unless HBENCH_INSTALL_DIR is set. |
--dataset name@version reads from oci://ghcr.io/node-q/hbench-datasets unless a registry flag names another; Registries says which names there are other tasks than the known datasets. A spec.json from the site names its board's registry itself.
To keep running the same bytes of an OCI dataset, add --lock-write. It records the dataset's manifest digest in hbench.lock in the current folder, and later runs there use that digest, with a warning if the tag has moved. --locked refuses a dataset that isn't in hbench.lock, or whose tag now names another digest.
hbench --help lists the other commands, for writing tasks and datasets and for analysing jobs.
Flags that change the board
The site reads a job's board from what it ran, so these flags decide which board it goes on. Cells and boards explains each part.
| Flags | What they set |
|---|---|
--dataset, --registry, --registry-url, --registry-path | The dataset, and for an unreviewed one its registry |
--n-tasks, --include-task-name, --exclude-task-name | The task set |
--model | The model |
--n-attempts | The attempts |
--timeout-multiplier and the other timeout multipliers | The time limits |
--override-memory-mb | The memory |
--path runs tasks from a folder on your machine, and its job goes on no board. The network allowlist has no flag: it's set in a job configuration, as in Run with an allowlist. Other flags, such as --agent, --ak, --extra-instruction and --max-retries, change the harness or the run, not the board.
In an issued run
Run a run the site issued from its spec.json, which sets the tasks, the agent, its options and the limits:
Add only --jobs-dir, to write the job elsewhere, or --max-retries, to rerun crashed trials. Any other flag that changes the spec, such as --n-tasks, --ak, --extra-allowed-host or --agent-timeout-multiplier, gets the upload refused. So do --retry-include and --retry-exclude when they let hbench retry a trial that got an answer, such as one that timed out. To read the tasks from a mirror, name it when you issue the run.
Flags of hbench run
Flags override the same settings in --config, and a flag marked repeatable can be given more than once. Three of hbench's own help lines need a note:
--max-retriessays "Retries per failed trial". A failed trial there is one that ended in an error: with hbench's default exclusions, a retry reruns a crash only, never a wrong answer.--aksays "Agent kwarg". It sets an agent option.--datasetsays it needs a registry flag. It doesn't: without one, it reads hbench's default registry.
The flags of hbench run in hbench 0.3.9, read from the binary. Where hbench's own help has no line for a flag, the description is ours.
| Flag | What it does |
|---|---|
-c, --config <CONFIG> | Job config (YAML or JSON). CLI flags override it |
-p, --path <PATH> | Local task dir, or a dir of task dirs |
-d, --dataset <DATASET> | Registry dataset, name[@version], from --registry, --registry-url or --registry-path, else from $HBENCH_REGISTRY, else from oci://ghcr.io/node-q/hbench-datasets |
--registry-url <REGISTRY_URL> | Dataset registry URL (a registry.json), instead of the default OCI registry |
--registry-path <REGISTRY_PATH> | Local dataset registry file (a registry.json), used instead of --registry-url |
--registry <REGISTRY> | OCI dataset registry, oci://<registry>/<namespace>, or OCI image layouts on disk, oci-layout:<dir>. Default: $HBENCH_REGISTRY, else oci://ghcr.io/node-q/hbench-datasets |
--locked | Refuse an OCI dataset that is not in ./hbench.lock, or whose tag now resolves to another digest |
--lock-write | Record each OCI dataset's manifest digest in ./hbench.lock |
-i, --include-task-name <INCLUDE>repeatable | Only tasks matching this glob |
-x, --exclude-task-name <EXCLUDE>repeatable | Skip tasks matching this glob |
-l, --n-tasks <N_TASKS> | Run at most N tasks |
-k, --n-attempts <N_ATTEMPTS> | Attempts per task |
--diff <DIFF>repeatable | Compare against earlier job(s): reuse / regrade / rerun |
-y, --yes | Don't ask for confirmation (with --diff) |
-a, --agent <AGENT> | Agent name (see hbench agents list) |
-m, --model <MODELS>repeatable | Model(s); one agent config per model |
--ak <AGENT_KWARGS>repeatable | Agent kwarg KEY=VALUE (JSON values allowed) |
--ae <AGENT_ENV>repeatable | Agent env var KEY=VALUE |
--n-concurrent-agent <N_CONCURRENT_AGENT> | Per-agent cap on concurrent agent runs |
--model-served-by <MODEL_SERVED_BY> | What served the model, recorded with each trial (e.g. lmstudio, vllm, openrouter:together) |
--model-quantization <MODEL_QUANTIZATION> | The served model's quantization, recorded with each trial (e.g. q4_k_m, fp8) |
--skill <SKILL>repeatable | Skill dir uploaded to /hbench/skills |
--extra-instruction <EXTRA_INSTRUCTION>repeatable | Text appended to every task instruction |
--extra-instruction-path <EXTRA_INSTRUCTION_PATH>repeatable | File whose contents are appended to every task instruction |
--install-only | Only install the agent; don't run it |
--skip-preflight | Don't check that agent credentials are set before starting |
--ee <ENV_ENV>repeatable | Environment (container) env var KEY=VALUE |
--ve <VERIFIER_ENV>repeatable | Verifier env var KEY=VALUE |
-n, --n-concurrent <N_CONCURRENT> | Concurrent trials |
--force-build | Rebuild task images even if cached |
--no-delete | Keep images/volumes after trials |
--keep-containers | Stop (don't remove) containers after trials |
--disable-verification | Skip the verifier (no rewards) |
--override-cpus <OVERRIDE_CPUS> | Override each task's CPU count |
--override-memory-mb <OVERRIDE_MEMORY_MB> | Override each task's memory (MB) |
--override-gpus <OVERRIDE_GPUS> | Override each task's GPU count |
--extra-allowed-host <EXTRA_ALLOWED_HOST>repeatable | Extra host allowed during network-restricted tasks |
--timeout-multiplier <TIMEOUT_MULTIPLIER> | Scale every timeout |
--agent-timeout-multiplier <AGENT_TIMEOUT_MULTIPLIER> | Scale the agent run timeout |
--verifier-timeout-multiplier <VERIFIER_TIMEOUT_MULTIPLIER> | Scale the verifier timeout |
--agent-setup-timeout-multiplier <AGENT_SETUP_TIMEOUT_MULTIPLIER> | Scale the agent setup (install) timeout |
--environment-build-timeout-multiplier <ENVIRONMENT_BUILD_TIMEOUT_MULTIPLIER> | Scale the image build timeout |
--max-retries <MAX_RETRIES> | Retries per failed trial |
--retry-include <RETRY_INCLUDE>repeatable | Retry only these exception types |
--retry-exclude <RETRY_EXCLUDE>repeatable | Never retry these exception types |
--jobs-dir <JOBS_DIR> | Where job dirs are created (default: jobs) |
--job-name <JOB_NAME> | Job dir name (default: timestamp) |
--artifact <ARTIFACT>repeatable | Artifact path to collect |
--quiet | No progress: no download bar, no per-trial lines |
-o, --observe | Print every trial's log and command output as it runs, each line under the trial's name |
--fail-on-error | Exit 1 if any trial errored (default: only when every trial errored) |
Variables hbench reads
| Variable | What it does |
|---|---|
DOCKER_HOST, DOCKER_CONTEXT | Which Docker daemon hbench uses, as for the docker command. |
HBENCH_REGISTRY | The registry a dataset comes from when nothing names one, written as for --registry. Without it, that's oci://ghcr.io/node-q/hbench-datasets. |
HBENCH_NO_UPDATE_CHECK | Set to 1 to turn off the notice that a newer release is out. hbench checks at most once a day, and only when it runs in a terminal. |
OPENAI_BASE_URL, OPENAI_API_KEY | Where an agent finds an OpenAI-compatible model server, and the key it sends. For a local server, any key works, such as local. |
DEEPSEEK_API_KEY and other provider keys | The key an agent sends to its model provider. It's read from your shell and never leaves your machine, except to the provider. |
--ae NAME=value passes a variable to the agent instead. hbench never writes the value of a variable whose name looks secret into the job: see How keys are redacted.
Reviewed on October 29, 2026.