Skip to content
Hbench
hbench

How-to guides

Choose a benchmark, a registry and a model

Pick the benchmark, its registry, the task set, the model and the attempts for a run, to join a board that already has runs or to start a new one.

Your choices decide which board your job goes on. To compare your harness with others, run what a board with runs already holds. To try something nobody has run, choose anything: your job starts a new board.

Join a board that has runs

  1. Pick a board on Benchmarks, such as SWE-bench Verified on Claude Opus 5.5.

  2. On Run a benchmark, pick the same board.

  3. Issue the run and run its spec.json unchanged, as in Issue a run first.

The spec sets the board's tasks, model and attempts, and also its time limits, memory and network allowlist. A job with any other settings goes on another board.

Pick the benchmark

Name a benchmark as name@version with --dataset, or -d:

bash
hbench run -d terminal-bench@2.0 -a mini-swe-agent -m deepseek/deepseek-flash

Five benchmarks are known: the maintainers reviewed their tasks, and only boards on them are featured. Any other dataset gets a board labelled unreviewed. Datasets and registries lists the five.

Name the registry

A registry is where hbench gets the tasks. Without a flag, hbench reads oci://ghcr.io/node-q/hbench-datasets. Name another with --registry oci://…, --registry-url <url> or --registry-path <file>.

The known HumanEvalFix and SWE-bench Verified come from a JSON registry, and the default registry's datasets of those names are other tasks. Name the JSON registry to run the known ones:

bash
hbench run -d humanevalfix@1.0 -a mini-swe-agent -m openai/qwen2.5-coder:7b \
  --registry-url https://raw.githubusercontent.com/laude-institute/harbor/main/registry.json

Registries says which names differ, and how the site tells datasets apart.

Pick the task set

hbench runs every task by default. Some benchmarks have named subsets, such as SWE-bench Verified's sample-50; Task sets lists them. Any other selection, made with --n-tasks, --include-task-name or --exclude-task-name, is a task set of its own, shared only by jobs that ran exactly the same tasks.

Name the model

Name the model with -m as provider/model, such as deepseek/deepseek-flash, anthropic/claude-opus-5-5, or openai/qwen2.5-coder:7b for a model on an OpenAI-compatible server. The board's model is the name your job's lock records without its provider, the part before the first /: openai/gpt-6-sol and gpt-6-sol join one board. Any other spelling starts another board.

The agent reads the provider's key from your shell; Models and keys lists the variables. For a model on your machine, see Use a local model server.

Set attempts, limits and network

Each of these is part of the board, so leave them at hbench's defaults unless the board you want to join sets them:

  • Attempts: --n-attempts, or -k, from 1 to 5. A job with 3 attempts goes on another board than one with 1.

  • Time limits: --timeout-multiplier and the other timeout multipliers.

  • Memory: --override-memory-mb.

  • Network: a plain hbench run doesn't enforce an egress allowlist, so its job goes on a board labelled open network. Network: allowlist or open shows how to run with one.

Try a few tasks first

Run --n-tasks 3 to check that the agent installs and reaches the model, then run the full choice. Before you upload, hbench jobs summary -p jobs/<job-name> shows whether any trials errored.

Reviewed on October 30, 2026.