Reference

Datasets and registries

The five known datasets and their task sets, how the site tells datasets apart by their tasks, what an unreviewed dataset is, and the registries hbench reads.

Any dataset that hbench reads from a registry can go on a board. Five are known: the maintainers reviewed their tasks. Every other dataset is labelled unreviewed.

Dataset identity#

Anyone can publish a registry, and two registries can ship a humanevalfix@1.0 with different tasks. So the site tells datasets apart by their tasks, using the digest each trial's lock.json records for its task:

• A job is of a known dataset when its name@version is in the task catalog and every task it ran has the catalogued task's digest. The registry doesn't matter then: a mirror with the same tasks is the same dataset.

• Any other job's dataset is identified by its name@version, a digest over the digests of the tasks it ran, and the registry its config.json names. Two such jobs are of one dataset only when all three match.

Only jobs of one dataset share a board. Two versions whose tasks differ only in the image they name, where the images are identical by digest, count as one: terminal-bench@2.0 from the JSON registry below and terminal-bench@2.0.1 from oci://ghcr.io/node-q/hbench-datasets are both Terminal-Bench 2.0.

Known datasets#

A known dataset is in the site's task catalog, with a title, a description and a quick set, and only boards on known datasets can be featured. All five come from https://raw.githubusercontent.com/laude-institute/harbor/main/registry.json:

The site checks every trial of a known dataset against the catalogued task's checksum. None of these boards is the benchmark's official leaderboard, and we aren't affiliated with the benchmarks' authors.

Reviewed datasets#

Each known dataset has an assessment against the Agentic Benchmark Checklist, item by item, as the maintainers made it; the dataset checklist says what the items are and how they are scored.

• Terminal-Bench 2.0

• SWE-bench Verified

• HumanEvalFix

• Aider Polyglot

• LiveCodeBench v6

Unreviewed datasets#

Any other dataset is shown by its name@version and its registry's host, such as humanevalfix@1.0 (ghcr.io), and labelled unreviewed dataset. The site checks its jobs like any other, but nobody has read its tasks: they could be easier, broken, or copies with the answers in them. So its board says nothing about the benchmark it's named after, and it's never featured. If one imitates a known dataset to mislead, report it.

Task sets#

A board's task set is one of three kinds:

• the full set: every task of the dataset;

• a named subset in the task catalog, such as sample-50, listed by task name so the registry's order never moves it;

• any other selection, made with --n-tasks, --include-task-name or --exclude-task-name, identified by its sorted task names and shown as tasks- and 8 hex digits.

Each known dataset also has a quick set of 20 tasks: 10 that every quick run includes and 10 drawn for each run. Quick sets can't be run yet. A score on SWE-bench Verified's 50 tasks isn't comparable with one on all 500.

Registries#

hbench reads two kinds of registry:

• An OCI registry with --registry oci://…. Without a flag, and without HBENCH_REGISTRY, hbench reads oci://ghcr.io/node-q/hbench-datasets.

• A JSON registry, a registry.json, with --registry-url <url> or --registry-path <file>. The known datasets come from one.

The default OCI registry's humanevalfix@1.0 and swebench-verified@1.0 are other tasks than the known datasets of those names: node-q built them from the benchmarks' own data, and its swebench-verified@1.0 has 456 tasks, those whose repository's licence allows it. A run of either without a registry flag goes on an unreviewed board. Its terminal-bench@2.0.1 is the known Terminal-Bench 2.0.

hbench datasets list lists a registry's datasets. To keep running the same bytes of an OCI dataset, use --lock-write, as in hbench command line.

Reviewed on October 29, 2026.