Documentation

Harness Playground docs

Run any benchmark on your own machine with hbench, upload the job, and see where your harness stands on its board.

Run a benchmark and upload itInstall hbench, run a benchmark in Docker, and upload the job. The site puts it on the board of the runs that ran exactly the same thing.
Understand the boardsWhat a board holds fixed, what a score measures, what the site checks on upload, and what it can't.
Fix an errorEvery problem we know of, with the message you see, why it happens and what to do.

Every page

Get started

Run a benchmark and upload it: Run HumanEvalFix on Qwen2.5-Coder 7B on your own machine with hbench, upload the job, and see your harness on the board. No model key needed.
Read your result and your board: What your score and each trial's outcome mean, how to find out why a trial failed, and how to read your row on the board.

How-to guides

Upload a job: Upload an hbench job folder or a .tar.gz of it, up to 95 MiB, in the browser or with curl, and publish it to its board.
Choose a benchmark, a registry and a model: Pick the benchmark, its registry, the task set, the model and the attempts for a run, to join a board that already has runs or to start a new one.
Run your own harness or ACP agent: Run a built-in agent with your own settings, an agent from the ACP registry, or your own code from a git repository, and put its job on a board beside the built-in agents.
Issue a run first: Have the site issue a run before you start it, so it knows of the run and checks the upload against its spec, from the run builder or from a script.
Withdraw, delete or report a run: Take your score off its board, delete a job from Your bench, or ask the maintainers to remove a run.
Use GitHub Actions: Run hbench in GitHub Actions with node-q/hbench-action, on every pull request or on a schedule, and upload or issue its runs.
Run on a Mac, Linux or WSL2: What changes between Docker Desktop on a Mac, Docker Engine on Linux, WSL2 on Windows and a GitHub Actions runner, from the model server's address to disk space.
Use a local model server: Serve Qwen2.5-Coder 7B, or any model, with ollama, LM Studio, vLLM or llama.cpp so the task containers can reach it.
Install a specific hbench version: Which hbench versions the site accepts, how to install a particular release, keep versions side by side, and check which one runs.

Reference

Cells and boards: The eight things a cell holds fixed, what makes two runs comparable, how the site finds a job's board, how a board ranks, and why most boards hold one harness.
Datasets and registries: The five known datasets and their task sets, how the site tells datasets apart by their tasks, what an unreviewed dataset is, and the registries hbench reads.
The dataset checklist: What makes a dataset reviewed, the Agentic Benchmark Checklist items each assessment answers, how they are scored, and what hbench and the site answer for every dataset.
Terminal-Bench 2.0: checklist assessment: How Terminal-Bench 2.0 measures up to the Agentic Benchmark Checklist, item by item, as the maintainers assessed it. 89 tasks an agent carries out in a Linux container, each checked by its own tests.
SWE-bench Verified: checklist assessment: How SWE-bench Verified measures up to the Agentic Benchmark Checklist, item by item, as the maintainers assessed it. 500 real GitHub issues from Python projects, each judged by the project's own tests.
HumanEvalFix: checklist assessment: How HumanEvalFix measures up to the Agentic Benchmark Checklist, item by item, as the maintainers assessed it. 164 Python functions with a planted bug, each graded by HumanEval's tests.
Aider Polyglot: checklist assessment: How Aider Polyglot measures up to the Agentic Benchmark Checklist, item by item, as the maintainers assessed it. 225 Exercism exercises in six languages, each graded by Exercism's tests.
LiveCodeBench v6: checklist assessment: How LiveCodeBench v6 measures up to the Agentic Benchmark Checklist, item by item, as the maintainers assessed it. 100 contest problems from release v6, each graded by hidden tests.
Network: allowlist or open: What the network part of a cell means, how hbench enforces an egress allowlist for the agent, why allowlist and open runs never share a board, and how to run with an allowlist.
The deny list: What the site never accepts, such as the oracle agent and options that swap the model, and what the maintainers curate for readers instead.
Agents: hbench's built-in agents, the eight the site issues runs of, every option you can set, the options a board holds, models and keys, and prompt templates.
hbench command line: The hbench commands a participant uses, the flags that decide a job's board, and every flag of hbench run, read from hbench's own help.
Limits: Storage, upload size, retention, attempts per task, issued runs a day, entries per board, token lifetimes and rate limits, per GitHub account.
Privacy and redaction: What stays private on Your bench, what publishing a run makes public, how to make its trajectories public, what the site keeps when you delete a job, how hbench and the site redact keys, and how to keep your model key safe.
Glossary: Each word Harness Playground uses, in one line, with a link to the page that explains it.

Troubleshooting

Troubleshooting: Each problem we know of in installing hbench, running a benchmark, serving the model and uploading a job, with what you see, why it happens and what to do.

The FAQ answers short questions, and the methodology describes how boards are measured.