Skip to content
Hbench
hbench

How-to guides

Run your own harness or ACP agent

Run a built-in agent with your own settings, an agent from the ACP registry, or your own code from a git repository, and put its job on a board beside the built-in agents.

Any harness hbench can run can go on a board: a built-in agent with your options and prompt, an agent from the ACP registry, or your own code. Your code runs on your machine, never on ours, so nobody reviews it first. The site checks its job like any other.

Tune a built-in agent

Agents lists hbench's built-in agents and their options. Set an option with --ak NAME=VALUE, where the value can be JSON:

bash
hbench agents schema mini-swe-agent
hbench run -d terminal-bench@2.0 -a mini-swe-agent -m deepseek/deepseek-flash --ak max_tokens=4096

The agent, its options, its prompt template and its extra instructions make up your configuration, with the version of the agent that ran. All runs of one configuration share a row on a board, whoever ran them, and a new version gets a row of its own.

A prompt template or extra instructions need an issued run: the site refuses to publish a job that used either, because it can't record them from the job alone. Issue the run on a named board to put a prompt on it.

Run an agent that speaks ACP

Your own agent has to speak the Agent Client Protocol (ACP) over stdin and stdout. hbench's acp agent installs it in each task's container, gives it the task as one prompt, and asks it for the model you name with -m. hbench finds the agent in one of two places:

  • The ACP registry. -a acp:<id> runs the agent listed as <id> in the ACP registry; -a acp:<id>@<version> pins a version.

  • A git repository. The source option names the repository and a commit. The repository holds a Python 3.12 project managed by uv, with pyproject.toml, uv.lock and hbench-agent.json at its root.

Package your code

Write hbench-agent.json to say how to start the agent:

json
{
  "schema_version": 1,
  "id": "my-agent",
  "version": "0.1.0",
  "protocol": "acp",
  "runtime": { "kind": "python-uv", "python": "3.12", "entrypoint": ["my-agent"] }
}

entrypoint is the command that starts your agent in the project's virtual environment, for example from [project.scripts] in pyproject.toml. Try it on three HumanEvalFix tasks:

bash
hbench run -d humanevalfix@1.0 -l 3 -a acp -m deepseek/deepseek-flash \
  --ak 'source={"repo_url": "https://github.com/you/my-agent", "ref": "<commit>"}' \
  --ae DEEPSEEK_API_KEY="$DEEPSEEK_API_KEY"

  • Use a commit as ref, so every run installs the same code. The job records the repository and the commit, and a run of another commit is another row on the board.

  • Pass the variables your agent needs with --ae. hbench records those whose names look secret as masked.

  • For a private repository on github.com, set HBENCH_ACP_SOURCE_GIT_TOKEN to a token that can read it. A board shows the repository's address and commit, not its code.

  • Keep the repository under 10,000 files and 100 MB, with no symlinks.

  • Run hbench agents schema acp for every option.

Put it on a board

Three tasks check that the agent works. For a score, run a full task set of a known benchmark and upload the job. The board compares your agent with every other harness that ran the same tasks on the same model.

No agent may set an option that swaps the model, and the oracle agent is never accepted: see The deny list. To run your agent on every pull request, see Use GitHub Actions.

Reviewed on October 24, 2026.