Skip to content
Hbench
hbench

How-to guides

Issue a run first

Have the site issue a run before you start it, so it knows of the run and checks the upload against its spec, from the run builder or from a script.

Issue a run when you want it to count on its board whether or not you publish it. The board counts an issued run once it's uploaded or expires, and its row shows how many issued runs are public, so readers can tell one published run from the highest-scoring of several. The site also checks your upload against what it issued. Issuing is optional: Self-reported runs and issued runs says what each way shows.

What you can issue

The site issues runs of what its policy lists: the known benchmarks with their full sets and named subsets, the models it names, the agents it lists for each model, and 1 or 3 attempts per task. That covers the named boards, such as HumanEvalFix on Qwen2.5-Coder 7B. Anything else, such as another model or an unreviewed dataset, can't be issued: upload the job instead.

Issue it on the site

  1. Sign in with GitHub and open Run a benchmark.

  2. Pick the board under Board.

  3. Pick an agent, and its settings under More options if you want any. The site files the run under your entry for that agent and those settings, and names it, such as terminus-2-a3f9c1. Another run of the same settings, in any version of the agent, joins the same entry; other settings start a new one.

  4. Press Issue the run, and copy the run token (hpr_…). The page shows it once.

  5. Run the page's Run it command. It makes a folder for the run, writes spec.json there, and prompt.j2 if there is one, and runs hbench run --config spec.json with the variables the board needs. Don't edit the files.

  6. Set HPG_RUN_TOKEN to the run token and run the page's Upload the job command within 48 hours, or upload the job another way. The upload needs no other token.

Add no flag to hbench run except --jobs-dir or --max-retries. Any other flag changes the spec, and the site refuses the upload with local-spec-mismatch. If the run stops, continue it with hbench jobs resume -p jobs/<job-name>.

Know what it counts

An issued run counts from the moment you issue it, uploaded or not. Each board limits how many runs you can be issued in 24 hours and how many can wait for upload at once: Limits has the numbers.

If you can't finish a run, abandon it: on Your bench, press Abandon beside the run, then Abandon this run?. It counts as expired and frees its slot at once.

Issue it from a script

Send POST https://nodeq.tech/api/local-runs with an API token as authorization: Bearer hpg_... and a JSON body:

FieldWhat it holds
arenaThe last part of the board's address, such as livecodebench-v6-deepseek-flash. Leave it out to choose with the next four fields instead.
dataset, task_set, model, n_attemptsThe benchmark as name@version, its task set (default full), the model, and the attempts per task (default 1).
declarative.agentThe agent, such as mini-swe-agent.
declarative.kwargsOptional. The agent's options, by name, as on Agents.
declarative.prompt_templateOptional. A prompt template, up to 20,000 characters.
declarative.extra_instructionsOptional. Up to 8 passages, each up to 4,000 characters.
registryOptional. {"url": "https://..."} or {"path": "..."}, to read the tasks from a mirror.
hardwareOptional. Up to 8 short fields about your machine, such as {"cpu": "...", "gpu": "..."}. The site's pages don't show it yet.

Leave the entry's name out: the site names it, and refuses a body with name with entry-name-generated. The answer's submission is the name it gave.

Write the answer's spec to spec.json, and its prompt_template, when it isn't null, to prompt.j2. token is the run token, sent only in this answer, and the only token the upload needs; expires_at is when it stops working; environment holds the variables the run needs besides your key; job_name is the folder hbench writes. A refusal comes back with a detail message, as listed in Troubleshooting.

Two more calls help a script:

  • GET https://nodeq.tech/api/local-runs lists your issued runs, newest first.

  • POST https://nodeq.tech/api/local-runs/<run_id>/abandon abandons one.

Use GitHub Actions has a workflow that issues, runs and uploads a run.

Reviewed on October 30, 2026.