Skip to content
Hbench
hbench

Get started

Run a benchmark and upload it

Run HumanEvalFix on Qwen2.5-Coder 7B on your own machine with hbench, upload the job, and see your harness on the board. No model key needed.

By the end of this page your harness has a score on the board HumanEvalFix on Qwen2.5-Coder 7B. The model runs on your machine, so nobody pays for tokens. A run takes up to about 3½ hours, by our estimate from the tasks' time limits.

Before you start

You need:

  • A GitHub account, to sign in.

  • Linux or macOS with Docker: Docker Engine with the Compose plugin on Linux, Docker Desktop on a Mac. On Windows, use WSL2.

  • About 10 GB of free disk: 1.6 GB for the 40 task images, 4.7 GB for the model.

  • ollama or another OpenAI-compatible model server. We estimate a GPU with 8 GB of memory and 16 GB of RAM; a CPU alone works, more slowly.

Install hbench

hbench is the benchmark runner. The install script downloads the latest release from node-q/hbench, checks it against the release's SHA256SUMS, and puts it in ~/.local/bin.

bash
curl -fsSL https://nodeq.tech/install.sh | sh

If the script prints add … to your PATH, add that folder to your PATH and open a new terminal. Then check the version:

bash
hbench --version

The site takes jobs from hbench 0.3.0 or later. To install a particular release, see Install a specific hbench version.

Read the script before you run it

nodeq.tech/install.sh redirects to the install script of hbench's latest release on GitHub. To read the script first, download it from there, read it, then run it:

bash
curl -fsSL -o install.sh https://github.com/node-q/hbench/releases/latest/download/install.sh
less install.sh
sh install.sh

Serve the model

The task containers reach your machine at host.docker.internal on Docker Desktop, or at 172.17.0.1 on Linux. This board allows no other address.

  1. Download the model:

    bash
    ollama pull qwen2.5-coder:7b

  2. Start ollama. On a Mac, start the ollama app. On Linux, bind it to Docker's bridge address:

    bash
    OLLAMA_HOST=172.17.0.1:11434 ollama serve

  3. Check that a container can see the model. On a Mac:

    bash
    docker run --rm curlimages/curl:8.10.1 -s http://host.docker.internal:11434/v1/models

    The output must include qwen2.5-coder:7b. On Linux, check from a new Docker network instead, as in Check from a container's network.

The name must be exact: qwen2.5:7b is another model. For LM Studio, or ollama as a service, see Use a local model server.

Issue the run

  1. Sign in with GitHub and open Run a benchmark.

  2. Under Board, pick HumanEvalFix on Qwen2.5-Coder 7B.

  3. Under Agent, pick one, such as mini-swe-agent. On a Mac with Apple silicon, avoid openhands, which doesn't install there.

  4. Press Issue the run. The site names your entry after the agent and its options.

  5. Copy the run token, which starts with hpr_. The page shows it once.

You now have 48 hours to upload the job. Issuing tells the site about your run before it starts, which makes your result easier to trust: see Issue a run first.

Save the run's files

The page's Run it command starts by making a folder for the run, such as hpg-run-412, and going into it. It then writes spec.json there, and prompt.j2 if your harness has a prompt template. Don't edit either file: the site checks your job against them.

Run it

Above the Run it command, pick Linux or Docker Desktop, and the model server address in the command follows. Copy the command and run it. Its last line runs hbench; on Docker Desktop it looks like this:

bash
OPENAI_BASE_URL=http://host.docker.internal:11434/v1 OPENAI_API_KEY=local hbench run --config spec.json

hbench runs each task in its own container and writes the job to jobs/<job-name>/. Add no flags except --jobs-dir or --max-retries. Any flag that changes the spec, such as --n-tasks, gets the upload refused.

If the run stops, for example when your machine sleeps, continue it:

bash
hbench jobs resume -p jobs/<job-name>

Look at the results

Print a summary:

bash
hbench jobs summary -p jobs/<job-name>

If every trial errored, the containers couldn't reach the model. Fix the server, move the job's folder out of jobs/, and run the same command again; the run token stays valid for its 48 hours. hbench view jobs opens a viewer in your browser, on 127.0.0.1 from port 8080.

Pack the job

The browser can upload the folder as it is. To keep one file, or to upload from a script, pack it:

bash
tar czf job.tgz -C jobs <job-name>

Upload the job

Set HPG_RUN_TOKEN to the run token and run the page's Upload the job command, which packs the job and uploads it. Or upload it in the browser:

  1. Open Your bench.

  2. Under Upload a job, paste the run token into Run token.

  3. Drop the job's folder or job.tgz on the upload area.

The site replaces values that look like keys with [REDACTED] and checks the job. If it refuses the job, the message ends with a rule code in brackets, which Troubleshooting explains.

The job stays private until you publish it. Open it on Your bench and press Publish, then Make it public?. Its board then shows how many of the configuration's issued runs are public, so a result picked from several runs shows as one public of several.

A job you didn't issue

You can skip issuing and upload any job you ran. The site keeps it private on Your bench until you open it and press Publish. Upload a job has the steps.

Check that it's listed

Your job's page on Your bench names its board. On the board, your run is in the row of its configuration: the agent with its options, prompt and version. A row shows a rank range once a second configuration has a score. Read your result and your board explains the columns.

Reviewed on October 31, 2026.