Skip to content
Hbench
hbench

How-to guides

Use GitHub Actions

Run hbench in GitHub Actions with node-q/hbench-action, on every pull request or on a schedule, and upload or issue its runs.

Run hbench on every pull request with node-q/hbench-action. The action installs hbench, checks its checksum and build provenance, runs it, and writes the scores to the job summary. Fork node-q/hbench-action-example for a repository with three tasks and working workflows; the action's README lists every input and output.

GitHub-hosted runners stop a job after 6 hours, so a longer run, such as SWE-bench Verified on Claude Opus 5.5 at about 5 to 7 hours, needs a self-hosted runner. A model served on the runner, such as Qwen2.5-Coder 7B, is slow without a GPU.

Run on each pull request

This workflow runs Claude Code on five tasks of Terminal-Bench 2.0, with the key from a repository secret. The job stays in the workflow: nothing goes on a board. Your provider bills your key for every trial.

yaml
name: Bench
on: [pull_request]
jobs:
  bench:
    runs-on: ubuntu-24.04
    steps:
      - uses: node-q/hbench-action@v1
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        with:
          dataset: terminal-bench@2.0.1
          registry: oci://ghcr.io/node-q/hbench-datasets
          agent: claude-code
          model: anthropic/claude-opus-5-5
          n-tasks: 5

To try a change on a known benchmark's tasks, set dataset and registry to the ones in Known datasets. Pull requests from forks get no secrets, so they can't call a paid model. Pin an hbench release with version, such as v0.3.5.

Upload a run from a workflow

To put a workflow's run on a board, run the full task set on your default branch or on a schedule, then upload the job. Add an API token as the secret HPG_API_TOKEN, give the action's step id: bench, and add this step after it:

yaml
      - name: Upload the job
        env:
          HPG_API_TOKEN: ${{ secrets.HPG_API_TOKEN }}
          JOB_DIR: ${{ steps.bench.outputs.job-dir }}
        run: |
          tar czf job.tgz -C "$(dirname "$JOB_DIR")" "$(basename "$JOB_DIR")"
          curl --fail-with-body -sS --data-binary @job.tgz \
            -H 'content-type: application/gzip' \
            -H "authorization: Bearer $HPG_API_TOKEN" \
            https://nodeq.tech/api/bench/uploads

Then publish the job from Your bench.

Issue a run from a workflow

Issue runs from a workflow you start by hand or on a schedule, not on every pull request: each board issues an account a few runs a day and holds one waiting for upload at a time (Limits). The workflow does four things:

  1. Issue the run through the API, as in Issue it from a script.

  2. Install hbench with the action, with no path, dataset or config, so that it only installs.

  3. Run hbench run --config spec.json in a step of your own. The spec sets the job name the site checks, and the jobs folder hbench writes the job to.

  4. Pack the job and upload it with the run token, the only token the upload needs.

Don't pass spec.json to the action's config input. The action then names the job itself, and the site refuses the upload with local-spec-mismatch: job_name is 'x', the spec's is 'y'.

Add your model key as a second secret. This workflow runs mini-swe-agent on LiveCodeBench v6 on DeepSeek-Flash, which takes about 1 to 2 hours:

yaml
name: Issued run
on:
  workflow_dispatch:
  schedule:
    - cron: "0 3 * * 1"
jobs:
  run:
    runs-on: ubuntu-24.04
    timeout-minutes: 300
    steps:
      - name: Issue the run
        env:
          HPG_API_TOKEN: ${{ secrets.HPG_API_TOKEN }}
        run: |
          curl --fail-with-body -sS https://nodeq.tech/api/local-runs \
            -H "authorization: Bearer $HPG_API_TOKEN" \
            -H 'content-type: application/json' \
            -d '{"arena": "livecodebench-v6-deepseek-flash", "declarative": {"agent": "mini-swe-agent"}}' \
            > issued.json
          jq .spec issued.json > spec.json
          jq -r '.prompt_template // empty' issued.json > prompt.j2
          echo "::add-mask::$(jq -r .token issued.json)"
          echo "HPG_RUN_TOKEN=$(jq -r .token issued.json)" >> "$GITHUB_ENV"
          echo "JOB_NAME=$(jq -r .job_name issued.json)" >> "$GITHUB_ENV"
          jq -r '.environment | to_entries[] | "\(.key)=\(.value)"' issued.json >> "$GITHUB_ENV"
      - uses: node-q/hbench-action@v1
      - name: Run it
        env:
          DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
        run: hbench run --config spec.json
      - name: Upload the job
        run: |
          tar czf job.tgz -C jobs "$JOB_NAME"
          curl --fail-with-body -sS --data-binary @job.tgz \
            -H 'content-type: application/gzip' \
            -H "hpg-run-token: $HPG_RUN_TOKEN" \
            https://nodeq.tech/api/bench/uploads

The run goes on its board when the upload is accepted. If the job fails part way, the run waits for upload until it expires after 48 hours; abandon it to issue another sooner.

Reviewed on October 30, 2026.