How-to guides
Run on a Mac, Linux or WSL2
What changes between Docker Desktop on a Mac, Docker Engine on Linux, WSL2 on Windows and a GitHub Actions runner, from the model server's address to disk space.
hbench runs each task in a Docker container, so it behaves the same wherever docker points. Four things differ by platform: the address of a model on your machine, the room Docker has, the CPU architecture, and Windows.
Check that Docker works
hbench needs the Docker daemon and the Compose plugin, version 2 or later. Check both:
hbench uses whichever daemon docker talks to, including one set by DOCKER_CONTEXT or DOCKER_HOST. On Linux, add yourself to the docker group, then log in again:
Point the containers at the model
The free boards' allowlist holds two addresses for a model server on your machine:
| Docker | Model server address | Start ollama with |
|---|---|---|
| Docker Desktop | host.docker.internal:11434 | its defaults |
| Linux | 172.17.0.1:11434 | OLLAMA_HOST=172.17.0.1:11434 ollama serve |
On Linux, host.docker.internal doesn't resolve inside the containers, and a server on 127.0.0.1 is out of their reach. 172.17.0.1 is Docker's default bridge address. Rootless Docker, or a changed bridge network, has no 172.17.0.1, so those setups can't reach a model under the free boards' allowlist. On Docker Desktop, 172.17.0.1 doesn't reach the Mac.
In Run a benchmark, pick Linux or Docker Desktop above the Run it command, and the command uses the right address. Use a local model server covers each server and the firewall.
Give Docker enough room
The free boards give each task container 2 CPUs and 2 GB of memory, and run two tasks at a time. HumanEvalFix's images need about 2 GB of disk. Terminal-Bench images take 1 to 3 GB each, so keep about 30 GB free for the 10 tasks of Terminal-Bench 2.0 on Qwen2.5-Coder 7B.
On Docker Desktop these limits belong to its virtual machine. Under Settings, then Resources, give it at least 4 CPUs and 6 GB of memory, by our estimate, and raise the disk limit if the images don't fit.
Free the space afterwards. --dry lists what would go; the second command asks before it removes anything:
Apple silicon
hbench has native builds for Apple silicon and Intel Macs. Many task images are x86_64, though, and Docker Desktop emulates them more slowly, so more trials hit their time limits.
In our test on Apple silicon, codex, vibe, trae-agent, openhands and hermes failed to install, with AgentSetupError on every trial. Pick another agent on a Mac.
Linux on arm64
hbench has a Linux arm64 build. x86_64 task images need qemu and binfmt support there, or they fail with exec format error. We haven't run a full benchmark on Linux arm64.
Windows
hbench has no Windows build. Run it inside WSL2: install the Linux build in your WSL2 distribution, with Docker Engine there or Docker Desktop's WSL2 backend. In the run builder, pick Linux for Docker Engine, or Docker Desktop for Docker Desktop's backend. We haven't run a full benchmark under WSL2 yet.
GitHub Actions
GitHub's ubuntu-* runners have Docker, so the Linux notes above apply. Use GitHub Actions has the workflows and the runners' 6-hour limit.
Reviewed on October 30, 2026.