Skip to content
Hbench
hbench

How-to guides

Use a local model server

Serve Qwen2.5-Coder 7B, or any model, with ollama, LM Studio, vLLM or llama.cpp so the task containers can reach it.

Serve the free boards' model, Qwen2.5-Coder 7B, with any OpenAI-compatible server that meets three conditions:

  • It answers under the exact name in the spec, qwen2.5-coder:7b. Under any other name, every trial ends with ModelNotFoundError.

  • The task containers reach it at host.docker.internal (Docker Desktop) or 172.17.0.1 (Linux). The free boards' allowlist blocks every other address while the agent works.

  • It answers at http://<address>:<port>/v1.

The site reads the model name each trial recorded, and can't see which model answered under it. Serve the model the name says.

ollama on a Mac

  1. Pull the model:

    bash
    ollama pull qwen2.5-coder:7b

  2. Start the ollama app, or run ollama serve. Its default address works with Docker Desktop.

  3. Check that a container can reach it:

    bash
    docker run --rm curlimages/curl:8.10.1 -s http://host.docker.internal:11434/v1/models

Use host.docker.internal:11434 as the model server address.

ollama on Linux

ollama listens on 127.0.0.1 by default, which containers can't reach. Bind it to Docker's bridge address:

bash
OLLAMA_HOST=172.17.0.1:11434 ollama serve

If ollama's installer set it up as a service, that service already holds port 11434. Set the address in the service instead:

bash
sudo systemctl edit ollama

Add these lines, save, and restart the service:

[Unit]
After=docker.service

[Service]
Environment="OLLAMA_HOST=172.17.0.1:11434"
bash
sudo systemctl restart ollama
OLLAMA_HOST=172.17.0.1:11434 ollama pull qwen2.5-coder:7b

ollama starts after Docker because the bridge address exists only while Docker runs. From then on, the ollama command needs OLLAMA_HOST=172.17.0.1:11434 too, as above. Use 172.17.0.1:11434 as the model server address.

Check from a container's network

hbench puts each trial on a Docker network of its own, so check from one:

bash
docker network create hbench-check
docker run --rm --network hbench-check curlimages/curl:8.10.1 -s http://172.17.0.1:11434/v1/models
docker network rm hbench-check

The output must list qwen2.5-coder:7b. If the command hangs, a firewall is probably dropping traffic from Docker's networks. With ufw, allow them:

bash
sudo ufw allow from 172.16.0.0/12 to 172.17.0.1 port 11434 proto tcp

If Docker gives your networks addresses in 192.168.0.0/16, allow that range too. Rootless Docker has no 172.17.0.1, so use Docker's usual daemon. With Docker Desktop for Linux, choose the Docker Desktop option under Where Docker runs.

LM Studio

Load the model under the spec's name with LM Studio's lms command, in ~/.lmstudio/bin:

bash
lms server start --port 1234
lms load <model-key> --identifier qwen2.5-coder:7b -y

<model-key> is the Qwen2.5-Coder 7B Instruct model you downloaded; lms ls lists them. On Linux, add --bind 172.17.0.1 to lms server start. The address is host.docker.internal:1234 on Docker Desktop and 172.17.0.1:1234 on Linux. /v1/models lists every downloaded model, so run lms ps to check that qwen2.5-coder:7b is the one loaded.

Other servers

Serve the model as qwen2.5-coder:7b on an address the containers can reach:

ServerName and address
vLLM--served-model-name qwen2.5-coder:7b --host 172.17.0.1
llama.cpp's llama-server--alias qwen2.5-coder:7b --host 172.17.0.1
A server in a containerPublish its port on the bridge address, such as -p 172.17.0.1:8000:8000

On Docker Desktop, listen on 127.0.0.1 or 0.0.0.0 instead, and use host.docker.internal.

Settings that change the score

The site doesn't check how you serve the model, but two settings change how well an agent does:

  • Context length. Qwen2.5-Coder 7B takes 32,768 tokens, and ollama picks a smaller default on most machines, which cuts off long conversations. Set OLLAMA_CONTEXT_LENGTH=32768, or -c 32768 on lms load.

  • Requests at once. The free boards run two trials at a time. A server that answers one request at a time makes trials wait and time out. Set OLLAMA_NUM_PARALLEL=2, or --parallel 2 on lms load.

Serve another model

Any model your server serves can go on a board of its own. Point the agent at the server with OPENAI_BASE_URL and name the model as openai/<name>. On Docker Desktop:

bash
OPENAI_BASE_URL=http://host.docker.internal:11434/v1 OPENAI_API_KEY=local \
  hbench run -d humanevalfix@1.0 -l 3 -a mini-swe-agent -m openai/qwen2.5-coder:14b \
  --registry-url https://raw.githubusercontent.com/laude-institute/harbor/main/registry.json

On Linux, use http://172.17.0.1:11434/v1. This plain hbench run enforces no allowlist, so any address works and the job goes on a board labelled open network; Network: allowlist or open explains the difference. The board's model is the name the trials recorded, here openai/qwen2.5-coder:14b.

Reviewed on October 24, 2026.