Skip to content
Hbench
hbench

Reference

Network: allowlist or open

What the network part of a cell means, how hbench enforces an egress allowlist for the agent, why allowlist and open runs never share a board, and how to run with an allowlist.

An agent that can reach the internet can look answers up, such as a later release of the project it's fixing. So the network is part of a run's cell, and a board holds runs of one kind:

NetworkWhat the agent could reach while it worked
allowlistOnly the hosts on hbench's egress allowlist. hbench refused every other connection.
openWhatever the task's own network settings allowed, which for most tasks is the internet.

A board on an open network says so beside its name. Read its scores knowing the agent could search for answers.

How hbench enforces it

hbench enforces the list when the agent's configuration sets enforce_allowlist: true. The agent then reaches only the hosts in its extra_allowed_hosts, or none if the list is empty. The list covers the agent's turn only: installing the agent and running the tests keep the task's own network, because both download packages.

Each trial's lock.json records whether hbench enforced the list, and the site reads that. If hbench finds that Docker on your machine can't enforce the list, it stops with an error instead of running the agent on an open network.

Run with an allowlist

The spec.json of an issued run sets its board's allowlist, so hbench run --config spec.json enforces it. To set one yourself, write a job configuration, such as job.yaml:

yaml
agents:
  - name: mini-swe-agent
    model_name: deepseek/deepseek-flash
    enforce_allowlist: true
    extra_allowed_hosts:
      - api.deepseek.com

Run it with the dataset: hbench run --config job.yaml -d terminal-bench@2.0. Command-line flags override the file. --extra-allowed-host is a different setting: it adds a host for tasks that restrict their own network, and doesn't turn the allowlist on.

The agent needs its model's API, so the list names that host. The named boards allow the model's host, plus PyPI and npm on the boards with a commercial model other than SWE-bench Verified. SWE-bench Verified allows no package index, because one could serve a later release of the project that already has the fix. The methodology shows each named board's list.

claude-code's web search

claude-code searches the web on Anthropic's side, out of the allowlist's reach. The named boards run it with disable_web_search: true: see Options a board holds.

Reviewed on October 24, 2026.