Troubleshooting
Troubleshooting
Each problem we know of in installing hbench, running a benchmark, serving the model and uploading a job, with what you see, why it happens and what to do.
Search this page for the exact message you see. Each entry says why it happens and what to do. If yours isn't here, ask in #support on Discord or email hello@nodeq.tech with the message and the run number.
Installing hbench
unsupported platform
You see unsupported platform: <os> <arch> from the install script.
Why. hbench has builds for Linux and macOS, on x86_64 and arm64, and nothing else. There's no Windows build.
What to do. Run it on Linux or macOS. On Windows, run it inside WSL2, as in Windows.
could not download hbench
You see could not download hbench-<tag>-<platform>.tar.gz of hbench <tag> from node-q/hbench, or could not find the latest hbench release of node-q/hbench.
Why. The first means the version you asked for doesn't exist as a release, or GitHub couldn't be reached. The second means HBENCH_VERSION is missing its leading v, or, without HBENCH_VERSION, the script couldn't look up the latest release on GitHub.
What to do. Use the tag as the releases name it, such as HBENCH_VERSION=v0.3.0, and check that you can reach github.com.
hbench: command not found
You see command not found: hbench after the install script said installed hbench.
Why. The folder it installed into, ~/.local/bin unless you set HBENCH_INSTALL_DIR, isn't on your PATH. The script says so but doesn't change your shell's settings.
What to do. Add the folder to PATH in your shell's profile, such as ~/.zshrc or ~/.bashrc, and open a new terminal:
Docker
docker daemon unreachable
You see docker daemon unreachable or ` docker version timed out after 30s `.
Why. Docker isn't running, or docker points at another daemon through DOCKER_CONTEXT or DOCKER_HOST.
What to do. Start Docker Desktop, or the Docker service on Linux, and check that docker version shows a server. On Linux, if it says permission denied, add yourself to the docker group and log in again. See Check that Docker works.
docker compose is required
You see ` docker compose (v2 plugin) is required `.
Why. hbench uses the Docker Compose plugin, version 2 or later, which your Docker doesn't have.
What to do. Install the Compose plugin; on Debian or Ubuntu with Docker's packages, that's docker-compose-plugin. Docker Desktop includes it.
No space left on device
You see no space left on device while images build or pull.
Why. Task images take 1 to 3 GB each, and Docker Desktop also has its own disk limit.
What to do. Free space with hbench cache clean, which removes only hbench's cache and the images it built, and on Docker Desktop raise the disk usage limit under Settings, then Resources.
Serving the model
Every trial errored
You see a run with no score, "all errored", and NetworkConnectionError, AgentAuthenticationError or ModelNotFoundError on its trials.
Why. The agent never got an answer from the model. With a model on your machine, the model server isn't reachable from the containers or doesn't serve the model's name. With a commercial model, the key isn't in the shell hbench ran from, such as DEEPSEEK_API_KEY for DeepSeek-Flash or ANTHROPIC_API_KEY for Claude Opus 5.5.
What to do.
With a model on your machine, check from a container, with your model server address:
It must list the model's name. On Linux, bind the server to
172.17.0.1and test from a new Docker network, as in Check from a container's network.With a commercial model, run
echo "${DEEPSEEK_API_KEY:+set}", or the same with your provider's variable, in the shell you run hbench from. It printssetwhen the key is there.Move the job's folder out of
jobs/and run the same command again. A run token stays valid until it's used or 48 hours pass.
Connection refused on Linux
You see NetworkConnectionError with Connection refused, and ollama works with curl http://localhost:11434.
Why. ollama listens on 127.0.0.1, which containers can't reach.
What to do. Bind it to 172.17.0.1:11434, as in ollama on Linux.
Address already in use
You see bind: address already in use from ollama serve.
Why. ollama is already running as a service, on port 11434.
What to do. Set OLLAMA_HOST in the service instead, with sudo systemctl edit ollama: see ollama on Linux.
The model isn't found
You see ModelNotFoundError on every trial.
Why. The server doesn't have a model under the name the specification uses. qwen2.5:7b and qwen2.5-coder:7b are different models.
What to do. Pull or load the model as qwen2.5-coder:7b, check it's listed at /v1/models, and run the same files again.
Agents
AgentSetupError
You see Agent setup failed before the agent started (environment/install problem, N attempts) on every trial.
Why. The agent didn't install in the task container. On Apple silicon, codex, vibe, trae-agent, openhands and hermes failed to install in our test. Elsewhere it's usually a network problem while downloading the agent.
What to do. On a Mac, pick another agent. Otherwise, check your connection and run again; open a trial's Logs to see where the install stopped.
mini-swe-agent needs provider/model
You see Model name must be in the format provider/model_name, or No API key found for model.
Why. -m named the model without its provider, or the provider's key isn't set.
What to do. Name the model as provider/model, such as openai/qwen2.5-coder:7b or deepseek/deepseek-flash, and set the key: OPENAI_API_KEY=local for a local server, DEEPSEEK_API_KEY for DeepSeek, or ANTHROPIC_API_KEY for Anthropic. See Models and keys.
Trials
AgentTimeoutError
You see Agent execution timed out after N seconds on some trials.
Why. The agent was still working at the task's time limit. Each task sets its own limit. On a 7B model or a CPU, agents often spend their turns on setup or repeat themselves.
What to do. Nothing in the run: the tests still ran on what the agent left, and the trial usually scores 0. The time limits are part of the board, so a job with longer ones goes on another board. To do better on the same board, change the harness: limit turns, catch repeated commands, keep the prompt short. Trials time out more on a slower machine, so close other work during a run.
Network blocked by the allowlist
You see NetworkConnectionError or ApiConnectionError on trials, and egress refused: <host>:<port> in its trial log, under Logs.
Why. The run enforced an egress allowlist, as a board's spec.json does, so during its turn the agent could reach only the hosts on the list: the model, and on some boards PyPI and npm. The agent tried to reach another host, for example to download a tool.
What to do. Make the harness work with the allowed hosts, which Network access lists. The site refuses a job with other hosts on the list for that board, with local-spec-mismatch. A job with no allowlist goes on an open-network board instead: see Network: allowlist or open.
The run stopped part way
You see a job with trials missing, after the machine slept or you pressed Ctrl-C.
Why. hbench stopped before every trial finished.
What to do. Continue it. Finished trials are kept, and so are trials that ended in an error; only cancelled ones run again:
job already exists
You see `job <name> already exists; use hbench jobs resume -p jobs/<name> `.
Why. A job with that name is already in the jobs folder, from an earlier attempt at the same run.
What to do. Resume it, as the message says. To start that run over, move the old folder out of jobs/ first.
Issuing a run
These come from Issue the run on Run a benchmark, or from POST /api/local-runs. Issuing is optional: you can upload a job without it and publish it to its board.
The daily limit is reached (issue-quota)
You see you've been issued 4 runs on '<board>' in the last 24 hours, the most it allows; issue another once the oldest is 24 hours old, or upload a job without issuing a run (issue-quota), with the board's own number.
Why. Every run issued on that board in the last 24 hours counts, including ones you didn't upload. The day is a rolling 24 hours, not a calendar day.
What to do. Wait until 24 hours have passed since the oldest of those runs was issued; then you can issue one more. Or run without issuing, and upload the job. Limits has each board's numbers.
A run is already waiting for upload (run-in-flight)
You see you already have 1 run waiting for upload on '<board>', as many as it allows at once; upload or abandon one first (run-in-flight).
Why. You already have a run waiting for upload on this board, and a board takes one at a time per account.
What to do. Upload it, or abandon it: on Your bench, press Abandon beside the run, then Abandon this run?. It counts as expired and frees the slot at once.
Three entries already (entry-limit)
You see you already have 3 entries on '<board>', the most one account can have; to run one of them again, use its agent and options (entry-limit).
Why. Each account can have three entries on a board, whether you issue runs or publish jobs. An entry is one agent with its options, prompt template and instructions.
What to do. Run or publish one of those agents with the same options again, in any version; that doesn't take another place.
A body that names its entry (entry-name-generated)
You see the site names each entry from your account and its agent and options; leave out name (entry-name-generated), with status 422, when a script issues a run or publishes a job. For a publish, it says submission.
Why. The site names every entry, so a run of the same agent and options always joins the same entry of yours.
What to do. Leave name out of the body, or submission out of a publish. The answer's submission is the name the site gave.
An option names a model (model-option)
You see option 'fallback_model' of agent 'claude-code' names a model or a model endpoint, and board '<board>' holds its model fixed; leave the option out (model-option), with your option's name, when you issue a run or publish a job.
Why. Every board holds its model fixed, so a run can't set an option that picks a model or the address of one. Options that swap the model lists the names that count. So does an option whose value is an object, or JSON text, with such a key at any depth: the message then names the path, such as config.model.
What to do. Leave the option out. To try a harness on another model, choose that model for the run: it goes on that model's board.
An option the board fixes (option-fixed)
You see board '<board>' runs agent 'claude-code' with disable_web_search: true; leave the option out, or set it to true (option-fixed) when you issue a run. When you publish a job, you see the board runs claude-code with disable_web_search: true, and the job ran it with null; run it again with --ak disable_web_search=true (option-fixed).
Why. On the named boards, claude-code runs with disable_web_search: true, whatever a run sets. A run that leaves the option out runs with it set, and one that sets it to another value is refused. A job you ran without issuing a run has to have run with the option set the same way.
What to do. Leave the option out, or set it to the board's value. For a job to publish, run it again with --ak disable_web_search=true. Options a board holds lists them.
Uploading and publishing
When the site refuses an upload or a publish, it stores nothing, and the message ends with a rule code in brackets. You can fix the job and try again; for a run you issued, with the same run token until it expires.
Too big, or storage full (bench-quota)
You see larger than the per-upload cap, larger than your remaining quota or the per-job cap, your workspace is full (<bytes> bytes); delete a job first, or more than 20000 files and folders in one job.
Why. One upload can be up to 95 MiB as sent, one job up to 512 MiB unpacked, and Your bench holds 1 GiB in all. A full SWE-bench Verified run takes 150 to 250 MiB unpacked, and about 30 to 50 MiB as sent.
What to do. Delete jobs you don't need, and check that you packed one job folder, not the whole jobs/ folder. A single job bigger than the cap can't be uploaded yet.
The site is full (bench-full)
You see Your bench has no room for new uploads right now; try again later.
Why. Every workspace's files together have reached the most the site stores. The limit is the site's, not yours, and counts each upload at the 512 MiB cap until it's unpacked.
What to do. Try again later: room comes back when a maintainer purges hidden and deleted files after a review.
Not a job archive (bench-layout)
You see not a gzipped tar archive, the job folder has no config.json, or the job has no usable name.
Why. The archive isn't gzipped, or its top folder isn't the job's folder.
What to do. Pack it from the jobs folder, naming the job: tar czf job.tgz -C jobs <job-name>. Or drop the job's folder itself on Your bench, which packs it for you.
A NUL character in the job's files (bench-nul)
You see <file> holds a NUL character (\u0000), which the site can't store.
Why. A config.json, lock.json, result.json or manifest.json of the job or of a trial has a NUL character in a name or a value, such as an agent option, an agent or model name, or where a task came from; or a trial's agent/trajectory.json names a model with one. The site stores what these files say, and can't store that character. A job already on Your bench that has one is refused the same way when it's published.
What to do. Run the job again without the character in its options or names.
A job with that name, or uploaded before (bench-duplicate)
You see you already have a job named '<name>'; delete it first, hbench job <id> was uploaded before, or a trial of <task> is one uploaded before: a job is uploaded once.
Why. Your bench already has a job of that name, or someone, maybe you, already uploaded this job or a trial of it. A job counts once, however it's renamed: it's known by hbench's job id, and a trial by its task, start, finish and tokens.
What to do. Delete the old job on Your bench, or upload the right one. To upload a job again with its run token, delete your private copy of it first. If you published that copy, the upload is refused as published before.
Files changed after the run (bench-manifest)
You see the job's files differ from its manifest.json, followed by what changed, is missing, or isn't in it; or hbench 0.3.0 seals every job with manifest.json, and this one has none.
Why. From 0.3.0, hbench seals every job it finishes with manifest.json, the SHA-256 of each file it wrote. A file was edited, removed or added after the run, or the manifest was deleted.
What to do. Upload the job folder as hbench left it. Run hbench jobs verify -p jobs/<job> to see what differs. An analysis written later by hbench analyze is fine.
Not one dataset and one agent (cell-job)
You see one of:
the job's config.json does not name one dataset and one agentthe job has no trials with a readable lock.jsona trial's lock.json records no task name and digestthe job ran two different tasks named <task>a trial of <task> records a timeout multiplier that is not a numberthe job's trials ran with different networks, timeouts or memorythe job's trials did not run the tasks its config.json chose
Why. The site reads a job's board from its config.json and its trials' lock.json. A job of several datasets, agents or models, such as one run with two -m flags, belongs on no single board. Nor does a job whose trials ran under different settings, for example because it was resumed with other flags.
What to do. Run one dataset with one agent and one model in each job, and resume a job with the same command it started with. Upload the job as hbench wrote it.
Not a registry dataset (cell-dataset)
You see the job's dataset is not a registry dataset with a version, so nobody else can run the same tasks, <task> ran from a local directory, so nobody else can run the same tasks, or the task catalog does not know the job's <dataset>, and its config.json names no registry for it.
Why. A board compares runs of the same tasks, so its dataset has to come from a registry, with a version. A job of tasks in a folder on your machine (--path) can't go on a board. A dataset the site doesn't know also needs the registry it came from, written in the job's config.json.
What to do. Keep a job of your own tasks on Your bench, as a private run. For a dataset, run it with -d name@version and a registry flag, as in Choose a benchmark, a registry and a model.
No model (cell-model)
You see the job's agent names no model.
Why. The board's model is the name the job's lock records for the agent, and this job's agent has none.
What to do. Run it again with -m provider/model, such as -m deepseek/deepseek-flash.
Too many attempts (cell-attempts)
You see the job's n_attempts is 10; a job makes 1 to 5, with the job's own number.
Why. A board takes jobs with 1 to 5 attempts per task.
What to do. Run it again with --n-attempts from 1 to 5. Several runs of the same configuration, each with fewer attempts, add up on the board instead.
Something the site never accepts (cell-denied)
You see the policy refuses the agent oracle: it runs the task's reference solution, so it would top every board without being a harness (cell-denied), or the same for another agent, a model, a dataset or a registry, with its reason.
Why. The job ran something on the deny list.
What to do. Run it with something else. The oracle agent is for checking that a task's own solution passes, not for a score.
Can't publish a job
You see job <id>'s files have expired, so it can no longer be checked, job <id> is already the result of run <run>, unknown board '<board>', or board '<board>' is closed and takes no new runs.
Why. Your bench hides most of a job's files 30 days after upload (Limits), and the site checks every file when you publish. A job is also published once: the second message names the run it already is. The last two name a board the site doesn't have, or one that has closed.
What to do. Run the benchmark again and upload the new job, then publish it within 30 days. A job that's already a run's is on its board, or was withdrawn. To publish to another board, choose one that Your bench offers for the job.
A board that takes no jobs (local-not-open)
You see board '<board>' takes no published jobs (local-not-open) when you publish a job, or board '<board>' is gone, so this run can't be uploaded (local-not-open) when you upload with a run token.
Why. The first board takes runs only from the site's own runner, so it takes no job you ran yourself. The second run was issued on a board the site has since removed, and the site can't check the job without it.
What to do. Publish the job to a board Your bench offers for it. For a run whose board is gone, upload the job without its run token, then publish it.
Published before (bench-published-before)
You see hbench job <id> was published before, and a job is published once: run the benchmark again for a new result (bench-published-before), or a trial of <task> was published before, and a job is published once: run the benchmark again for a new result (bench-published-before).
Why. This job, or one of its trials, already has a score: someone published it or uploaded it with its run token. The score stays when the job is deleted from Your bench, so the site keeps the hbench id and each trial's task, start, finish and tokens of every job it scored. That way the same work counts once, whoever uploads it again and under any name. You see this message when you publish a copy or upload one with a run token.
What to do. Keep the copy on Your bench as a private job: Your bench offers it no board. For a new score, run the benchmark again and upload the new job. Delete a job has more.
Run token expired or unknown (local-expired)
You see this run token is unknown, someone else's, or expired.
Why. More than 48 hours have passed since the run was issued, or you abandoned it, or the token was mistyped. Or the token belongs to a different account from the API token or sign-in sent with it.
What to do. Paste the token again in full, and send it alone or with your own API token. If it has expired, upload the job without a token and publish it, or issue a new run and run it again.
Already uploaded (local-duplicate)
You see run <id> already has an upload.
Why. A run token takes one upload.
What to do. Nothing, if the first upload was the one you meant. To add another run to the score, run the same configuration again, issued or not. Check a job with hbench jobs summary before you upload it.
hbench too old (local-hbench-version)
You see the job ran hbench 0.2.6, which doesn't seal its jobs; run it again with hbench 0.3.0 or later.
Why. The site accepts jobs from hbench 0.3.0 or later, the releases that seal each job with manifest.json. There was no 0.2.7: the release after 0.2.6 is 0.3.0.
What to do. Install the latest release with curl -fsSL https://nodeq.tech/install.sh | sh, or read the script first, and run it again in a new folder or after moving the old job out of jobs/.
Not an hbench release (local-hbench-unreleased)
You see the job ran hbench 0.3.8-dev, which is not a published release of node-q/hbench; run it again with one from https://github.com/node-q/hbench/releases, or for one published since the site last read that list, upload again in 2 minutes, or the site could not read hbench's releases from GitHub to check hbench 0.3.7; try again later.
Why. The site takes jobs only of the hbench versions published on the releases page; a draft, a prerelease or a build of your own is not one. It reads that list from GitHub again at most every two minutes for each page of it, so a release published in the last few minutes may not be known yet. When GitHub can't be reached, it can't check a version it hasn't seen, and refuses the upload for now.
What to do. Install a release with curl -fsSL https://nodeq.tech/install.sh | sh, or read the script first, and run the benchmark again. If the release is new, or the message says to try again later, upload the same job again after the minutes it names.
Results epoch hbench didn't write (local-results-epoch)
You see the job's lock.json records results epoch "x", and hbench records one as major.minor, such as 0.4, or the job ran hbench 0.4.0, whose results epoch is 0.4, and its lock.json records a newer one, 9.0.
Why. hbench records in lock.json the results epoch its release scores in: two numbers, never newer than the release's own major.minor. A board ranks the runs of its newest epoch, so the site takes no other. The job's lock.json was changed after hbench wrote it.
What to do. Run the benchmark again with an hbench release from the releases, and upload the job as hbench left it.
Retries that could re-roll an answer (local-retries)
You see the job would retry AgentTimeoutError, …, and a retry may rerun only a crash; run it again with --max-retries and no other retry flags.
Why. Retrying is fine while it only reruns trials that crashed, which is what --max-retries does with hbench's default exclusions. The job's retry settings also retried an exception that can follow a real attempt, such as a timeout or a missing reward, so a retry could have replaced a wrong answer.
What to do. Run it again with --max-retries alone, leaving which errors hbench retries at its default. A run's page shows how many retries it used.
Started before it was issued (local-started-early)
You see a trial of <task> started before the run was issued.
Why. The job you uploaded with the run token was run before the token was issued, so it's an older job, not this run.
What to do. Upload the job you ran after issuing the run. To put an older job on its board, upload it without the token and publish it instead.
Doesn't match the specification (local-spec-mismatch)
You see a message such as job_name is 'x', the spec's is 'y', agent options differ from the spec, allowed hosts differ from the spec, agent.override_timeout_sec differs from the spec, did not use the issued prompt.j2, or, when you publish, <trial> used a prompt template or extra instructions, which the site can't read from a job; issue the run to put a prompt on a board (local-spec-mismatch).
Why. The job ran with settings the run wasn't issued with: an edited spec.json or prompt.j2, or extra flags on hbench run. A job you publish to a board that a spec defines is compared with the spec that board would issue for its agent and options: its allowed hosts with the allowlist enforced, its timeouts and resources, and none of the settings only a spec sets, such as an absolute timeout, skills, MCP servers or mounts.
What to do. Save the run's files again from the site, unchanged, and run hbench run --config spec.json with no other flags except --jobs-dir or --max-retries. For a harness with a prompt template or extra instructions, issue the run instead of publishing a job.
Missing trials (local-trial-count)
You see N trials, and the board takes M; missing [...]: the job stopped early, or ran only part of the board's tasks, or unexpected [...]: tasks the board doesn't run, of another task set or dataset.
Why. The job doesn't have one trial for each of its board's tasks and attempts, usually because the run stopped part way, or because a run you issued ran with --n-tasks or --include-task-name.
What to do. If the run stopped part way, resume the job so every task finishes, then pack and upload it again. If it ran with --n-tasks or --include-task-name, run it again without them.
Incomplete job (local-bundle)
You see the job needs readable config.json, lock.json and result.json, or a message that a mean or count doesn't follow from the trials.
Why. The job's files are missing, or were edited after the run.
What to do. Upload the job folder as hbench wrote it. If a trial is missing its files, resume the job first.
Marked invalid (local-agent-mismatch, local-task-checksum)
You see a run you issued accepted, but marked Invalid with no score. When you publish a job, the site refuses it instead, with a message such as the job ran <model>; board '<board>' runs <model> (local-agent-mismatch) or <trial> ran a <task> that differs from the board's task of that name; its lock says it came from <source>; the board's tasks come from <registry>, which the site's spec.json names (local-task-checksum).
Why. A trial recorded a different agent or model name from the specification, or ran a task that differs from the catalogued one. That happens when an issued run ran with --agent, --model or an edited spec.json, or took the dataset from another registry whose tasks of that name are different, such as hbench's default registry for humanevalfix@1.0 and swebench-verified@1.0. The message says where the trial's lock records the task came from.
What to do. Issue a new run and run its files unchanged with hbench run --config spec.json. The invalid run stays in your counts. For the known datasets, run with --registry-url https://raw.githubusercontent.com/laude-institute/harbor/main/registry.json.
Not signed in, or too many uploads
You see sign in at /auth/login (401), unknown API token, or rate limit: at most 100 per 3600s (429).
Why. curl sent no API token and no run token, or an API token that was deleted; or you've uploaded 100 times in the last hour, with or without run tokens.
What to do. For a run you issued, check HPG_RUN_TOKEN holds its run token. For any other job, check HPG_API_TOKEN holds a token from Account settings. For the 429, wait for the time in the retry-after header.
My run isn't listed
Every job you upload is private until you publish it. Open it on Your bench: if it shows Publish, press it, as in Put it on its board. If Your bench offers no board, the job can't go on one: it ran tasks from your own folder, it fails a check described above, or it was published before.
A run you issued joins its board once you publish its upload. If it doesn't, check its status on Your bench:
Private: you haven't published it yet. Press Publish, then Make it public?.
Waiting for upload: the job wasn't uploaded with its run token. An upload without the token is a private job. Upload it again with the token, after deleting the private copy if the name clashes, or publish the private copy.
Invalid: a trial didn't match the specification. See Marked invalid.
Expired: it wasn't uploaded within 48 hours, or you abandoned it.
Withdrawn: you withdrew its score.
Taken down: we took it off the board. The run still counts as uploaded.
Not listed: it belongs to an older results epoch than the board's, so it's on no board.
Every other run is on its board, in its configuration's row: a newer run joins the score instead of replacing it. A run whose trials all errored counts as crashed in its row and isn't scored.
If your run is on a board you didn't expect, it ran something other than you meant: another task set, model, attempts, time limits, memory or network. The board's name says what differs, and Cells and boards lists each part.
Reviewed on October 30, 2026.