FAQ
Short answers about agent harnesses, how a board keeps the model and tasks fixed, what a score means, and how to put your run on a board.
Reviewed
No question matches . Email hello@nodeq.tech and we'll add it.
The program around a model that turns its text into actions: the prompt, the tools, the loop that runs commands and shows the model their output, and the rule for when to stop.
A model on its own only writes text. The harness is what lets it edit files, run tests and recover from a failed command.
Harness Playground compares harnesses. Each board fixes the model, tasks, attempts and limits, so its rows differ only in the harness.
Held fixed by the board
Your harness, the part that changes
Measured
The pass rate over the board's tasks, with its 95% interval: see the boards.
Almost nothing. Harness and scaffold both name the code that turns a model's output into actions. A framework is a library for building that code.
Papers tend to say scaffold; tool makers tend to say harness. An evaluation harness is different: it runs a benchmark and grades the result. Here that's hbench, and it's the same for every run.
Often, yes. In an audit of 254 SWE-bench entries, the same model scored up to 29.8 points apart depending on the harness.
Across different models, the model usually matters more. That's why each board fixes the model: what's left to compare is the harness.
Harnesses differ in how they show files, apply edits, run tests, recover from failed commands and decide to stop. Claude Code and Codex CLI are harnesses in this sense too. A choice that suits a large model can trap a small one in a loop, so we expect small models to show the biggest gaps.
The audit (arXiv 2609.17394) also found the top 30 entries within 8.8 points of each other. It was observational, so it doesn't prove the harness caused every gap.
It compares harnesses on one model at a time. The SWE-bench leaderboard ranks whole systems, each a model and an agent together.
Every run on a board uses the same benchmark, model, tasks and attempts. Anyone can open a new board by uploading a run, so boards cover many benchmarks and models, and you can check whether a harness leads on more than one. Scores here rarely match the official leaderboard (why).
One benchmark on one model, with the same task set, attempts, time limits, memory and network for every run. Its name says which, for example Terminal-Bench 2.0 on DeepSeek-Flash.
Because everything else is fixed, a board's rows differ only in their harness. Each row is one configuration (an agent with its options, prompt and version), scored over every run of it by anyone. A run with a different model, task set or setting goes on a different board, so scores from different conditions never mix.
The boards on Qwen2.5-Coder 7B: you serve the model on your own machine, so they need no key and cost nothing in model fees.
Start with HumanEvalFix on Qwen2.5-Coder 7B: its tasks are short, and a 7B model solves some of them. Terminal-Bench 2.0 on Qwen2.5-Coder 7B is for checking that a harness runs from start to finish, because a 7B model solves almost none of its tasks. Any other model your machine serves is free too, on a board of its own. You need hbench, which installs with one command.
Any benchmark from a registry. Five are known benchmarks, reviewed by the maintainers: Terminal-Bench 2.0, SWE-bench Verified, Aider Polyglot, LiveCodeBench v6 and HumanEvalFix. The benchmarks page lists every board on them.
They test different work: shell tasks in a container, fixing real repository issues, editing code in six languages, contest problems, and repairing one buggy function. Running a harness on several shows whether it's good at more than one kind. Any other dataset is marked unreviewed (what that means). We aren't affiliated with any of these benchmarks.
The maintainers haven't read its tasks, so its board tells you nothing about the benchmark it's named after.
Anyone can publish a registry. A dataset the site's task catalog doesn't know is shown by name, version and registry, such as humanevalfix@1.0 (ghcr.io), and marked unreviewed. Its tasks could be easier, broken, or include their answers. The site still checks every job and tells datasets apart by their tasks, so an unreviewed board never mixes with a known benchmark of the same name, and it's never featured. If one imitates a known benchmark to mislead, use Report a submission.
Any agent hbench can run, including its built-in Claude Code, Codex CLI and Cursor CLI. The benchmarks page shows which agents have results on each board.
Compare two agents only on a board where both have results. Some agents need their own account or key, so check an agent's options before you run it.
89 tasks an agent completes in a Linux container through the shell, such as building software from source or training a small model. Each task has its own tests.
Each task sets its own time limit, and the tests check the container's final state, so an agent scores only if it leaves the container right. Two named boards use it: Terminal-Bench 2.0 on Qwen2.5-Coder 7B runs the first 10 tasks, and Terminal-Bench 2.0 on DeepSeek-Flash runs all 89.
500 real GitHub issues from Python projects, each judged by the project's own tests. One named board runs a fixed 50 of them, so a run takes an estimated 3 to 5 hours.
The 50 tasks keep the full set's mix of repositories, such as Django and SymPy. They were picked once and listed by name, so they never change.
SWE-bench Verified on Claude Opus 5.5 runs all 500 tasks. By its own estimate, a run takes 5 to 7 hours and about $80 in model fees.
Aider Polyglot is 225 Exercism exercises in C++, Go, Java, JavaScript, Python and Rust. HumanEvalFix is a set of small Python functions, each with one bug to fix until its tests pass.
Aider Polyglot shows whether a harness edits code reliably outside Python. Aider's own leaderboard uses it with Aider fixed and the model varied. HumanEvalFix comes from HumanEvalPack. A 7B model solves some of its tasks, so differences between harnesses show up in the score, which they rarely do on the free Terminal-Bench board.
Contest problems from LeetCode, AtCoder and Codeforces, each tagged with its release date. The known version is 100 problems from release v6, published between May 2023 and April 2025.
The dates let you check whether a problem appeared after a model's training data was collected. For DeepSeek-Flash most didn't, so treat them as public tasks like the rest. The problems reward algorithmic reasoning under tight time limits more than work in a large codebase.
A 7B model runs on an ordinary GPU with no cost per task, so anyone can run a free board on their own machine.
A small model also exposes harness weaknesses: it loops, misuses tools and runs out of time where a larger model would recover. A ranking on Qwen2.5-Coder 7B tells you how harnesses do on that model, not whether the order holds on a larger one.
The share of trials whose tests passed. A trial is one attempt at one task: it scores 1 if the tests pass and 0 if they don't. The score estimates pass@1, the chance that one attempt solves a task.
With one attempt per task, the score is the share of tasks solved. Boards don't rank by pass@k (the chance that at least one of k attempts succeeds), because it flatters an agent that is right only sometimes.
A trial that errors before the tests run scores 0. A trial that times out is still graded. Each row's score on a board is the mean over every run of it, by anyone. Once a row has five full runs (a result for every task), its highest and lowest runs are left out.
The number of attempts is part of what defines a board. You can run 1 to 5 attempts per task, but a job with 3 goes on a different board from one with 1, so match the board you want to join. To make a score steadier, run the same configuration again: each run is added to its score and narrows its 95% interval.
By a few points. A study of 60,000 SWE-bench Verified trajectories found single-run scores moving by 2.2 to 6.0 percentage points, depending on which run was picked.
The same study found a standard deviation above 1.5 points even at temperature 0 (arXiv 2602.07150). Each score on a board has a 95% interval, bootstrapped over its tasks, and each row's rank range shows which places the paired test allows. Treat a gap of a few points as a tie.
Only if both runs are on the same board. The site puts each run on the board whose runs share its benchmark tasks, model, attempts, time limits, memory, network and hbench results epoch.
On one board, only the harness differs. Your bench compares your run task by task with the others there, and each row's Δ on the board shows which gaps the paired test can tell apart.
Scores on different boards don't compare. 40% on HumanEvalFix with a 7B model and 40% on Terminal-Bench with DeepSeek-Flash measure different things, as do 40% with one attempt and 40% with three. To see whether a harness leads on more than one benchmark, check its place on each board.
A board doesn't fix your machine's speed, how your model was served, or the hbench release within a results epoch. A slower machine hits more time limits. See what the site can't check.
Because the conditions differ. The model, task subset, attempts, time limits and harness version rarely match another leaderboard's.
SWE-bench Verified (50) on DeepSeek-Flash runs 50 of the 500 tasks, so its scores can't be compared with the official leaderboard. Every board here also fixes one model, which other leaderboards often don't. SWE-bench Verified on Claude Opus 5.5 runs all 500 tasks and comes closest, but its limits, attempts, retries and hbench version can still differ. Treat any comparison across leaderboards as a rough guide.
By score, highest first. Once two configurations on a board have a score, each row shows the rank range the paired test allows it. Many boards hold one harness because anyone can open a board, and most settings are one person's choice.
A row's score is the mean over every run of it on the board, by anyone, with a 95% interval over the tasks. One run from one participant is enough.
To be compared with others, run what an active board already uses: its benchmark, task set, model, attempts and settings. The featured boards on the home page are a good place to start.
No trial got a result from the tests. Usually the agent couldn't reach the model: a key was missing, or the task containers couldn't reach the model server.
The run gets no score, because 0% would describe a setup problem, not the harness. Its board still counts it as crashed, so readers can see how often a configuration fails. Open its trials in Your bench to see the error each one ended with.
Each errored trial counts as 0, the same as a trial whose tests failed. Your bench shows how many trials errored in each job, and the public API gives the count for every published run as n_errors.
An error means something went wrong outside the tests: the agent didn't install, couldn't reach the model, or crashed. Every planned trial counts toward the score, so errors are part of a harness's result. On a 7B model, many errors often mean the agent couldn't drive the model, so look at them before you compare.
Usually the agent spends its turns on setup or repeats itself, and on a 7B model it reaches the task's time limit before it finishes.
Each task sets its own time limit, and the limits are part of the board. A trial that times out is still graded, and usually scores 0. The free Terminal-Bench board is for checking that a harness runs from start to finish, because a 7B model solves almost none of its tasks.
The run's model cost, as hbench records it, divided by the number of trials solved.
It shows what an agent spends per result, which can differ a lot between harnesses on the same model. We estimate about $0.04 a trial for mini-swe-agent on Terminal-Bench 2.0 with DeepSeek-Flash, or $3 to $4 for all 89 tasks, based on one run before the results reset. Free boards have no model fee, so they show no cost.
As far as it tells you its conditions. Every board here states its model, tasks, attempts and network, counts its failed runs, and shows how many of each harness's issued runs are public.
Before trusting a gap, check what the board fixes, how many attempts each run had, how many issued runs were published, and whether the gap is bigger than run-to-run noise. Every result is self-reported, and some things can't be checked at all, such as which model served a run on someone's machine. The methodology lists them.
Probably, but it matters less here than elsewhere. Every run on a board uses the same model, so contamination affects them all, though it can raise absolute scores.
Its tasks are public, like the other known benchmarks', so models may have seen them in training. We assume contamination doesn't favour one harness, but we haven't tested that: a harness that leads the model to recall a memorised fix could gain more. None of the known benchmarks has held-out tasks, so no score here is a held-out result.
Several checks, none complete on its own. hbench copies the tests in only after the agent finishes, and on a board with an allowlist the agent can reach only a short list of hosts.
hbench removes grading files from the agent's container and stops the agent before the tests run. Services the agent started in the background keep running, because some tasks are graded on them. The terms forbid tampering with tests, and maintainers can take down any score. The agent and the tests share a container, so a determined agent could still interfere. That's why anyone can report a run.
Every upload is checked, but a careful forger can still pass the checks. That's why every result is self-reported.
The site checks that the job's files are sealed and unchanged, that its totals match its trials, that it ran one dataset, agent and model, and that retries reran only crashes. It doesn't rerun the job. Boards also count runs issued on them, published or not (how). If something looks wrong, use Report a submission. Maintainers can take down any score.
You ran it on your own machine, and the site saw only the job you uploaded. Every result here is self-reported.
The site checks every file of a job, but it doesn't rerun it and can't see your machine or which model answered. Every published run that passes the checks counts toward its row's score, whether it was issued or not.
To show readers more, issue your next run on Run a benchmark before you start it (how). The board then counts it once it's uploaded or expires, published or not, and the row shows how many runs were issued and how many are public. Readers can then tell a single published run from the highest-scoring of several.
Yes. The maintainers upload their runs like anyone else, and their runs rank alongside everyone's.
Their uploads go through the same checks, and rely on the maintainers running hbench honestly, like everyone's. Boards don't mark them; Results lists them as the maintainers' runs.
A maintainer removed it from its board for one of five reasons: its owner asked, it broke a published rule, a review found it wasn't genuine, a legal or safety request, or we published it by mistake.
The run leaves its board and its public page, and Your bench shows it as Taken down. An issued run still counts toward its owner's limits. The public audit log lists each take-down with its reason and a short note, which never names a person. A maintainer can put a run back on its board, and the log shows that too.
Use Report a submission for a wrong result, a run that breaks the rules, or a board on a dataset that imitates another. Include the entry name or run number.
For a security problem, email us as the security page describes, and don't post it publicly first. For anything else, email hello@nodeq.tech.
Yes. The site and hbench are free, and the free boards have no model fees because you serve Qwen2.5-Coder 7B on your own machine.
Other models bill your own key: a DeepSeek key for DeepSeek-Flash, an Anthropic key for Claude Opus 5.5 (what a run costs). Uploading is free within Your bench's limits.
Run a benchmark with hbench on your own machine, upload the job on Your bench, and publish it. The site places it on the board whose runs ran exactly the same thing.
Every result here is an hbench job. hbench installs with one command on Linux or macOS and runs each task in Docker (how to get it).
Sign in with GitHub to upload. The site accepts any finished job of a registry benchmark that hbench 0.3.0 or later sealed, up to 95 MiB as sent, with any model, any agent, and 1 to 5 attempts per task. The job must use one dataset, one agent and one model, be unchanged since hbench sealed it, and have rerun only crashed trials. The site refuses the oracle agent, which runs the task's own solution, and any option that swaps the model. A published job can't carry a prompt template or extra instructions yet; issue the run for those.
For a named board, Run a benchmark issues the run and gives you a spec.json that sets its tasks, model, limits and allowlist, which makes your result easier to trust. For anything else, it builds a plain hbench command. A job of tasks from your own folder stays on Your bench and never goes on a board, because nobody else can run the same tasks.
Run it with hbench on any benchmark and model, then upload and publish the job. It goes on the board of runs that ran the same thing, next to the built-in agents there.
Your agent must speak the Agent Client Protocol (ACP) over stdin and stdout. hbench installs it from the ACP registry, or from a git repository as a Python 3.12 project managed by uv, at the commit you name. The job records the repository and commit, and a different commit is a different row on the board.
Your code runs only on your machine, so nobody reviews it first. To keep a result to yourself, leave the job unpublished on Your bench. Run your own harness or ACP agent has the command and the files.
As often as you like. The site accepts up to 100 uploads an hour and limits only runs you issue first: 4 a day on most named boards, 1 on SWE-bench Verified on Claude Opus 5.5.
An issued run counts from the moment you issue it, over a rolling 24 hours. Each account can have up to 3 entries per board. Rerunning one of them under the same name doesn't use another.
Yes to both. Rerun the same configuration as often as you like, and withdraw any run you published or issued.
A rerun of the same agent and options is added to your entry for them and doesn't take a new entry. A withdrawn score leaves its board for good, and you can't publish a withdrawn job again. An issued run still counts toward your limits after you withdraw it. To have someone else's run removed, use Report a submission.
Run the install script from nodeq.tech on Linux or macOS. It installs the latest hbench release into ~/.local/bin.
Run curl -fsSL https://nodeq.tech/install.sh | sh. The script downloads the release for your system (x86_64 or arm64) from node-q/hbench on GitHub and checks it against the release's SHA256SUMS. To read the script first, download it, read it, then run it. Set HBENCH_VERSION to install another release, or HBENCH_INSTALL_DIR to install elsewhere. You also need Docker. On Windows, run hbench inside WSL2.
Yes, any model your agent can call. Your job goes on that model's board, next to other harnesses on the same model, and opens the board if nobody has run that model yet.
To join an existing board, use its model under the same name. The free boards use Qwen2.5-Coder 7B, usually served through ollama, so they need no key. A commercial model bills your own key, which hbench reads from your shell: a DeepSeek key for DeepSeek-Flash, an Anthropic key for Claude Opus 5.5.
A model on your machine needs an OpenAI-compatible server that the task containers can reach, such as ollama or LM Studio. The run builder shows the address. Name the model as you serve it, such as openai/qwen2.5-coder:32b, so others who run it land on the same board. The site reads the model name each trial records and can't see which model answered, so serve the model the name says.
Runs on different models never share a board, because across models the model usually matters more than the harness.
Up to about 3½ hours on one machine, estimated from the tasks' time limits. We haven't timed a full run on a typical machine yet.
HumanEvalFix on Qwen2.5-Coder 7B has 40 tasks with a 10-minute limit each, run two at a time, so the agents take at most about 3 hours 20 minutes. The 10 Terminal-Bench tasks have limits from 15 minutes to 3 hours 20 minutes, about 3 hours 40 minutes at two at a time. Building images and running tests add to that, and most trials finish before their limit.
Start with HumanEvalFix: its tasks are short, and a 7B model solves some of them. A 7B model scores close to 0 on the Terminal-Bench tasks, so that board mostly shows whether your harness runs end to end.
We estimate a GPU with 8 GB of memory, 16 GB of RAM and about 10 GB of free disk. A CPU alone also works, much more slowly. We haven't measured a full run on a typical machine yet.
Qwen2.5-Coder 7B through ollama is a 4.7 GB download (ollama's library page), and needs a little more than that in memory. Each task container gets 2 CPUs and 2 GB of memory, two at a time. HumanEvalFix's 40 task images share one base of about 1.6 GB. Terminal-Bench images are 1 to 3 GB each, so keep about 30 GB free for its free board's 10 tasks. On a CPU, more trials hit their time limits.
Yes, with most agents. hbench has native builds for Apple silicon and Intel Macs and runs tasks through Docker Desktop, but 5 built-in agents don't install there yet.
Serve the model with ollama on the Mac; the task containers reach it at host.docker.internal:11434, and the run builder fills in the address. In our test on Apple silicon, codex, vibe, trae-agent, openhands and hermes failed to install. We haven't run a full benchmark on a Mac yet.
A 7B model often repeats a failed command, writes tool calls the harness can't parse, or loses track of the task, and a harness built for larger models may not catch it.
Harnesses that limit turns, spot repeated commands, repair malformed tool calls or keep the prompt short tend to do better on small models. Open a trial in Your bench to see where the agent got stuck.
Only for a run you issued: upload it within 48 hours, or it expires. Upload any other job whenever you like.
An expired run still counts toward its board's daily limit. You can still upload its job without the run token, and publish it.
Yes. node-q/hbench-action installs and runs hbench in a GitHub Actions workflow, so each pull request can get a quick run of its own.
To put a workflow's run on a board, run the full task set on your default branch or on a schedule, then upload and publish the job. Issue runs only from workflows you start by hand or on a schedule, because each board issues an account only a few runs a day.
Your private workspace for hbench results. Upload a job folder or a .tar.gz of it, open any trial to see what the agent did and why the tests passed or failed, and publish a job to its board.
Only you can open it, except maintainers handling an abuse report or a support request. A job stays private until you publish it. You also create API tokens there.
Yes, as far as the site goes: it doesn't take model keys. hbench reads the key from your own shell, and the key leaves your machine only to reach your provider.
During a run the key is in the task's environment, so it ends up in a trajectory if the agent prints it. hbench replaces it with [REDACTED] when the trial ends. Use a separate key with a spending limit. If you saved a key here before, it's stored encrypted, and you can delete it in Account settings.
A model key pays a model provider. An API token lets a script act as you on this site. A run token uploads one run you issued.
A model key stays on your machine. An API token starts with hpg_ and lets a script upload jobs and issue runs as you (issue a run from a script). A run token starts with hpr_ and comes with a run you issue. For 48 hours it uploads that run's job without an API token, and does nothing else.
Its score, its entry name, your GitHub handle, the agent's name and a summary of its configuration. A job you don't publish stays private.
A board shows your agent's name, not its repository or code, so your harness doesn't need to be open source, and you keep ownership of what you upload. The site never publishes your contact email. Your trajectories stay private unless you make them public (how that works).
Only if the run's owner makes them public. A trajectory is the record of what an agent did and saw in a trial.
A trajectory can contain a printed key and shows how your agent works, so the choice is yours. Make a run's trajectories public from its run page or its job in Your bench, within 30 days of the upload.
Before publishing, the site replaces your saved keys and values in common key formats with [REDACTED] again, then scans each trial's trajectory with gitleaks' secret-detection rules and for nodeq.tech's own API and run tokens. A trial with a finding stays private, and the run page says how many did. Anyone can read the rest while the run's score is on its board, until you hide them. They don't expire.
The node-q organisation, as an independent project. It isn't affiliated with any model lab or benchmark.
Model fees go to each participant's own key or model server.
Partly. hbench, the benchmark runner, is open source in node-q/hbench under the Apache 2.0 licence. The site's own code isn't public yet.
Every run on the site is graded by hbench, so anyone can read the code that grades a result.
Yes. Link the run's own page (or its board), and give the board's name, the entry name, the run's date and its hbench version.
A board's name stays tied to its conditions: a run with a different model, task set or setting goes on a different board. A new hbench version can start a new results epoch, after which the board ranks only that epoch's runs, so the version pins down which results you mean.
Didn't find your question? Ask in #support on Discord or email hello@nodeq.tech.