How we measure harnesses

How good? pass@1 with its 95% interval

A run is one harness working through every task in its task set, once per attempt, and each of those attempts is a trial. When the agent finishes or runs out of time, the task's own tests run in the task container and give the trial a reward: 1 if they pass, 0 if they don't. A task can give partial credit between 0 and 1, but every task in the known datasets gives 1 or 0.

A run's pass@1 is the mean reward over every trial it planned. With one attempt per task, that's the share of tasks solved. With more, each task's attempts are averaged first, then the tasks. A trial that ends without a reward counts as 0.

A board has one row per configuration: the agent with its options, prompt template, extra instructions and version. A row's pass@1 is the mean over every run of it on the board, by anyone: each task's reward is averaged over the runs that ran it, then over the tasks. Once a row has five full runs, its highest- and lowest-scoring runs are left out. Each row shows a 95% interval, bootstrapped over the board's tasks, which narrows as runs are added. On today's boards, interval half-widths range from about 2 to 11 points, so a gap of a few points between two rows can come from variation alone. The next section explains when a board can tell two rows apart. Running a configuration again joins its row instead of adding a new one.

Better than whom? The rank range

A board tells two rows apart with one test: their paired difference. Over the tasks both rows ran, it takes row A's reward minus row B's, task by task, and bootstraps over those tasks for a 95% interval. If the interval leaves out zero, the two rows are told apart. This is the strongest test a board allows, because every run on it ran the same tasks, and it's the site's only definition of a tie.

Each row's rank range follows from this test. Its best rank is 1 plus the number of rows above it that it's told apart from. Its worst rank is the number of scored rows minus the rows below it that it's told apart from. A range of 1–3 means the row could be first, second or third, and two rows whose ranges overlap can't be ordered by this board. There's no correction for comparing many rows at once (LMArena makes none either), so on a board with many rows, expect a few ranges to be narrower than they should be.

A board's resolution says what size of gap it can tell apart: the median, over neighbouring scored rows, of the half-width of their paired interval, in points. The board's facts strip shows it as "tells apart gaps of N points or more". Wide row intervals make the resolution coarse however many rows a board holds, and more runs of each row sharpen it.

Every board orders its rows by pass@1, highest first. A row shows a rank range once two configurations on its board have a score, and one run from one participant is enough.

Worth what? Δ against the reference, and $ per solved task

A small notch above each bar on a board marks one reference row's pass rate. A row above the reference lightens from the notch to the tip of its bar, so the lightening part is its gain. A row below it ends short of the notch. Under each pass rate, Δ is the row's paired difference from the reference in points, over the tasks both ran, with its interval beside it, for example −2.6 (−4.6 to −0.6). When the interval includes zero, it says "too close to call". When the two rows share fewer tasks than the board has, it says how many, because then Δ and the gap between the bars can differ. The reference row is the benchmark's reference harness when the policy names one, so a harness's Δ reads the same way on every model of that benchmark; otherwise it's the leader. Click any row to make it the reference for your own reading.

$ per solved task is the cost per task divided by pass@1: what one solved task cost on average. A row without a cost shows as not reported.

Efficiency: tokens, time and cost per task

Each row's efficiency figures count the same trials as its pass@1: the trials of its scored runs on the board's tasks, leaving out the highest- and lowest-scoring run once the row has five full runs. Errored trials count, as they do in pass@1. A figure per task is a mean over trials, so with three attempts, each of a task's three trials counts once.

Tokens per task

The mean of each trial's input plus output tokens, with cached input tokens included as hbench reports them. Trials that report no tokens are left out. Cached input tokens usually cost much less than new ones, so a harness that reuses its context can use many tokens and still cost little. Where a board has costs, compare those instead.

Time per task

The mean time the agent worked on a trial, from start to finish, without setup or the tests. Trials that report no time are left out.

Cost per task

The mean cost of a trial in US dollars. If every trial reported a cost, the row shows what the agent reported. Otherwise, if every trial reported its input, cached and output tokens, we estimate the cost from the model's OpenRouter prices and mark it "est.", with the prices' date on hover. A row with neither shows no cost.

Pareto frontier

A board's chart plots pass@1 against tokens, time or dollars per task. It offers each measure that at least two rows report, and leaves out any row that doesn't report it. A configuration is on the frontier when no other one uses the same or less and has the same or higher pass@1, while being better on one of the two. The frontier shows which harnesses get their pass@1 cheaply. It compares means, so read it alongside each row's interval.

Behaviour: pass^k, refusals, infrastructure failures and the harness profile

Pass^k

On a board with more than one attempt per task, pass^k is the share of tasks a run solved in all k of its attempts. Each task is averaged over the row's runs, then over the tasks, as for pass@1. A trial solves its task when its reward is at least 0.5. A pass^k well below the pass@1 means the harness solves some tasks only some of the time.

Refusals

Trials the model refused, as hbench records them, and their share of the trials the row counts. A refused trial counts as unsolved.

Infrastructure failures

Trials that failed before the agent could work, such as a rejected API key or an environment that didn't start, counted beside each row. They count as unsolved. The errors section explains how the site recognises them.

Harness profile

A harness's page draws its profile on each board as a hexagon with six axes. Each axis shows the harness's place among the board's configurations that have that figure, with first place on the outer ring; an axis with no figure is dashed. A board has profiles once at least three of its configurations have trajectory figures. Each trajectory figure counts only the trials whose trajectory reports it.

• Lean context is the median of each trial's largest context, in tokens. Less is better.
• Cache use is the share of input tokens the provider served from its prompt cache. More is better.
• Tests before finishing is the share of trials with an edit that ran a test after their last edit. A test is a recognised test command, such as pytest or npm test. More is better.
• Direct path is the trials' steps added up, divided by how many of the trials solved their task. Fewer is better.
• No loops is the mean number of shell commands in a trial that repeat an earlier one exactly. Fewer is better.
• Speed is the time per task above. Less is better.

Reading a board's table

Rank is the rank range above, and shows once a row has one. Runs appears only when some configuration has more than one run, or a run that failed. Under it, spread is the standard deviation of pass@1 across a configuration's runs, and infra counts its infrastructure failures. "issued 10 · uploaded 7 · 1 public" means the site issued 10 runs of the configuration, 7 were uploaded, and one of them is public. An upload stays private until its owner publishes it, so the other 9 were kept private, withdrawn, taken down, refused as invalid, or never uploaded. That line appears only when some issued runs aren't public. Pass rate is in the chart above the table, and tokens, time and cost per task are on the Value tab.

What a cell holds fixed

A board holds one cell: everything that changes a score except the harness. A cell is the dataset (identified by its tasks' digests) with its task set, the model, the attempts per task, the time limits, the memory per task container, the network (allowlist or open), and the hbench results epoch. The site reads each part from the job itself and puts the job on the board whose runs share its cell. So a gap in score on one board comes from the harness or from run-to-run variation.

Scores compare only within one board. Across models, the model usually matters more than the harness, so the model is part of the cell. An agent that can reach the internet can look answers up, so the network is part of it too. A board that runs a subset, such as SWE-bench Verified (50) on DeepSeek-Flash, can't be compared with the benchmark's official leaderboard (why scores differ). The table lists the open named boards; any other cell gets a board of its own when its first job is published.

Attempts, and why single runs vary

The same harness on the same tasks doesn't score the same twice. The model samples its answers, commands take varying amounts of time, and a slow reply from the model can push a trial past its time limit.

Bjarnason, Silva and Monperrus measured this on SWE-bench Verified in On Randomness in Agentic Evals (arXiv 2602.07150). Across 60,000 trajectories, single-run pass@1 varied by 2.2 to 6.0 percentage points depending on which run was picked, with standard deviations above 1.5 points even at temperature 0.

Every named board runs one attempt per task, so each run is a single pass over the tasks. A job can run up to five attempts, but attempts are part of the cell, so such a job goes on a different board. Within one board, more runs of the same configuration narrow the variation: each one joins its row's score and shrinks the 95% interval beside it. Until a row has several runs, expect a wide interval and a rank range that overlaps its neighbours'.

The number of tasks also limits how finely a board can tell harnesses apart. One task is worth 100/n points of pass rate: 0.2 on 500 tasks, 2 on 50, 2.5 on 40. When most rows score near 100%, only a handful of tasks separate them, and some of those may be broken or in the model's training data. Such a board says little about which harness is better, however many runs it has.

Errors, timeouts and all-errored runs

A trial errors when something goes wrong outside the tests: the agent can't be installed, it can't reach the model, it crashes, or it runs out of time.

When the agent reaches its time limit, hbench stops it and the tests run on whatever it left in the container, so a trial that times out can still pass. An error that leaves no reward counts as 0 in the score.

A trial that failed on its setup rather than its agent still counts as 0. The site marks a trial as an infrastructure failure when its error is a setup or environment error, or when it reported 0 input and 0 output tokens, earned no reward and called no tool, as happens with a rejected API key. A board's table counts these beside each row. If more than 10% of a run's trials failed this way, the site refuses to publish it from Your bench, so you can fix the setup and run it again. Issued runs, and runs published before 26 October 2026, aren't refused this way.

If every trial in a run errors, the run is listed as all errored and has no score. The usual cause is a missing key, or a model server the task containers can't reach. A 0% would only describe that setup problem, so the run counts in its row as crashed and stays out of the score.

Self-reported runs and issued runs

Every result on the site is self-reported: someone ran hbench on their own machine and uploaded the job. The site reads and checks every file, but doesn't run the job again. Every published run that passes the checks counts toward its row's score, whether or not the site issued it.

A run issued on Run a benchmark before it starts counts on its board once it's uploaded or expires, published or not. Each row's Runs column shows how many runs were issued, how many were uploaded, and how many of those are public, so readers can tell one published run from the highest-scoring of several. The site also checks an issued run's upload against what it issued, and that no trial started before the run was issued. A run published without being issued shows only once it's published, so it could be the highest-scoring of many private runs, as on any self-reported benchmark.

The API gives each run a level, which says how much of it the site saw. Runs at every level rank together, and a run's page names only the verified level.

Registered

Issued before it started, and uploaded with its run token.

Imported

Uploaded without a run token and published from Your bench. The site saw only the finished job.

Verified

Run by our own runner, which gave every run the same hardware and limits. The site no longer runs anything, so only runs from that time are verified.

The maintainers' own runs pass the same checks and rank beside everyone's. A board doesn't mark them, and Results lists them as the maintainers' runs.

A maintainer can take a score down for one of the listed reasons, such as a broken rule or its owner's request, and you can withdraw your own. A score that's taken down leaves its board, though an issued run still counts toward its participant's limits. The public audit log shows each take-down with its reason and a short note, which never names a person. To ask for one, use Report a submission.

To check a result, run it yourself. On a named board, Build a run for this board opens the run builder with the board's benchmark, model and settings filled in. Pick the same agent, and your run joins the same board as the one you're checking.

The checks every upload passes

You run a benchmark with hbench on your own machine and upload the job. Before storing anything, the site reads every file the job and its trials recorded, and checks that:

• the job is sealed and complete: its manifest.json matches every file, every trial has its result and lock files, none appears twice, and the totals match;
• the mean in the job's results follows from the trials' own rewards;
• it ran hbench 0.3.0 or later, and its retries could rerun only trials that crashed;
• it ran one dataset from a registry, with one agent and one model in every trial, and every task of a known dataset matches the published task by its checksum;
• no agent option swaps the model, and the agent isn't on the deny list, which names the oracle agent;
• neither the job nor any of its trials was uploaded before, by anyone.

A job that fails a check is refused, and nothing is stored. A run the site issued is also compared with what it issued: the same agent, options, prompt, tasks, attempts, limits and allowlist. If its agent, model or tasks don't match, it's kept, marked invalid, and gets no score. Each run token accepts one upload.

These checks catch mistakes and careless fakes. They can't prove a reward is real: see what we can't check.

Keeping the answers out of the agent's reach

hbench runs every trial, on every participant's machine. It follows these rules so an agent can pass a task only by solving it:

• Hidden tests and reference solutions are removed from the task container before the agent starts. The tests are copied in after the agent has finished.
• The agent is stopped at its time limit, before the tests run. Services it started in the background keep running, because some tasks are graded on them.
• On a board whose network is allowlist, the agent's outbound network is limited to the board's allowlist, described in the next section. On the named boards, claude-code also runs with its web search off, because a search runs on Anthropic's side, where the allowlist doesn't reach.
• A job configuration with a key hbench doesn't know is an error, so a misspelt setting can't quietly switch a limit off.

The agent calls the model with your model key, so the key is in the task's environment while the agent works, and the agent can read it. When the trial ends, hbench replaces the key with [REDACTED] in the trial's files. When you upload the job, the site redacts known key formats, and any key you saved here, again. Use a separate key with a spending limit.

Harness code runs on its author's machine, never on ours, so nobody reviews it before it runs; each board shows which harness ran. We don't describe these controls in more detail. If you find a way around one, report it privately.

Network access

Network is part of the cell. A run is allowlist when hbench enforced an egress allowlist for the agent, and open otherwise, and the two never share a board. While an allowlist run's agent works on a task, it can reach only the hosts on the list, and every other connection is refused. On the SWE-bench Verified boards that's the model's API alone, with no PyPI or npm, so an agent can't download a later release of the project that already has the fix. The other named boards with a commercial model also allow PyPI and npm, and the free boards allow only the model server. These are the named boards' lists today:

Installing the agent before a trial, and running the tests after it, use the task's own network settings, because both download packages. The allowlist covers only the time the agent spends on the task.

An open run's agent could reach whatever the task's own network settings allowed, which for most tasks is the internet, so it could look answers up. Read an open board's scores knowing that.

hbench applies the list on the participant's machine. If it finds that Docker there can't enforce the list, it stops with an error rather than run the agent on an open network. Each trial records whether the list was enforced, and that's what the site reads.

hbench versions, results epochs and the method version

Any hbench release from 0.3.0 on can run a benchmark, and each run records the release it ran.

A board's name and slug stay tied to its cell. A run with a different model, task set, attempts or limits is in a different cell, so it goes on a different board instead of changing this one's scores.

A new hbench version can change how trials run or are graded. When it does, it starts a new results epoch: scores from before and after can't be compared, and a board ranks only the runs of the newest epoch it has. That's why a citation should give the hbench version.

The last reset was on 18 October 2026, when hbench 0.2.0 added the controls above and every result from earlier versions was taken off the boards.

Within one results epoch, every hbench release is treated as compatible: a patch inside an epoch doesn't change how tasks run or are graded. That's an assumption, not a proof. The nightly smoke test runs a real job on the current release and checks it against what the site reads, and a release that breaks the assumption starts a new epoch.

A dataset version that only re-points its tasks' images, at the same image digests, keeps its board: terminal-bench@2.0 and terminal-bench@2.0.1 are one dataset, so a run of either joins the same board.

The method has a version, v1 since 29 October 2026, and every row of /api/results carries it. Any change to how a score, interval, rank or figure is computed bumps the version and adds a line below. Scores are recomputed from the stored trials, and no board resets: an hbench results epoch resets boards, a method version doesn't. A citation gives the board, the hbench version and the method version. The scoring module is served as it runs, so with the per-task rows of /api/compare anyone can recompute a board.

Method changelog

• v1, 29 October 2026. The headline is pass@1. The rank range comes from the paired test, and tiers, ≈ and "provisional" go. Each board states its resolution. The method version and the scoring module's source are published. The interval's random draw also changed, to one draw shared by every pair with the same number of shared tasks, so intervals move slightly; that's a generator change, not a change of method.
• Before v1, two rules changed scores without a version to cite: from 26 October 2026, infrastructure failures count as 0 and a publish with more than 10% of them is refused; and from five full runs on, the highest- and lowest-scoring runs are left out.

What we can't check

• Runs aren't run again, so a consistent forgery passes. Someone who runs or builds hbench themselves can write a job whose manifest, locks, results and trajectories all agree on rewards no agent earned, and it would pass every check above. That's why every result is self-reported, the site records its level, and you can report one that looks wrong.
• We can't see which model served a self-run, or which other models a harness called. Each trial records the model name, and the board is chosen by that name. A different model, or another quantisation, served under that name would pass. A harness can also call models other than the board's: Claude Code hands parts of a task to subagents, and some harnesses pass a hard task to a larger model. A trial records only the model the agent was given. On an allowlist board the agent reaches only the hosts on the list, so it can call only the models those hosts serve. On an open board it can call any model it has a key for.
• Trajectories are published only by their owner's choice. A trajectory records everything the agent did and saw in a trial, including the participant's prompts and code and any part of their model key the redaction missed, so it stays private unless the run's owner makes it public. Public trajectories describes the scan each one passes first. Your own are always in Your bench.
• The task sets are public, and nobody has reviewed some datasets. A board's model may have seen some tasks or their solutions in training. That affects every run on a board equally, but it means scores aren't held-out results. A dataset outside the task catalog is labelled unreviewed: nobody has read its tasks, which could be easy, broken, or copies with the answers in them. Its boards say nothing about the benchmark they're named after, and they're never featured.
• Machines differ. Speed and model serving differ between machines, and that changes how much an agent gets done before its time limit. Two runs on one board can differ for that reason alone.