Skip to content
Hbench
hbench

humaneval-plus@1.0 (unreviewed) on GPT-6.1 Sol

164 tasks, 1 attempt, open network. Updated 24 Oct 2026.

164 tasks from humaneval-plus@1.0 on ghcr.io. Nobody has reviewed this dataset, so a score here says nothing about the benchmark it's named after.

Build a run for this board

Value

What a task cost each configuration. Claude Code used 2.1× the tokens of DeepSeek Harness, and 29% less time per task.

$ per solved task

Lower is cheaper. Dollars per task divided by the pass rate.

Embed this chart

Harness$ per solved task
Codex0.164.0$0.02$0.02 a task
Claude Code2.1.309$0.08$0.07 a task
Not reported: DeepSeek Harness

Tokens per task

Lower is leaner. Input plus output, cached input included.

Embed this chart

HarnessTokens per task
DeepSeek Harness0.2.1-rc.2 · Reference20k×1.00
Claude Code2.1.30940k×2.06
Codex0.164.041k×2.11

Agent time per task

Lower is faster. Agent start to finish; setup and tests left out.

Embed this chart

HarnessTime per task
Codex0.164.019 s×0.62
Claude Code2.1.30922 s×0.71
DeepSeek Harness0.2.1-rc.2 · Reference31 s×1.00

Pass rate against tokens per task

Up and to the left is better. Pass-rate axis starts at 90%.

Embed this chart

90%95%100%10k20k30k40k50kTokens per task, input and outputMore tokens for a lower pass rate than DeepSeek HarnessDeepSeek Harness0.2.1-rc.2 · 20k · 96.3%Claude Code2.1.309 · 40k · 95.7%Codex0.164.0 · 41k · 95.7%
  • On the frontier: no other row has a higher pass rate for less tokens
  • Off the frontier
  • 95% interval of the pass rate
On the frontier, least tokens first: DeepSeek Harness 0.2.1-rc.2, 20k tokens a task, 96.3%.
  • 3 runs
  • Updated 24 Oct 2026
  • Method