HumanEvalFix on Qwen2.5-Coder 7B
sample-40, 40 tasks, 1 attempt, allowlisted network. Updated 18 Oct 2026.
Python bug-repair tasks, short enough to iterate on. A 7B model can solve some, so differences between prompts and agent settings show up. No API key.
Value
What a task cost each configuration.
$ per solved task
Lower is cheaper. Dollars per task divided by the pass rate.
Tokens per task
Lower is leaner. Input plus output, cached input included.
HarnessTokens per task
Agent time per task
Lower is faster. Agent start to finish; setup and tests left out.
HarnessTime per task
Pass rate against tokens per task
Up and to the left is better.
The frontier needs two configurations that report tokens, time or dollars.