Reference
HumanEvalFix: checklist assessment
How HumanEvalFix measures up to the Agentic Benchmark Checklist, item by item, as the maintainers assessed it. 164 Python functions with a planted bug, each graded by HumanEval's tests.
Reviewed 2026-10-29 by the maintainers, against the checklist in the dataset checklist. Source: laude-institute/harbor-datasets datasets/humanevalfix at ab02ff13250fae8d91b93a6e4c11ce0bdcb78215, public, no licence file in the repository. 164 Python functions from HumanEvalPack (Muennighoff et al., 2023), each with one bug planted by hand, graded by HumanEval's tests for that function. Difficulty, as upstream labels it: easy 77, medium 87. Named subset sample-40, the free boards' task set. Run with hbench 0.3.9.
Task validity
| Item | Answer | Evidence |
|---|---|---|
| T.4 | yes | A fresh container per trial (hbench). |
| T.5 | yes | Tests removed before the agent starts, copied in after (hbench). |
| T.6 | yes | Tasks pinned by digest and checksum in the task catalog. |
| T.7 | not assessed | The planted bugs are upstream's; not re-read by the maintainers. |
| T.8 | not assessed | |
| T.9 | yes | Every task ships the fixed function; hbench's oracle agent runs it. |
| T.10 | not assessed |
Score: 4 of 7.
Outcome validity
| Item | Answer | Evidence |
|---|---|---|
| O.b.3 | yes | The function has to pass its tests. |
| O.d.1 | no | HumanEval's tests are few per function and accept some wrong solutions (EvalPlus, Liu et al., 2023); they weren't strengthened here. |
| O.d.2 | no | EvalPlus measured the gap; no metric is applied here. |
| O.f.2 | yes | Deterministic unit tests, no network or services. |
Score: 2 of 4.
Benchmark reporting
| Item | Answer | Evidence |
|---|---|---|
| R.1 | yes | Tasks public; hbench Apache-2.0. |
| R.3 | no | HumanEval has been public since 2021 and is widely used in training. |
| R.5 | yes | The benchmark page says what it measures: repairing one buggy function until its tests pass. |
| R.7 | yes | The methodology's "what we can't check". |
| R.9 | no | No measurement of contamination's effect here. |
| R.10 | yes | 95% interval per row, paired test between rows. |
| R.11 | yes | The methodology. |
| R.12 | no | No reference harness named. |
Score: 5 of 8.
Reading a board on it
Short tasks that a 7B model solves some of the time, so differences between prompts and agent settings show. A high score says the harness drives the model through small repairs, not that it works in a codebase.
Reviewed on October 29, 2026.