Reference
LiveCodeBench v6: checklist assessment
How LiveCodeBench v6 measures up to the Agentic Benchmark Checklist, item by item, as the maintainers assessed it. 100 contest problems from release v6, each graded by hidden tests.
Reviewed 2026-10-29 by the maintainers, against the checklist in the dataset checklist. Source: laude-institute/harbor-datasets datasets/livecodebench/release_v6 at cfad8d97c75fb9e28c7b6b74b7fadb23dae57ab6, public, no licence file in the repository. 100 problems sampled upstream from LiveCodeBench's release v6: contest problems from LeetCode, AtCoder and Codeforces, each tagged with its release date and graded by hidden tests. Difficulty, as upstream labels it: easy 29, medium 33, hard 38. Run with hbench 0.3.9.
Task validity
| Item | Answer | Evidence |
|---|---|---|
| T.4 | yes | A fresh container per trial (hbench). |
| T.5 | yes | Tests removed before the agent starts, copied in after (hbench). |
| T.6 | yes | Tasks pinned by digest and checksum in the task catalog. |
| T.7 | not assessed | Problem statements and tests are upstream's; the 100-problem sample was drawn upstream, not by the maintainers. |
| T.8 | not assessed | |
| T.9 | yes | Every task ships a solution; hbench's oracle agent runs it. |
| T.10 | not assessed |
Score: 4 of 7.
Outcome validity
| Item | Answer | Evidence |
|---|---|---|
| O.b.3 | yes | Hidden tests, with inputs the agent doesn't see. |
| O.d.1 | not assessed | Upstream's tests, partly generated; not read problem by problem. |
| O.d.2 | not assessed | |
| O.f.2 | yes | Deterministic tests under a time limit; a slow machine can still turn a pass into a timeout. |
Score: 2 of 4.
Benchmark reporting
| Item | Answer | Evidence |
|---|---|---|
| R.1 | yes | Tasks public; hbench Apache-2.0. |
| R.3 | yes | Each problem's release date lets a reader check it against a model's training cutoff; the site doesn't filter by date. |
| R.5 | yes | The benchmark page says what it measures: algorithmic problems under tight time limits. |
| R.7 | yes | The methodology's "what we can't check"; the FAQ says most problems predate DeepSeek-Flash's data. |
| R.9 | no | |
| R.10 | yes | 95% interval per row, paired test between rows. |
| R.11 | yes | The methodology. |
| R.12 | no | No reference harness named. |
Score: 6 of 8.
Reading a board on it
Rewards reasoning on a single problem under a time limit, not work in a large codebase. Treat the problems as public tasks like the rest unless you check their dates against the model's cutoff.
Reviewed on October 29, 2026.