Reference
Aider Polyglot: checklist assessment
How Aider Polyglot measures up to the Agentic Benchmark Checklist, item by item, as the maintainers assessed it. 225 Exercism exercises in six languages, each graded by Exercism's tests.
Reviewed 2026-10-29 by the maintainers, against the checklist in the dataset checklist. Source: laude-institute/harbor-datasets datasets/aider_polyglot at f30b14415dd733c83627204bad0af69a89ceb46f, public, no licence file in the repository. 225 Exercism exercises in C++, Go, Java, JavaScript, Python and Rust, the ones Aider's own runs found hardest, each graded by Exercism's tests. The catalog labels every task medium. Trials get 4 GB. Run with hbench 0.3.9.
Task validity
| Item | Answer | Evidence |
|---|---|---|
| T.4 | yes | A fresh container per trial (hbench). |
| T.5 | yes | Tests removed before the agent starts, copied in after (hbench). |
| T.6 | yes | Tasks pinned by digest and checksum in the task catalog. |
| T.7 | not assessed | Exercism's exercises and solutions; not re-read by the maintainers. |
| T.8 | not assessed | Six language toolchains in the images; not rebuilt by the maintainers. |
| T.9 | yes | Every task ships a solution; hbench's oracle agent runs it. |
| T.10 | not assessed |
Score: 4 of 7.
Outcome validity
| Item | Answer | Evidence |
|---|---|---|
| O.b.3 | yes | The exercise has to pass its tests. |
| O.d.1 | not assessed | Exercism's community-maintained tests; not read task by task. |
| O.d.2 | not assessed | |
| O.f.2 | yes | Deterministic unit tests, no network or services. |
Score: 2 of 4.
Benchmark reporting
| Item | Answer | Evidence |
|---|---|---|
| R.1 | yes | Tasks public; hbench Apache-2.0. |
| R.3 | no | Exercism exercises are public and widely copied. |
| R.5 | yes | The benchmark page says what it measures: editing code reliably in six languages. |
| R.7 | yes | The methodology's "what we can't check". |
| R.9 | no | |
| R.10 | yes | 95% interval per row, paired test between rows. |
| R.11 | yes | The methodology. |
| R.12 | no | No reference harness named. |
Score: 5 of 8.
Reading a board on it
Aider's own leaderboard uses this set with Aider fixed and models varied; a board here holds the model fixed and varies the harness, so the two don't compare. The named board on DeepSeek-Flash takes no uploads.
Reviewed on October 29, 2026.