📊 An honest benchmark
A tool-vs-tool comparison — Polypus, OpenCode and Aider —
running the same model (deepseek-v4-flash), on the
same tasks, on the same machine. The verdict is objective (real unit
tests), and cost is measured from a neutral source (the OpenRouter API,
a dedicated key per tool) — never what each tool self-reports. Philosophy:
diagnosis first, marketing second. The original goal was to find where
Polypus breaks — and it did (see the fixed bugs below).
Methodology
| Model | deepseek-v4-flash (via OpenRouter) — the same for all three, single provider (no routing variance). |
| Verdict | node --test — real unit tests. Partial score X/N (how many subtests passed), not just pass/fail. |
| Cost | Spend delta on the OpenRouter API, one dedicated key per tool. Neutral source, immune to self-reporting. |
| Isolation | Each task runs in a disposable git workspace. Identical prompt for all three. |
| Anti-cheat | Test files are hashed before/after: if a tool edits the test to "pass", the run is disqualified. |
| Reproducible | Open scripts in bench/demo/ in the repo. Run ./run-all.ps1. |
A task is a folder with a PROMPT.md (identical instruction), the source
file(s) and *.test.mjs. Tools get only the prompt and the workspace — they
figure out the rest themselves. Tasks range from simple algorithms (leap year) to
debugging (the code ships already broken) and multi-file
(the bug is in one of two files and the tool has to find which).
Results per task
| Task | Type | Polypus | OpenCode | Aider |
|---|---|---|---|---|
| leap | leap year | 7/7 | 7/7 | 7/7 |
| rle | encode/decode | 12/12 | 12/12 | 12/12 |
| roman | edge cases | 14/14 | 14/14 | 14/14 |
| anagram | case + traps | 8/8 | 8/8 | 8/8 |
| fix-flatten | debugging | 8/8 | 8/8 | 8/8 |
| stats-bug | multi-file | 7/7 | 7/7 | 7/7 |
Correctness: a perfect tie. At this difficulty, with a strong model, all three tools solve everything. On the multi-file task all three found the right file and left the other one (the "decoy") untouched. Where they diverge is efficiency.
leap— "is it a leap year?" by the Gregorian rules (÷4, except ÷100, except ÷400). Short algorithm.rle— run-length encoding: compress"AAAB"→"3A1B"and decompress back.roman— integer → Roman numeral (1–3999) with subtractive notation (4=IV, 9=IX). Full of edge cases.anagram— given a word and candidates, find the anagrams (case-insensitive, excluding the word itself).fix-flatten— debugging: the nested-array flatten code ships BROKEN; the tool has to find and fix the bug.stats-bug— multi-file: the bug is in a function that imports from another file; the tool must discover which of the two holds it.
7/7— subtests passed / total (partial score, not just pass/fail).- In the token table: total tokens processed on the task (less = more efficient).
Round cost (6 tasks)
Tokens processed per task
| Task | Polypus | OpenCode | Aider |
|---|---|---|---|
| leap | 18,378 | 27,095 | 1,067 |
| rle | 26,902 | 19,662 | 2,400 |
| roman | 18,820 | 21,058 | 1,474 |
| anagram | 25,035 | 28,152 | 1,622 |
| fix-flatten | 26,958 | 36,063 | 2,500 |
| stats-bug (multi) | 63,496 | 45,118 | 1,827 |
The honest nuance behind the numbers. Aider is ~25× cheaper — but part
of that is because it makes you say which files to edit; it doesn't navigate, it
gets everything spoon-fed and solves in one shot. Polypus and OpenCode are real
agents: they read the repo, run the tests and iterate on their own — and they pay
for it in tokens. This is starkest on the multi-file task (stats-bug): Polypus
spent 63k tokens discovering which file held the bug; Aider spent 1.8k because it was handed
both files. These are different categories of tool — a cost comparison has to account for
that.
What the data says
✅ Equivalent correctness
Polypus delivers the same hit rate as established tools, on the same model. The "equivalent quality" thesis holds up in the data — it's not an opinion.
💸 On par with the agentic competitor
Against OpenCode (the direct agentic competitor), Polypus costs the same or less — the prompt-caching advantage shows up once the measurement is honest.
🧭 Correct multi-file navigation
On the task with a scattered bug, Polypus found the right file and never touched the decoy — autonomous navigation the one-shot approach can't do without human help.
🎯 A clear improvement target
Polypus's weak flank is token efficiency, especially multi-file. It's a concrete, measurable gap — a roadmap item, not an excuse.
The benchmark found 4 real bugs — and we fixed them
The value of an honest benchmark isn't the pretty chart — it's what it breaks. In the first rounds Polypus failed, and each failure became a fix:
- Tool-mode fallback — OpenRouter routing sometimes stopped emitting native tool calls and the agent returned prose instead of editing. → emulated mode (immune) + issue opened
- CDATA leak — the emulated-mode parser wrote
<![CDATA[…]]>into the file, breaking the code. → fixed + regression test - Finish with a red test — the agent ran the test, it failed, and it declared "done" anyway. → finish-gate rejects finish after a failed command
- Broken cost attribution — discovered Aider ignored the dedicated key (used an internal OAuth key), invalidating its cost measurement. → key forced via flag, neutral measurement restored
Methodological honesty
This is a small, transparent benchmark, not a definitive leaderboard. Limits we took on purpose: (1) one model this round; (2) 6 self-contained tasks — enough for diagnosis, not for a universal ranking; (3) Aider is handed the files explicitly, which makes its cost cheap in exchange for requiring a human in the loop; (4) tasks at this difficulty don't separate the tools on correctness — for that, the next step is a weaker model or larger tasks. Everything is reproducible: the scripts, the tasks and the criteria are in the repo.
Want to reproduce it? cd bench/demo and ./run-all.ps1. Each tool
uses its own OpenRouter key; the scoreboard lands in runs/results.csv and the
neutral cost in ./show-costs.ps1. Open source:
github.com/GaberRB/polypus.