Agentic Thunderdome
Two agents enter, one agent leaves.
Agentic Thunderdome benchmarks AI coding tools against 19 standardized programming tasks in isolated Docker containers. Each orchestrator gets a task prompt, a workspace, and a time limit. Scoring is deterministic -- automated tests and static analysis, no LLM judges. The dataset currently on disk spans 2,991 scored trials (4,442 including crashes) across 121 orchestrator variants.
Results
Composite scores ranked by Overall (average of Standard and Hard suite means). Crash trials (cost=$0, or duration <15s for local models) excluded. Entry requires 8+ standard AND 8+ hard non-crash trials. Tables are generated by scripts/gen-scores.py from the meta.json files in results/.
Scoring corrections, 2026-07-24 — numbers below are NOT comparable to earlier revisions of this README. Two scorer bugs were fixed and all affected history was re-scored:
- Lint (
td-4nq). npm 12.0.1's release made node:20's bundled npm print an update notice on every script run. A cleannpm run linttherefore emitted output with zero diagnostics, which tripped a conservative "unrecognized formatter" branch and scored 0.5 instead of 1.0. Because standard composite =tests*0.7 + static_analysis*0.3, every flawless standard task was capped at 0.85 from 2026-06 onward — roughly 8pp off any run in that window. Older runs were unaffected, so pre- and post-June numbers were silently incomparable.- Hidden-test entry points (
td-kiz, plus earlier T2/T5/T8 work). Several graders hard-coded an entry point the task spec never required, so spec-compliant implementations failed to load and cascaded to zero. See "Scoring" below.Net effect: most scores rose, the standard suite is now genuinely saturated (~92-95% for frontier models) rather than artificially capped, and all remaining separation between strong contenders is in the hard suite. Rows that no longer meet the 8+/8+ entry bar were dropped rather than carried forward with unverifiable numbers.
Cost-accounting corrections, 2026-08-02. Five per-adapter pricing defects were found and fixed this cycle;
pricing.yamlhas no entries for third-party models, so each adapter's hardcoded rate is the only source of truth and nothing downstream validates it. (1) Adapters cloned from another model inherit the SOURCE model's rates -- a GPT-5.6 Luna arm billed at Kimi K3's $3/$15 and reported 30x its true cost. (2) Rates go stale as providers cut prices -- GLM-5.2 billed $1.40/$4.40 against a live $0.97/$3.04. (3)cache_creation_input_tokensmust count as input (verified against OpenRouter/api/v1/generation); omitting it understated one review 55x. (4) A harness-reportedtotal_cost_usdis meaningless against a non-native backend -- Claude Code priced a $0.08 DeepSeek review at $4.82, and the same bug inflated the Qwopus+DeepSeek row from $0.146 to a long-quoted $0.578. (5) DeepSeek Flash carried 2026-05 pricing at half the current rate. Historical trials keep their recorded figures; only post-correction runs are right. Verify any $/task against the provider's live listing before relying on it.
Tools You Can Use
Third-party orchestrators and harnesses — things you can install and run today.
| Rank | Orchestrator | Overall | Standard | Hard | Trials | $/task | Model |
|---|---|---|---|---|---|---|---|
| 1 | Conclave Review (Opus 5) | 91.7% | 92.9% | 90.6% | 67 | $5.71 | Opus 5 |
| 2 | Conclave Review | 88.3% | 88.4% | 88.2% | 141 | $1.83 | Opus 4.6 |
| 3 | BMAD-METHOD | 87.9% | 85.8% | 89.9% | 57 | $1.74 | Opus 4.6 |
| 4 | Conclave v10 Routed (duo) | 87.3% | 86.1% | 88.6% | 38 | $0.25 | Haiku -> Opus/Sonnet |
| 5 | Conclave v6 (Sonnet) | 87.0% | 86.8% | 87.2% | 84 | $1.11 | Sonnet 4.6 |
| 6 | Conclave v8 (Opus) | 86.7% | 85.2% | 88.3% | 37 | $1.29 | Opus 4.8 |
| 7 | Conclave v10 Routed (trio) | 86.6% | 84.5% | 88.7% | 38 | $0.90 | Haiku -> Opus/Sonnet |
| 8 | Conclave v6 (Opus) | 86.2% | 87.2% | 85.2% | 76 | $2.12 | Opus 4.6 |
| 9 | Gas Town | 83.6% | 80.9% | 86.3% | 61 | $2.38 | Opus 4.6 |
| 10 | GSD | 83.5% | 85.6% | 81.5% | 57 | $1.13 | Opus 4.6 |
| 11 | Gemini CLI | 81.3% | 81.8% | 80.8% | 60 | $0.14 | Gemini 2.5 Pro |
| 12 | ExoMonad | 74.8% | 66.8% | 82.9% | 22 | $0.84 | Opus 4.6 + Gemini |
Note the model column: every framework below except row 1 is still measured on Opus 4.6 or Sonnet 4.6 — one to two model generations behind the vanilla baselines in the next table. Row 1 is the same Conclave Review scaffold re-run on Opus 5 specifically to separate the harness contribution from the model generation; see the harness-vs-model section below for the decomposition (+3.4pp from the model, -0.7pp from the scaffold, at n=3).
Rows carried by earlier revisions that no longer meet the 8+/8+ entry bar (Conclave v7 Lite, v7 Double Review, ExoMonad v2, Conclave v8 on Sonnet 4.6) were dropped rather than reprinted with numbers the current results/ can't reproduce.
Harness and Model Tests
Same harness with different models, or same model through different harnesses. Tests which component drives performance.
Headline figures in this table were regenerated 2026-07-24. Per-row prose may still quote per-task deltas computed before the lint and hidden-test corrections; treat the bolded Overall/Standard/Hard numbers as current and the inline task-level deltas as historical.
| Orchestrator | Overall | Standard | Hard | Trials | $/task | Model |
|---|---|---|---|---|---|---|
| Claude Code + Opus 5 (OAuth) | 92.4% | 93.3% | 91.4% | 63 | $2.37 |
claude-opus-5, vanilla headless -p OAuth. The top score on the board, above every orchestration framework -- see the harness-vs-model section below for why that comparison is confounded. n=3: 21/21 tasks completed, zero crashes, zero auth failures. Exceptionally low variance -- 7 of 21 tasks scored identically across all three runs, most sd <= 0.016; only analytics-dashboard is genuinely noisy (sd 0.142). Beats the flag-matched Opus 4.8 baseline by ~1.6pp overall, and the entire gain is hard-suite (91.4 vs 89.6) -- Standard is a tie. Costs ~2.7x the tokens and ~2.2x the wall clock of 4.8 for that gain. |
| Claude Code + Opus 4.8 (OAuth) | 91.1% | 92.5% | 89.6% | 26 | $1.13 |
claude-opus-4-8 pinned, vanilla headless -p OAuth, hermetic flags. The apples-to-apples baseline for Opus 5 -- same adapter, only the model id differs. NOTE: an older claude-code-oauth-opus-48 entry (rank 17, 85.5%) uses the floating --model opus alias without hermetic flags; it is not a valid 4.8 baseline, and because the alias floats it would no longer even run 4.8 today. |
| DeepSeek v4 Flash GA (solo) | 90.7% | 93.6% | 87.5% | 21 | $0.096 |
deepseek-v4-flash via DeepSeek's first-party /anthropic endpoint, vanilla Claude Code. The value result on this board: 2.6pp under Opus 5 at 1/25th the cost -- 23x better cost-per-point ($0.0011 vs $0.0253). Beats Kimi K3 by 3.3pp at 26x less. Crucially it is the ONLY cheap model tested that does NOT collapse on hard (87.5 vs Luna's 72.7, K3's 81.1); it scored 0.832 on circuit-debugger, the task that ground K3 to 0.08 over 150 minutes. 21/21, zero crashes, $2.01 for the whole suite -- of which circuit-debugger alone was |