Reading · Developers Digest

This site does not allow itself to be embedded. You can still read it on the original site — the toolbar below keeps your place in the directory.

A Huawei-Queen's study finds open coding models fine-tuned under OpenHands degrade sharply under other scaffolds - SWE-Lego-Qwen3-32B drops from 52.6% to 8.4% Pass@1 on OpenCode. The fix: train planning as a model capability, not a scaffold artifact.

Open coding agents have converged on a single training environment, and that is quietly breaking them everywhere else. Trajectory datasets used to fine-tune open models - SWE-Gym, Nebius, SWE-Lego, CoderForge - are collected almost exclusively under OpenHands. The models trained on those traces score well under OpenHands and degrade substantially anywhere else. A new paper from Huawei Canada and Queen's University measures the gap, isolates the cause, and shows it can be fixed by training planning as a model capability instead of relying on the harness to supply it. The headline number is brutal. SWE-Lego-Qwen3-32B scores 52.6% Pass@1 on SWE-bench Verified under OpenHands, the scaffold it was trained under. Deployed under OpenCode, it collapses to 8.4%. The same model family's untrained base, Qwen3-32B, shows no such divergence: 29.0% under OpenHands, 23.2% under Claude Code, 18.4% under OpenCode, 8.0% under mini-swe-agent. The base model's spread across scaffolds is far smaller than the fine-tuned model's, which tells you the gap is installed by fine-tuning, not fixed by the scaffold interface. ## The two kinds of planning The paper's central claim is that what fine-tuning installs is a scaffold's planning conventions, in two distinct senses. Explicit planning is the pre-execution step where a model produces a plan as a first-class artifact before acting. Claude Code has this as Plan Mode, OpenCode as its Plan agent. OpenHands distributes planning across its CodeAct cycle instead of concentrating it in a dedicated stage, and mini-swe-agent, a deliberately minimal ~100-line bash-only scaffold, has no explicit planning structure at all. Implicit planning is the structural behavior every scaffold imposes on the agent loop turn by turn: how work decomposes into sub-steps, when exploration gives way to action, how tool calls sequence, how failures trigger replanning. A model trained under one scaffold learns that scaffold's blend of the two, and deployment under another exposes the mismatch. Underneath the surface differences, all CLI scaffolds share a ReAct-style act/observe loop, so the capacity is there - what the model lacks under a non-training scaffold is familiarity with its planning conventions. ## DCAS: decoupling scaffold from model To test the hypothesis, the authors built DCAS, a backend-substitution interception layer that routes API traffic between any CLI scaffold and any backend model without modifying the scaffold. That unlocks three things the ecosystem did not have: controlled cross-scaffold evaluation of the same backend model, trajectory collection that captures both senses of planning under any scaffold, and fine-tuning on those trajectories without touching the scaffold itself. The experiments run Qwen3-Coder-30B-A3B-Instruct as the executor under Claude Code 2.0.76, with SWE-bench Verified as the benchmark and a 100-turn cap. The first research question isolates plan quality. With no planning step, the model scores 42.8%. Let the model plan for itself: 48.2%. Plug in an open-weight planner (Qwen3-Coder-480B-A35B): 49.2%. Supply a frontier plan - Claude Sonnet 4.5 - and the same executor, same scaffold, same benchmark jumps to 57.8%, a 15-point swing attributable entirely to the plan. Plan quality scales with planner capability, and the swing exceeds the cross-scaffold drops the paper measures (SWE-Lego's 8.4-point drop from OpenHands to Claude Code). Notably, Sonnet 4.5 beats Opus 4.5 as a planner here, which the authors attribute to plans better calibrated to the executor's capability profile. The takeaway: for a fixed executor model, the plan you hand it can be worth more than the model itself. ## Planning can be trained in The second question is whether planning can be internalized. The authors fine-tune the same 30B executor on 576 two-phase trajectories collected under Claude Code via DCAS, using GLM-4.7 as the trajectory source model - deliberately not a frontier model, so any gain comes from the scaffold's planning conventions rather than distillation. Full-parameter SFT with LLaMA-Factory, 65K context, BF16. Two dataset variants decompose the result. PlanOnly training, which captures implicit planning conventions alone, gets 53.8% Pass@1 with no planning step and gains nothing extra from a self-plan at inference - the implicit conventions are now baked into turn-by-turn behavior. Plan+Exec training, which captures both senses, gets 52.8% no-plan plus 3.0 points under self-plan, landing at 55.8% - matching or approaching the 57.8% of an external frontier planner without needing one at inference time. The third question checks whether the capability generalizes. The fine-tuned model improves to 57.2% under a newer release of the training scaffold (Claude Code 2.1.73), and gains consistently on scaffolds it never saw during training: +3.4% on OpenCode and +7.0% on mini-swe-agent under self-plan. The learned behavior is structural, not scaffold-specific memorization. ## What this means for your fine-tune The practical consequences land in two places. First, if you fine-tune open coding agents, your fine-tune is scaffold-locked by default. A model that crushes benchmarks under OpenHands can lose 80% of its performance under a different CLI - and practitioners choose scaffolds on cost, licensing, latency, and data-privacy constraints, not on which scaffold their preferred open model was trained under. The paper's path forward is to train planning as a structural skill on trajectories collected under scaffolds that expose the conventions you want, which is exactly what DCAS enables, with the weights and trajectory data released publicly. Second, even without fine-tuning, the RQ1 result is a free lunch for anyone running open models in CLI scaffolds: plan quality is worth more than executor choice in this regime. If your 30B model is underperforming, the fastest lever may not be a bigger model - it may be a stronger planner supplying the plan, or simply enabling the scaffold's planning phase. The boundaries are stated honestly. Every experiment uses SWE-bench Verified and one executor model, so the magnitude of the gains may not transfer to other scales or task types. Cross-scaffold evaluation covers OpenCode and mini-swe-agent, not Codex CLI or Gemini CLI. Pass@1 does not capture turn efficiency, and training kept only successful trajectories, which biases toward easier instances. Claude Code's closed-source nature means the scaffold itself can change under the results, which is why the authors pin exact versions and release raw trace logs. The bigger idea is worth sitting with: the open model ecosystem bet everything on one harness, and the harness became part of the model. If planning conventions can be moved from scaffold artifact to learned capability, then the next open fine-tune can be scaffold-portable - and that changes what "open" means for coding agents, because the model you train is finally the model you can run anywhere. ## Continue Reading - [The $500 RL Fine-Tune That Beats Frontier Models](/blog/500-dollar-rl-fine-tune-beats-frontier-models) - how far small-budget fine-tuning on open models gets today - [Why Software Factories Fail: Harness Engineering](/blog/software-factories-fail-harness-engineering) - the harness as the real product, and what that means for agent fleets - [Pi: A Minimal Harness and the Cost Per Task](/blog/pi-minimal-harness-cost-per-task-hn-analysis) - what a minimal scaffold actually buys, and what it hides - [Long-Running Agents Need Harnesses](/blog/long-running-agents-need-harnesses) - why the loop around the model matters more than the model - [GLM 5.2 vs DeepSeek V4 vs Qwen3: Open-Weights Coding Showdown](/blog/glm-5-2-vs-deepseek-v4-vs-qwen3-open-weights-coding-showdown) - the current open-weights field the fine-tunes are built on ## Sources - [DCAS: Decoupling CLI Agent Scaffolding to Internalize Planning across Scaffolds (arXiv:2608.06113)](https://arxiv.org/abs/2608.06113) - abstract page, submitted Aug 6, 2026, ASE '26 - [Full paper PDF](https://arxiv.org/pdf/2608.06113v1) - Tables 1, 4, RQ2/RQ3 results, and validity discussion - [ASE '26 conference record](https://doi.org/10.1145/3832783.3834485) - ACM DOI for the accepted version

DCAS: Why Fine-Tuned Coding Agents Fall Apart When You Switch Scaffolds

Comments