Layered Loops: Why 764 AI Agent Sessions Still Needed 21 Human Interventions

The orchestrator is a while read loop. That is the actual answer when someone asks how I ran AI agents across 98 models and ~161 fixture and test files to migrate model tests from RSpec to Minitest. The shell script is maybe 100 lines. It reads a list of files and calls a Claude command for each one, sequentially. The layers of error handling below that loop determined whether the project succeeded or stalled.

The previous article covered the pipeline itself: the command file, agent definitions with I/O contracts, gate types, and the Clean Room context strategy. This article covers what happens when you take that pipeline and run it across an entire codebase: two batch orchestrators, their aggregate execution data, and the layered failure handling that determined the real intervention rate.

flowchart LR
    DISC["Discovery Script"] --> LOOP["Shell Loop"]
    LOOP -->|"claude -p"| PIPE["6-Gate Pipeline"]
    PIPE --> FIX{"/fix-tests?"}
    FIX -->|"pass"| COMMIT["Git Commit"]
    FIX -->|"fail, retry"| PIPE
    COMMIT --> HUMAN{"Human Layer"}
    HUMAN -->|"~85% pass (model level)"| DONE["Done"]
    HUMAN -->|"~15% intervene"| LOOP

    style DISC fill:#2a5f8f,stroke:#4a9eed,color:#ffffff
    style LOOP fill:#2a5f8f,stroke:#4a9eed,color:#ffffff
    style PIPE fill:#1e6e45,stroke:#4ae68a,color:#ffffff
    style FIX fill:#7a5c2e,stroke:#e6b44a,color:#ffffff
    style COMMIT fill:#5a2e7a,stroke:#b44ae6,color:#ffffff
    style HUMAN fill:#8b3a3a,stroke:#e06060,color:#ffffff
    style DONE fill:#1e6e45,stroke:#4ae68a,color:#ffffff

Two orchestrators ran 764 Claude sessions across ~259 unique files over 16 working days. Ralph migrated 98 models through the 6-gate pipeline, producing close to 10,000 tests in 283 sessions. Felix cleaned up the fixture debt Ralph left behind: enum corrections, consolidation, and YAML anchors across ~161 unique fixture and test files in 481 sessions. Between them, 21 problems required a human to step in: an ~85% autonomous rate at the model level (where go/no-go decisions happened) and ~92% at the file level. Each layer caught a different category of failure, so the rate that reached the human stayed low even when individual layers failed often.

Batch Orchestrating RSpec to Minitest Migration

flowchart TD
    subgraph Discovery["Discovery Phase"]
        direction LR
        D1["Ruby script finds untested models"] --> D2["Sort by size, largest first"]
    end

    subgraph Loop["Main Loop"]
        direction LR
        L1["/minitest-migration, 6-gate pipeline"] --> L2{"Pass?"}
        L2 -->|Yes| L3["Commit"]
        L2 -->|No| L4["/fix-tests, max 3 retries"]
        L4 --> L3
    end

    subgraph Output["Per-Batch Output"]
        direction LR
        O1["Git commit per component"] --> O2["PR with all migrations"]
    end

    Discovery --> Loop
    Loop --> Output

    style D1 fill:#2a5f8f,stroke:#4a9eed,color:#ffffff
    style D2 fill:#2a5f8f,stroke:#4a9eed,color:#ffffff
    style L1 fill:#1e6e45,stroke:#3cb371,color:#ffffff
    style L2 fill:#7a5c2e,stroke:#d4a04a,color:#ffffff
    style L3 fill:#1e6e45,stroke:#3cb371,color:#ffffff
    style L4 fill:#8b3a3a,stroke:#e06060,color:#ffffff
    style O1 fill:#5a2e7a,stroke:#9b59b6,color:#ffffff
    style O2 fill:#5a2e7a,stroke:#9b59b6,color:#ffffff
    style Discovery fill:#1e3a5f,stroke:#4a9eed,color:#e0e0e0
    style Loop fill:#1a3d2e,stroke:#3cb371,color:#e0e0e0
    style Output fill:#2e1a3d,stroke:#9b59b6,color:#e0e0e0

Ralph is the first layer. The name comes from Geoffrey Huntley's Ralph Wiggum Loop: run Claude in a while true loop, let it fail, let it retry, let naive persistence do the work. My version adds structure beyond that: discovery scripts, a 6-gate pipeline, bounded retries, and commit-after-each-component. But the core pattern is the same. It processes one model at a time: discover which models lack Minitest coverage, run the /minitest-migration pipeline for each, handle failures with a bounded retry loop, commit after each component, and move on.

Discovery: Finding What to Migrate

The loop starts with a Ruby discovery script that scans the codebase for models without corresponding test files:

# Simplified discovery script
# Finds models without Minitest coverage, sorted by size (largest first)
model_files = Dir.glob("app/models/**/*.rb")

untested = model_files.select do |model_path|
  test_path = model_path
    .sub("app/models/", "test/models/")
    .sub(".rb", "_test.rb")
  !File.exist?(test_path)
end

sorted = untested.sort_by { |f| -File.readlines(f).size }

sorted.each { |f| puts f }

Largest first is deliberate. The biggest models are the hardest, and I wanted to hit the hardest problems early when I had the most energy for debugging. A batch of small models at the end is a reward, not a risk.

Shell Loop for Sequential Claude Sessions

The shell script structure is straightforward:

#!/bin/bash
# Simplified Ralph main loop structure

BATCH_SIZE=10
COMMIT_INTERVAL=3
counter=0

discover_models | head -n "$BATCH_SIZE" | while read -r model_path; do
  echo "Migrating: $model_path"

  # Run the 6-gate pipeline
  claude -p "/minitest-migration $model_path" \
    --dangerously-skip-permissions

  # Check if tests pass
  test_file=$(echo "$model_path" | sed 's|app/models/|test/models/|; s|.rb|_test.rb|')
  if ! bin/rails test "$test_file" 2>/dev/null; then
    fix_tests "$test_file"
  fi

  counter=$((counter + 1))
  if [ $((counter % COMMIT_INTERVAL)) -eq 0 ]; then
    git add -A && git commit -m "batch: migrate $counter models"
  fi
done

The /minitest-migration command is where the actual intelligence lives. It runs the 6-gate pipeline with I/O contracts and parallel writers. Ralph just calls it and handles the result.

Automated Test Fix Retries

flowchart TD
    RUN["bin/rails test"] --> CHK{"Exit code 0?"}
    CHK -->|"yes"| DONE["Continue to next model"]
    CHK -->|"no"| EXT["Extract failure lines"]
    EXT --> FIX["claude -p /fix-tests"]
    FIX --> RUN
    FIX -->|"max 3 retries"| WARN["Log warning, move on"]

    style RUN fill:#2a5f8f,stroke:#4a9eed,color:#ffffff
    style CHK fill:#7a5c2e,stroke:#e6b44a,color:#ffffff
    style DONE fill:#1e6e45,stroke:#4ae68a,color:#ffffff
    style EXT fill:#8b3a3a,stroke:#ed4a4a,color:#ffffff
    style FIX fill:#8b3a3a,stroke:#ed4a4a,color:#ffffff
    style WARN fill:#5a2e7a,stroke:#b44ae6,color:#ffffff

About 40-50% of models did not pass on the first /minitest-migration run. This is where the second layer kicks in. The fallback function extracts the specific failure from test output and feeds it back to Claude with focused context:

fix_tests() {
  local test_file="$1"
  local max_retries=3
  local attempt=0

  while [ $attempt -lt $max_retries ]; do
    # Capture test output, extract failure block
    test_output=$(bin/rails test "$test_file" 2>&1)
    exit_code=$?

    [ $exit_code -eq 0 ] && return 0

    # Extract only the failure lines for focused context
    failures=$(echo "$test_output" | grep -A 5 "Failure:\|Error:")

    claude -p "/fix-tests $test_file" \
      --dangerously-skip-permissions \
      <<< "$failures"

    attempt=$((attempt + 1))
  done

  echo "WARN: $test_file still failing after $max_retries retries"
}

Send the extracted failure block, not the full test output. A 2,000-line model test can produce pages of output. The agent needs the five lines around the failure, not the 200 lines of passing tests.

Migration Results: 98 Models in 283 Sessions

Ralph ran across 6 batches over 3 calendar weeks:

The Batch 1 line count (+47,732) is an outlier because it includes the agent infrastructure itself: agent definitions, pattern files, Claude commands, and documentation alongside the generated tests. Later batches are pure test output.

Session counts reveal complexity better than any other metric. Batch 5 averaged 1.6 sessions per model (smaller models that mostly passed on the first run). Batch 3 averaged 5 sessions per model (the largest, including models at 2,195 and 1,282 lines, each needing multiple fix iterations and manual debug sessions). By Batch 6, the pipeline was mature: 50 models in 29 sessions, 0.6 sessions per model. Established fixture patterns and refined agent instructions meant small models often passed on the first invocation.

The first-pass success rate was roughly 50-60%. Models under 200-300 lines of source usually passed clean. Above that, expect at least one fix iteration. Per-model wall-clock time: small models (~15 minutes), medium models (~25 minutes with one fix cycle), large models (~45+ minutes with multiple iterations).

The Payment model shows what the pipeline produced at its best: the original RSpec file contained 1 test. The migration expanded it to 124 Minitest tests with 98.68% line coverage (150 of 152 lines).

Ralph required 15 manual interventions across 98 models, an ~85% autonomous rate at the model level. For roughly every 7 models processed, 1 needed human attention: sometimes just restarting after a crash, sometimes debugging a fixture cascade, sometimes making a strategic decision to accept partial coverage and move on. The five most representative failure categories are covered in the Human Supervision section below.

Scaling to Nearly 10,000 Tests

flowchart LR
    W1["Week 1: 1,907 tests"] -->|"overnight runs"| W2["Week 2: 4,830 tests"]
    W2 --> W3["Week 3: 8,201 tests"]
    W3 --> W4["Week 4: ~10,000 tests"]

    style W1 fill:#2a5f8f,stroke:#4a9eed,color:#ffffff
    style W2 fill:#7a5c2e,stroke:#e6b44a,color:#ffffff
    style W3 fill:#1e6e45,stroke:#3cb371,color:#ffffff
    style W4 fill:#5a2e7a,stroke:#9b59b6,color:#ffffff

The pipeline did not produce nearly 10,000 tests on the first attempt. It scaled in visible increments, each one surfacing new failure categories that fed back into the agent instructions.

After the first week, the test suite was already running faster than the RSpec suite it was replacing:

bin/rails test test/models/
15.23s · 4,830 tests (317.15/s) with 9,453 assertions (620.71/s)
Line Coverage: 14.19% (14,582 / 102,732)

By the end of week two, the test count had grown to 8,201 across 32 seconds of wall-clock time:

bin/rails test test/models/
32.62s · 8,201 tests (251.4/s) with 15,907 assertions (487.62/s)
Line Coverage: 17.58% (17,645 / 100,378)

The coverage percentages look small, but those are whole-app numbers (100,000+ lines). The per-model coverage on migrated models was already hitting 80-95%. As the pipeline processed the remaining models over the following week, the test count climbed to close to 10,000 across 98 models, with line coverage approaching 20%.

141,000 lines of generated test code deserve scrutiny. That is a lot of code to maintain, regardless of who wrote it, and “generated code” carries legitimate risk: if the generator produces shallow tests (tests that exercise code paths without verifying meaningful behavior), you end up maintaining code that provides coverage numbers without confidence. Three factors make the maintenance burden manageable in practice, though the long-term cost is still unproven. First, model tests are structurally repetitive (association tests, validation tests, scope tests), so reading and modifying them is fast even at scale. If a factory changes or a validation moves, the fix is mechanical. Second, the mathematical derivation from the Analyst/Writer split means every test traces to a specific technique and source method, which makes it possible to regenerate sections rather than hand-editing. Third, Minitest's flat structure (no nested context blocks, no let chains) means each test method is self-contained and readable in isolation. One comparison from later in the project: TransferProcessingService had 19.39% line coverage under RSpec (71 examples, 82 seconds). The Minitest migration reached 87.0% coverage (228 tests, 13.67 seconds). More coverage, more tests, 6x faster. The real test of this claim is not now, when the code is fresh, but six months from now, when the models have evolved and the generated tests need updating. I do not yet have that data.

Fixture Cleanup: Why a Second Orchestrator Was Needed

flowchart LR
    subgraph Phase1["Phase 1"]
        P1["Selective Loading, reverted"]
    end

    subgraph Phase2["Phase 2"]
        P2["Fix Enums, 65 files"]
    end

    subgraph Phase3["Phase 3"]
        P3["Consolidation, 96 files"]
    end

    subgraph Phase4["Phase 4"]
        P4["YAML Anchors, low impact"]
    end

    Phase1 --> Phase2
    Phase2 --> Phase3
    Phase3 --> Phase4

    style P1 fill:#2a5f8f,stroke:#4a9eed,color:#ffffff
    style P2 fill:#8b3a3a,stroke:#e06060,color:#ffffff
    style P3 fill:#7a5c2e,stroke:#d4a04a,color:#ffffff
    style P4 fill:#5a2e7a,stroke:#9b59b6,color:#ffffff
    style Phase1 fill:#1e3a5f,stroke:#4a9eed,color:#e0e0e0
    style Phase2 fill:#3d1a1a,stroke:#e06060,color:#e0e0e0
    style Phase3 fill:#3d2e1a,stroke:#d4a04a,color:#e0e0e0
    style Phase4 fill:#2e1a3d,stroke:#9b59b6,color:#e0e0e0

Ralph creates fixture debt as a side effect, the same category of structural overhead that TestProf surfaced in the RSpec suite. Every model migration generates JIT fixtures, adds references, and modifies YAML files. After 98 models, the fixture layer accumulated systematic problems that individual migrations could not see. This is the third layer in the error-handling stack: Felix exists to catch what Ralph structurally could not, because the problems only become visible across files, not within them.

Felix is structured as four separate discovery-and-fix loops, each targeting a specific class of fixture problem:

Phase 1: Selective Loading (96 files, 144 sessions, fully reverted). The goal was to replace the global fixtures :all default with explicit per-file fixtures : declarations, so each test class only loads the fixtures it needs. The orchestrator ran overnight, traced fixture dependencies for all 96 files, and added declarations. The next morning, I discovered that the model dependencies in this codebase were too deeply interconnected for selective loading to work. Foreign key chains meant that loading one fixture table required loading most of the others anyway. I reverted to fixtures :all. Those 144 sessions, roughly 21% of Felix's total, produced zero lasting value. A manual spike on 2-3 files would have revealed the interconnection problem in an hour, before committing to a full batch run.

Phase 2: Fix Enums (65 files, 227 sessions). Fixture files contained hardcoded integer enum values (state: 1) instead of symbolic references (state: <%= Order.states[:active] %>). When enum definitions change, hardcoded integers silently produce wrong test data. This phase was the most session-heavy because each file required the agent to read the model source, trace enum definitions through concerns, build the integer-to-symbol mapping, and replace values. That is why a “simple” transformation averaged 3.5 sessions per file.

Phase 3: Consolidation (96 files, 88 sessions). Fragmented fixture references accumulated across migrations. One test file referenced 43 separate fixtures. The consolidation pass merged related fixture references and removed unused ones.

Phase 4: YAML Anchors (5 files, 22 sessions). I expected this to be a big win: the largest fixture files had massive duplication across 50-141 entries. In practice, only 5 files qualified, and the impact was minimal. Most fixture files needed entirely new fixture designs rather than deduplication of existing ones.

Each phase had its Ruby discovery script. Phase 2's discovery, for example, scanned fixture YAML for integer values in known enum columns:

# Simplified Phase 2 discovery: find fixtures with hardcoded enum integers
fixture_files = Dir.glob("test/fixtures/**/*.yml")

fixture_files.each do |path|
  content = File.read(path)
  # Match lines like "  state: 1" or "  status: 0"
  if content.match?(/^  \w+:\s+\d+\s*$/)
    puts path
  end
end

(The initial regex was ^\s+\w+, which matched nested YAML hashes and produced 80 false positives across 7 files. Tightening it to an exact 2-space indent eliminated the false matches. I caught this mid-run and had to restart, one of the interventions discussed below.)

Fixture Cleanup Results: 481 Sessions Across 4 Phases

Phases 1 and 3 operate on the same 96 test files. Phases 2 and 4 operate on fixture files (Phase 4's 5 files are a subset of Phase 2's 65). The 262 figure counts file-phase operations; the unique file count is 96 test files + 65 fixture files = ~161.

Felix generated 1.7x more sessions than Ralph (481 vs 283) despite targeting narrower per-file transformations. Two factors explain the volume: ~161 files vs 98 models, and Phase 2's enum replacement being harder than it looked (3.5 sessions per file, because tracing enum definitions through model inheritance and concerns required more context than a “simple find-and-replace” would suggest). Volume dominates. Simple work multiplied by hundreds of file-phase operations produces more total agent sessions than complex work on fewer files.

Felix required 6 manual interventions across all 4 phases, mostly concentrated in early Phase 2 debugging. Different failure modes than Ralph: regex false positives in discovery scripts, parallel execution blockers from partial fixture state, and the Phase 1 revert decision. The third layer catches different problems than the first two.

Combined Results: 764 Sessions Across 259 Files

flowchart TD
    subgraph Ralph["Ralph: Migration"]
        direction LR
        R1["98 models"] --> R2["~283 sessions"]
        R2 --> R3["15 interventions"]
    end

    subgraph Felix["Felix: Cleanup"]
        direction LR
        F1["~161 unique files"] --> F2["~481 sessions"]
        F2 --> F3["6 interventions"]
    end

    subgraph Combined["Combined: 16 Days"]
        direction LR
        T1["~259 unique files"] --> T2["764 sessions"]
        T2 --> T3["21 interventions"]
    end

    Ralph --> Combined
    Felix --> Combined

    style R1 fill:#2a5f8f,stroke:#4a9eed,color:#ffffff
    style R2 fill:#2a5f8f,stroke:#4a9eed,color:#ffffff
    style R3 fill:#2a5f8f,stroke:#4a9eed,color:#ffffff
    style F1 fill:#7a5c2e,stroke:#d4a04a,color:#ffffff
    style F2 fill:#7a5c2e,stroke:#d4a04a,color:#ffffff
    style F3 fill:#7a5c2e,stroke:#d4a04a,color:#ffffff
    style T1 fill:#1e6e45,stroke:#3cb371,color:#ffffff
    style T2 fill:#1e6e45,stroke:#3cb371,color:#ffffff
    style T3 fill:#8b3a3a,stroke:#e06060,color:#ffffff
    style Ralph fill:#1e3a5f,stroke:#4a9eed,color:#e0e0e0
    style Felix fill:#3d2e1a,stroke:#d4a04a,color:#e0e0e0
    style Combined fill:#1a3d2e,stroke:#3cb371,color:#e0e0e0

The timelines overlapped. Felix's Phase 1 started while Ralph's later batches were still running. The 16 working days is the calendar span from first Ralph batch to last Felix phase, not sequential execution time.

The model-level rate (~85%) is the most meaningful because that is where each go/no-go decision happened: migrate a model, check the result, decide whether to intervene. At the file level (~92%) or the session level (~97%), the numbers look better because the denominator grows, but the model is the natural unit of work.

Peak session days reveal the burst nature of batch orchestration: one day produced 112 Ralph sessions (the largest batch starting), another produced 257 (Felix Phases 2 and 3 running in parallel). Most days were under 20 sessions.

On cost: I ran this on a Claude team plan with a fixed monthly rate, so the 764 sessions consumed token allocation rather than producing a per-session bill. I did not track per-session token usage, which means I cannot give a precise cost figure or compare it meaningfully to alternative approaches. If you are considering a similar project on a usage-based plan, track tokens from session one.

The 98 models covered here are about 19% of the 507 models in the codebase. Building Ralph and Felix was not an investment for one batch. It was infrastructure for every batch that follows. The economics depend on that reuse assumption. If the remaining 409 models were the only future use, the investment would likely pay for itself. If the orchestration pattern never runs again, the 16 working days of supervised automation compares unfavorably to a developer manually migrating the same 98 models over 4-6 weeks (estimating 2-3 models per day for a developer familiar with the codebase and testing patterns).

When AI Agents Need Human Intervention

flowchart TD
    subgraph Layer1["Layer 1: Generation"]
        direction LR
        L1A["Pipeline runs"] --> L1B["40-50% fail first pass"]
    end

    subgraph Layer2["Layer 2: Auto Recovery"]
        direction LR
        L2A["/fix-tests, max 3 retries"] --> L2B["Catches most failures"]
    end

    subgraph Layer3["Layer 3: Felix"]
        direction LR
        L3A["Felix: systematic cleanup"] --> L3B["Different failure modes"]
    end

    subgraph Layer4["Layer 4: Human"]
        direction LR
        L4A["21 interventions"] --> L4B["Crashes, strategy, cascades"]
    end

    Layer1 -->|"failures escape"| Layer2
    Layer2 -->|"structural debt escapes"| Layer3
    Layer3 -->|"irreducible problems"| Layer4

    style L1A fill:#1e6e45,stroke:#4ae68a,color:#ffffff
    style L1B fill:#1e6e45,stroke:#4ae68a,color:#ffffff
    style L2A fill:#2a5f8f,stroke:#4a9eed,color:#ffffff
    style L2B fill:#2a5f8f,stroke:#4a9eed,color:#ffffff
    style L3A fill:#7a5c2e,stroke:#e6b44a,color:#ffffff
    style L3B fill:#7a5c2e,stroke:#e6b44a,color:#ffffff
    style L4A fill:#8b3a3a,stroke:#e06060,color:#ffffff
    style L4B fill:#8b3a3a,stroke:#e06060,color:#ffffff
    style Layer1 fill:#1a3d2e,stroke:#4ae68a,color:#e0e0e0
    style Layer2 fill:#1e3a5f,stroke:#4a9eed,color:#e0e0e0
    style Layer3 fill:#3d2e1a,stroke:#d4a04a,color:#e0e0e0
    style Layer4 fill:#3d1a1a,stroke:#e06060,color:#e0e0e0

The orchestrators ran on a local machine with --dangerously-skip-permissions (no interactive prompts). The scope was test files and fixtures, with Webmock blocking external HTTP calls, so the blast radius was limited to the repository. A clean git branch with frequent commits meant any damage was revertible.

Across both orchestrators, 21 problems reached the human layer. They cluster into five categories that reveal what no amount of automated layering can handle.

Orchestrator Crashes

The first real Ralph run crashed because the shell script tried to parse Claude's markdown output as a bash conditional. The Claude CLI returned the full gate report text (starting with a markdown heading) where the script expected an exit code. Bash interpreted the markdown as a [[ test expression and errored out. I had to stop the run, fix the script to extract the exit code separately from the report content, and restart.

A separate run hit fatal: failed to write commit object, an SSH signing failure that killed the commit step. Manual re-authentication and restart.

False Success Detection

The fix loop for the Deal model reported “96 passing, 0 failing” but then started another iteration and errored out. The orchestrator could not distinguish “Claude says tests pass in its summary” from “the bin/rails test exit code was 0.” The script parsed Claude's natural language summary as loop control instead of checking the actual test runner exit code.

After fixing this, the orchestrator used only exit codes for control flow and treated Claude's text output as logging. Trust machine-readable signals (exit codes, JSON output, file existence checks), not natural language summaries. The agent can hallucinate “all tests pass” just as easily as it can hallucinate a code fix.

Cross-File Fixture Cascades

Migrating deal_test.rb broke invoice_test.rb, sellable_test.rb, and shop_theme_test.rb. The /fix-tests agent could handle one or two cascading failures. Complex cases, where a fixture modification rippled across four test files, required looking at git diff, identifying which fixture change caused the regression, and either reverting manually or guiding the next Claude invocation with more specific context about the dependency chain. The pipeline article covers why fixture cascades were so destructive.

Fixture cascades are a class of problem that no single-file agent can solve. The agent sees one test file and its fixtures. The cascade is a cross-file dependency that only a human (or a future agent with broader context) can trace.

Partial Coverage on Complex Models

The Seller model (1,015 lines, tight coupling to two external CRM services) hit 34.86% coverage after three fix iterations. For comparison, the pipeline's typical output was 75-90% method coverage on well-structured models. 34.86% meant the agent could not reach methods that depended on live CRM connections, mocked state that did not exist in fixtures, or internal transaction callbacks. Another model had LockManager deadlocks that caused non-deterministic test failures.

These required human judgment: accept the partial coverage and move on, or burn 10 more fix iterations for diminishing returns. I chose to move on both times. Some problems are not worth solving at batch scale. Flag them, skip them, and handle them individually later.

Mid-Run Discovery Script Fixes

Felix Phase 2's regex false-positive issue (described above) required stopping the orchestrator, fixing the discovery script, and restarting. This eliminated 80 false positives across 7 files. The 2,104 legitimate enum violations were fixed correctly.

This category (broken tooling around the agent, not a broken agent) accounted for roughly a third of Felix's interventions. Better discovery scripts would have prevented several restarts.

Improving Batch AI Agent Orchestration

Four changes if I were doing this again:

1. Structured intervention logging from Batch 1. Ralph's 15 interventions is an approximate count because I did not track precisely until Batch 2.
2. Track token usage per session for cost modeling. Without per-session data, I cannot compare this approach to alternatives on a usage-based plan.
3. Build Felix cleanup phases into the Ralph pipeline rather than running them as a separate orchestrator. Fixture cleanup was predictable from how fixture data dominated migration failures. Running it as a second pass doubled the total session count.
4. Validate batch-wide assumptions with a manual spike before committing to a full run. Felix Phase 1's 144 reverted sessions could have been avoided by testing selective fixture loading on 2-3 files first.

Two metrics worth tracking are sessions per unit (agent invocations divided by work items completed) and interventions per layer. Ralph averaged 2.9 sessions per model overall, ranging from 0.6 (Batch 6, mature pipeline) to 5.0 (Batch 3, largest models). A falling ratio across batches means your pipeline is improving. A spiking ratio means you hit a class of problem the pipeline cannot handle. If most interventions cluster in one layer, that layer needs a new sub-layer.

The 764 sessions covered here were the model layer. Service and integration test sessions brought the indexed total to 871 by April, requiring different agent designs and a different orchestration pattern.

764 sessions. 21 human interventions, roughly 1 in 7 models. Build for layers, not for perfection.

Available as a Service

I help engineering teams design and run AI agent pipelines like the batch orchestrators described here, from single-file automation to codebase-wide migrations. If your team is exploring where AI agents fit into large-scale engineering tasks, book a free 30-minute call or check my services page for details.