COMPACTION

The Experiment

We scripted a 20-message Effect-TS coding session. The agent builds Clock and Logger services, writes tests, and makes architectural decisions. Along the way, the user establishes conventions, corrects mistakes, and discusses trade-offs.

Then we simulate compaction by controlling what context the model sees at probe time. Five conditions:

Five probes, one per knowledge category. Each asks the model to demonstrate retention of a specific type of information. Three runs per condition at temperature 0.2 for variance measurement. GPT-4.1 for probes, GPT-4.1-mini as automated judge.

Scoring: 0 = lost, 1 = partial, 2 = retained. Each probe has a specific rubric. Zero ambiguous scores across all 75 runs.

The Results

Color scale: retained (1.8–2.0)   partial (1.2–1.7)   degraded (0.5–1.1)   lost (<0.5)

The columns differed more than the overall row averages. In this task, goals and files were easier to recover than conventions or prior reasoning.

Goals and Files: Retained in These Runs

Goal recall: 2.0/2.0 across every condition. File awareness: 2.0/2.0 across every condition. No variance. No exceptions.

Even the 200-word summary retained "we're building Clock and Logger services in Effect-TS." The CLAUDE.md condition also received full credit for goal recall. The goal was repeated and prominent in the synthetic session, which likely made it easier to recover.

This does not show that goals survive compaction generally. It shows that this repeated goal survived the five representations tested here.

Reasoning Trajectory: Partial in Every Condition

1.0/2.0 across every single condition. Including full context.

The full-context condition also scored partial on the reasoning probe. The response recovered the preferred entry-point composition but not the full sequence: "we tried inline provision, it got messy, we switched."

The result is consistent with reports that a session can retain decisions while losing the path to them. More tasks, models, and human grading would be needed before treating that as a general property.

Conventions: The Variable Middle

Convention adherence is where the conditions actually diverge. The probe asks the model to write new code following three mid-session conventions: Effect.fn, @scope/Name tags, Effect.logInfo.

In these three runs, the 500-word model summary scored 2.00 while full context scored 1.33 on convention adherence. The summarizer distilled the conventions into explicit, prominent statements. In the full conversation, conventions are buried in back-and-forth..corrections mid-message, asides, code blocks.

This result is consistent with the "Lost in the Middle" work: the conventions were buried in a 20-message conversation and explicit in the summary. In these runs, moving them to a prominent position improved the convention score.

Corrections: Mostly Safe

Correction retention averaged 1.87/2.0 across conditions. The probe asks the model to create a new Layer..does it use Layer.effect (corrected) or Layer.sync (original mistake)?

Most conditions scored 2.0. The exception: CLAUDE.md dropped to 1.33. One of three runs reverted to the uncorrected pattern. The rules file said "use Layer.effect" but without the conversational context of why that correction was made, the directive was weaker.

Corrections are more resilient than conventions because they're binary. The model either uses the corrected form or doesn't. Conventions require combining multiple behaviors, which is harder to maintain.

The Retention Hierarchy

The average scores in this task formed this order:

One possible explanation is salience. Goals and files were repeated and prominent. Corrections asked for a binary choice. Conventions combined several behaviors, while the reasoning probe required reconstructing a sequence across messages.

The harder something is to compress into a single statement, the less likely it survives compaction.

What to Externalize

These scores changed what I put in CLAUDE.md, though the recommendation still needs testing on real repositories.

Compaction Insurance

If you run agents in loops..autonomous sessions that restart periodically..every restart is a compaction event. The context resets. The loop starts fresh.

CLAUDE.md files, AGENTS.md files, and health checks carry conventions and corrections across session boundaries because each new session reads them again.

Our data shows why this works: the CLAUDE.md condition scored 1.47 overall, losing to model summaries (1.80) in a single-session test. But that's the wrong comparison. A CLAUDE.md persists across unlimited sessions. A model summary exists for one compaction event and is gone.

CLAUDE.md scored below the model summaries in this single-session test, but it tests a different property: the file remains available when the session and its summary are gone.

Limitations

This is one experiment with one model (GPT-4.1), one task scenario (Effect-TS services), and 75 data points. The retention hierarchy is consistent across conditions, which builds confidence, but:

• We tested simulated compaction (controlled context), not real compaction (model-internal). Real compaction implementations vary by provider and are opaque.
• Three runs per cell is enough to spot patterns, not enough for statistical significance. The zero-variance cells (goal, files, reasoning) are more trustworthy than the variable ones (conventions).
• The automated judge used GPT-4.1-mini with specific rubrics. Zero ambiguous scores suggests clean rubrics, but could also indicate rubrics that are too easy to satisfy.
• Effect-TS has distinctive syntax. Convention adherence results may differ for less syntactically unique frameworks.

The experiment ran for under $5 in API costs. It could be replicated across models and scenarios for stronger claims. The harness is open source.

The Numbers

Primary hypothesis confirmed: compaction preserves goals but destroys conventions and reasoning. Not "sometimes" or "partially"..with remarkable consistency across all conditions.

Secondary hypothesis confirmed: retention is categorical. Goals (2.0) and reasoning (1.0) don't overlap. The hierarchy is stable.

Null hypothesis rejected: compaction is not lossless. But the loss is predictable, and that predictability is useful.

I externalize what degrades. Re-state what nothing preserves. No point wasting tokens on what survives everything.