Eliminate Ghost State Changes, or Be Haunted Forever

A ghost state change is a mutation the system cannot explain.

Truly terrifying. For any state transition that matters, the system should be able to answer three questions:

1. What changed?
2. Why was the change valid?
3. What new facts followed from that change?

As Leslie Lamport put it, a bug is an invalid state. If the system can detect that it is in an invalid state but cannot reason about how it reached that state, it stops being debuggable. At least a known bug points developers and operators in the right direction, but an untracked mutation leaves you reconstructing the blast radius by reading entrails, consulting a ouija board, and huffing gas in a cave to ask the oracle.

Consulting with spirits to debug your systems comes at a high cost, and that cost surfaces in four distinct failure modes.

The problem persisted because the incentives allowed it. Businesses do not usually reject durability on principle. They discount it while the consequences of ambiguity remain low. During the long era of cheap capital, speed was rewarded more than durability.¹ Systems were built to deploy quickly, not to preserve a trustworthy explanation of how their state changed.

But that trade-off only works while failure is cheap. The first serious incident, regulatory inquiry, or public failure makes the ignored constraints reassert themselves all at once.

A Haunting Incident

Ghost state changes are not theoretical, and they are not benign. In the UK Post Office Horizon scandal, people were wrongly accused and wrongfully convicted of fraud because the system showed financial shortfalls they could not explain, challenge, or disprove. What made the scandal so devastating was not just that the numbers were wrong, but that the system’s record carried more weight than the testimony of the people accused.

In the December 16, 2019 Horizon Issues judgment, Mr Justice Fraser found a “significant and material risk of inaccuracy” in branch accounts caused by bugs, errors, and defects in Horizon.² He also found that Fujitsu could remotely insert or edit branch transactions without the sub-postmaster’s knowledge and without the change being identifiable as remote.³

The Criminal Cases Review Commission later described the prosecutions arising from Horizon as the biggest single series of wrongful convictions in British legal history.⁴ When a system can change financial state through paths its operators cannot see, trust, or challenge, the result is catastrophic on a human level and erodes societal trust in technology. Non-technical people may look at current state as authoritative when it is anything but.

Most Persistence Implementations Are Vulnerable

Horizon was an extreme case, but the underlying weakness is ordinary. Most production systems persist current state far more reliably than they preserve the transitions that produced it.

Relational databases are not the problem. Many strong systems use them. The problem is the dominant CRUD pattern that treats the latest snapshot as the book of record while relegating the transitions that produced it to logs, WALs, audit tables, or other secondary telemetry.

In Stack Overflow’s 2024 Developer Survey, the four most-used databases were PostgreSQL (48.7%), MySQL (40.3%), SQLite (33.1%), and Microsoft SQL Server (25.3%), all relational systems.⁵ The DB-Engines ranking for March 2026 likewise places Oracle, MySQL, SQL Server, and PostgreSQL in the top four.⁶

In those systems, you can query what is true now, but unless the system is explicitly architected to preserve and project transitions, you cannot authoritatively reconstruct how it became true. Relational databases do not provide that for free, and certainly not on the hot path. That is the opening ghost state changes exploit.

Where Ghosts Come From

Ghost state changes arise when multiple components can mutate the same state through different paths, and none of those paths produce a durable record of the transition. The system can show you what is true now but not how it became true.

Consider a mutable row in a relational database. The application updates it. A scheduled job updates it. An operator runs a one-off UPDATE during an incident. Each mutation overwrites the previous value. The system retains the current state but not the sequence of transitions that produced it.

-- Each UPDATE destroys the previous value
UPDATE accounts SET balance = 1000 WHERE user_id = 42;
UPDATE accounts SET balance = 3000 WHERE user_id = 42;
UPDATE accounts SET balance = 2000 WHERE user_id = 42;

After these three statements, the balance is 2000. The system cannot determine that it was ever 1000 or 3000, cannot attribute any of the changes to an actor, and cannot explain why any of the transitions were valid.

Secondary artifacts such as WALs, trigger-based audit tables, and CDC streams do not solve this problem. They record that a change occurred, but they are operational side channels. They are not the source of truth. When the WAL rotates out or the audit table is truncated for space, the record disappears. The system’s explanation of its own history was never authoritative to begin with.

This is why the remedy has to be structural. You cannot bolt trustworthy history onto a system that treats important mutations as out-of-band side effects. The only reliable fix is to make the mutation path itself authoritative.

Making State Explicit

Eliminating ghost state changes requires treating state transitions as first-class records.

The CHAIN maturity model⁷ describes the difference between systems that can explain their state changes and systems that cannot. At low maturity, changes happen, but the system cannot reliably say what caused them, who made them, or how the current state was reached. At higher maturity, important transitions are preserved, linked to their causes, and recorded with enough context to explain themselves.

In practice, three structural properties make that higher maturity possible and help resolve ghost state changes:

1. Append-only event journals record every transition as an immutable fact. This addresses History. Past states survive rather than being overwritten.
2. Causal linking connects each event to the event or command that triggered it. This addresses Causality. The system can traverse the full chain from any state change back to its origin.
3. Wide events capture actor, intent, and context alongside the transition itself. This addresses Agency and Intent. Each record is self-describing.

We’ll cover each of the properties below, along with examples.

Append-Only Event Journals

The first property is durable history. State transitions are recorded as immutable, ordered events rather than as destructive updates to mutable rows.

CREATE TABLE account_events (
    event_ulid      text PRIMARY KEY,   -- monotonic ULID (lexicographic order)
    account_id      int,
    sequence        bigint,
    event_time      timestamp,
    event_type      text,
    amount          decimal
);

INSERT INTO account_events (event_ulid, account_id, sequence, event_time, event_type, amount)
VALUES (
    gen_monotonic_ulid(),
    42,
    1,
    now(),
    'FundsDeposited',
    1000
);

Current state becomes a derived projection. The journal is the source of truth.

-- Derived read model, not the authoritative record
CREATE TABLE balance_by_account (
    account_id  int PRIMARY KEY,
    balance     decimal
);

This preserves the fact that a transition occurred and prevents destructive overwrites from erasing prior state. But the journal alone still does not explain what caused each transition.

Causal Linking

The second property is that each event carries an explicit reference to what triggered it.

CREATE TABLE trade_events (
    event_ulid         text PRIMARY KEY,
    trade_id           int,
    event_time         timestamp,
    event_type         text,
    causal_parent_ulid text,            -- the event or command that triggered this one
    correlation_ulid   text,            -- groups related events into a workflow
    payload            jsonb
);

The causal_parent_ulid field points to the upstream event or command that caused the transition. The correlation_ulid field groups related events into the same workflow. Together, these two fields turn a flat event log into a traversable causal graph.

A TradeSettled event points back to the TradeExecuted event that caused it, which points back to the OrderPlaced command that initiated the workflow. Given any recorded event in the workflow, you can walk the causal_parent_ulid chain back to the command that started the sequence.

If a trade settles at the wrong price, the causal chain helps reconstruct what happened in a single query.

-- Walk the causal chain backward from a specific event
WITH RECURSIVE causal_chain AS (
    SELECT event_ulid, event_type, causal_parent_ulid, payload
    FROM trade_events
    WHERE event_ulid = '01HY3...'  -- the suspicious event

    UNION ALL

    SELECT t.event_ulid, t.event_type, t.causal_parent_ulid, t.payload
    FROM trade_events t
    JOIN causal_chain c ON t.event_ulid = c.causal_parent_ulid
)
SELECT * FROM causal_chain;

The result is the recorded sequence of transitions that produced the current state, ordered by cause. Every link in the chain is an immutable record. Every transition points to the transition that triggered it. Within the recorded chain, there are no gaps to reconstruct and no secondary logs to correlate.

Wide Events

The third property is recording enough context for each transition to be self-describing.⁸

{
  "event_id":       "01HY3...",
  "event_type":     "AccountClosed",
  "ts":             "2025-05-15T12:07:13.044Z",
  "actor_id":       "operator:jsmith",
  "intent":         "customer_requested_closure",
  "correlation_id": "ticket-OPS-4421",
  "source":         "admin_cli",
  "payload":        {
    "account_id": "acct-9931",
    "reason_code": "GDPR_REQUEST"
  }
}

A wide event answers all three questions in a single record. It identifies what changed (AccountClosed), why the change occurred (customer_requested_closure under a GDPR request), and the surrounding context: which actor initiated it, from which interface, and under which correlation ID. Wide events drastically reduce after-the-fact reconstruction because the explanation travels with the transition itself.

A Trade Through Both Designs

Suppose Alice places an order to buy 100 shares of GME.

In a snapshot-first system, the workflow is scattered across mutable state. By the end, the system can tell you what appears to be true now, but not authoritatively how it became true.

With explicit transitions, Alice’s trade workflow begins with a PlaceOrder command. The order aggregate evaluates it against current state and either rejects it or emits an OrderPlaced event. Fund reservation, partial fills, and cancellations are recorded as causally linked transitions rather than ad hoc mutations. The command records what was attempted. The events record what occurred. The causal chain preserves the sequence.

Replay also has maturity levels. At an earlier level, a system supports narrative replay. Enough history and causality are preserved to reconstruct what happened and explain it in a postmortem or root cause analysis. That is already a meaningful improvement over opaque state, because the system can do more than report the final value. It can tell a defensible story about how that value was reached.

At a higher level, replay becomes operational rather than merely explanatory. The system can feed recorded transitions back through its own logic and authoritatively rebuild derived state, showing exactly how one value became another. That is the stronger test. If the journal can reproduce state from the recorded transitions, the record is doing real architectural work. If it cannot, some part of the truth still lives outside the system.

The Litmus Test: Replay

The definitive test for ghost state changes is whether current state can be projected from recorded state changes. If the system derives its read models from the event journal, replay is a direct consequence of how state is built. Discard the derived read models and rebuild them from the journal.

TRUNCATE account_balance_projection;

INSERT INTO account_balance_projection (account_id, balance, last_sequence)
SELECT
    e.account_id,
    SUM(e.amount)       AS balance,
    MAX(e.sequence)     AS last_sequence
FROM account_events e
GROUP BY e.account_id;

If the rebuilt projections match production state, every transition that affected the system was recorded in the journal. If they diverge, something mutated state through a path the journal does not cover. The divergence is direct evidence of ghost state changes.

Deterministic replay requires a stronger property than mere event logging. Given the same events in the same order, the system must produce the same state. This holds only when all state-modifying paths flow through the journal. Any bypass breaks the invariant. Direct database writes, operator mutations, and background jobs that update rows without emitting events all introduce state changes the journal cannot account for.

Closing

Ghost state changes are mutations without evidence. They are transitions the system cannot attribute, explain, or reproduce. They arise whenever state can be modified through paths that do not produce durable, causally linked records.

The corrective is structural. Record every transition as an immutable event. Link each event to its cause. Capture enough context for the record to be self-describing. Validate the result through replay.

These ideas are formalized in the CHAIN maturity model, which defines progressive levels of causality, history, and intent tracking for reflective systems.