On Commit Logs

Before computers had Write-Ahead Logs, accountants had ledgers. For centuries, the most reliable way to track financial transactions was in a book where you only ever added new entries to the end. You never erased or modified a past entry; you corrected it by appending a new, counteracting entry. This simple, append-only, ordered sequence is one of the most powerful and resilient concepts in computing, and it’s called a commit log.

Last week, I was reading up on the architecture of Apache Kafka, and in one of the docs, it’s defined as a distributed commit log. Commit logs are also the core principle behind a lot of tech, from relational database's durability to Kafka's massive throughput.

At its core, a commit log is a simple data structure. It's a durable, ordered, append-only sequence of records.

• Append-only: New data is always added to the end. You don't go back and modify old records. This makes writes extremely fast because they are sequential disk operations, not random seeks.

• Ordered: Records exist in the exact sequence they were written. This chronological ordering is a feature, not a bug.

• Durable: Once a record is written to the log and acknowledged, it's permanent and survives system crashes.

Think of it as the opposite of a typical database table where you can perform UPDATE and DELETE operations in place. With a log, the history of changes is the data.

This is the classic use case and the reason databases don't lose data when the power goes out. The 'D' in ACID (Durability) is almost universally implemented with a Write-Ahead Log.

The process is straightforward:

1. A transaction begins (e.g., UPDATE users SET status = 'active' WHERE id = 123).

2. Before the database modifies the actual users table on disk, it writes a record of the intended change to the log file.

3. Once the record is safely on disk in the log, the transaction can be considered committed.

4. Only then does the database get around to applying the change to the table files themselves, often in a lazy, batched fashion for efficiency.

If the server crashes mid-operation, it doesn't matter if the table file was updated or not. On restart, the database recovery process reads the WAL, sees the committed transaction record, and replays it to ensure the table reflects the correct state. The log is the source of truth.

Kafka didn't just use a commit log; it's architected as a distributed commit log. A Kafka topic is nothing more than a sharded log, where each partition is a strict, ordered commit log.

• Producers append records to the end of the log partitions.

• Consumers read the log sequentially, keeping track of their position with an offset.

This design is what makes Kafka so powerful:

• Decoupling: Producers just write to the log. Consumers read at their own pace. They don't need to know about each other.

• Scalability: Throughput is scaled by adding more partitions (and more brokers to host them).

• Replayability: A consumer can reset its offset to the beginning of the log and re-process all the data from scratch. This is invaluable for recovering from bugs in consumer logic.

• Durability: The log is replicated across multiple brokers for fault tolerance.

Kafka effectively turned the commit log implementation detail into the primary public interface.

Once you understand the pattern, you see it everywhere.

• Event Sourcing: Instead of storing the current state of an application, you store the full sequence of events that happened to it. The current state is simply a projection or fold over that log of events. The log is the system's state.

• Distributed System Replication: How do you keep a replica of a system in sync with the primary? The primary writes its operations to a log, and the replicas consume that log and apply the same operations in the same order. This is state machine replication in a nutshell.

• Change Data Capture (CDC): Tailing the database transaction log (the WAL) is the most efficient way to capture every single change (INSERT, UPDATE, DELETE) and stream those changes to other systems like caches, search indexes, or data warehouses.

The commit log is not a complex algorithm. It's a fundamental data structure whose power comes from its constraints. By forcing operations into an immutable, ordered sequence, it provides a foundation for durability, replication, and building complex, decoupled systems that can handle failure and scale effectively.