Kafka
Kafka is a good thing.
TopicPartition
Last 10 notes on TopicPartition
Latest posts
Kafka Community Spotlight 🔦
Kafka Community Spotlight is an initiative to bring more life, depth, and visibility to the Kafka community by regularly highlighting the people involved in Kafka ...
KIP-1150 in Apache Kafka is a big deal (Diskless Topics)
TL;DR KIP-1150 introduces Diskless Kafka topics that write directly to S3 instead of replicating between brokers. It literally reduces costs by 97% (from $1.8M to $20K annually for a 1GiB/s cluster) and brings operational benefits for diskless topics like: no disk management headaches, instant broker scaling, and simplified load balancing.
KIP-1150 Diskless Topics cost savings compared to Confluent Freight
TL;DR KIP-1150 introduces Diskless Kafka topics that write directly to S3 instead of replicating between brokers. It literally reduces costs by 5x+ off of the most optimized Kafka deployment (10x+ otherwise) and brings operational benefits for diskless topics like: no disk management headaches, instant broker scaling, and simplified load balancing.
Ursa - a new Iceberg-first storage engine for Kafka
In case you haven’t been following, the event streaming Kafka space has been in fierce competition recently1. What used to be dominated by unicorn-turned-public-company Confluent now consists of 20+ vendors (of which IBM controls 5 now after it bought Confluent)2.
Consumer Groups in Kafka
Consumers are clients that read data from Kafka via the KafkaConsumer API. Consumer Groups are a bunch of Consumers that share state and coordinate how to distribute the work between them.
The Log Data Structure
The Log is the data structure that Kafka is based on. It’s very simple. It’s an ordered structure of bytes that only supports appends — you can’t edit or delete records in place.
Topic Partition
a topic partitionn.
Welcome to TopicPartition
This is a blog about Apache Kafka. Here’s some of my best writing: Kafka is fast — I’ll use Postgres Why Was Apache Kafka Created? No One Will Tell You The Real Cost of Apache Kafka If you’d like to reach out, you can find me on LinkedIn or X (Twitter).
Everyone is talking their book
An uncomfortable truth is that in public tech circles, everyone is just shilling their product. Everyone is talking their book. Can’t blame them - when your competition is, you need to too, in order to stand out.