No new episodes this month. I may have been AWOL in August but the project itself never stops moving: 4.4.0 cleared both feature freeze and code freeze, and a follow-up Cluster Mirroring KIP landed. Plus the usual selection of choice blogs for your perusal.
I grumbled about what I see as an emerging pattern of people building the same tools in corners without collaborating on r/apachekafka. I think the best part of the Open Source community is people collaborating and innovating together and I think AI coding tools have encouraged a strange attitude of creating something new rather than participating in this. Let me know what you think: does this also bother you or is this an old man yells at cloud moment?
Apache Kafka performance #1 - linger.ms kicks off a new ad-hoc series from Jack Vanlightly, and it’s another belter. He noticed Kafka 4.3 looked meaningfully faster than 3.7.2 in his benchmarks, then traced most of the gain back to a single client default: linger.ms moved from 0 to 5ms in Kafka 4.0. He walks through the math of how linger.ms interacts with your per-producer, per-partition send rate, and shows why a linger that transforms tail latency at 100K records/s does nothing at 5K. Like a lot of ASF projects, defaults can be a real gotcha when it comes to performance, Jack does a great job exploring quite how significant this particular changed default is.
Deploying Apache Kafka® Streams next to your data from my lovely colleague Tony Ibbs is an extremely useful, practical resource if you’re looking to get started working with Kafka Streams. Tibs has an example for understanding all the moving parts of a Kafka Streams architecture: Avro messages, covering both GenericAvroSerde and SpecificAvroSerde, with instructions for running it standalone or deploying it as an Aiven App against a Karapace schema registry.
Kafka 4.4.0 hit both of its remaining milestones this month: feature freeze on July 29, and code freeze on August 12, with Omnia Ibrahim as release manager. Per the release plan, expect at least four weeks of stabilisation after code freeze, so the earliest realistic release date is September 9. KIP-1279 (Cluster Mirroring), the big one everyone’s been tracking, remains under discussion on the dev mailing list and missed the 4.4.0 cut again. If you want to learn more about cluster mirroring why not check out my conversation with Omnia Ibrahim on the topic!
A follow-up proposal, KIP-1360: Cluster Synchronous Mirroring, builds directly on KIP-1279. It proposes a synchronous mode where a record only counts as mirrored once every ISR (in-sync replica) member on every mirror cluster has received it, opening the door to zero replication factor disaster recovery setups. I know I’ll be keeping a close eye on this one.
July’s had a big batch of new KIPs, here’s some I’d like to give a belated mention: KIP-1365 improves observability for records that Kafka Connect transformations skip due to errors or filters, adding metrics and a callback so you can actually see what’s being dropped and why. KIP-1366 makes the segment interval for Streams window and session stores configurable, rather than pinned to half the retention period, helpfully avoiding retention changes forcing a full state restore. And KIP-1368 lets clients report the framework they’re embedded in (Streams, Connect, or a third-party wrapper), not just the client library itself.
Thanks for reading, and keep an eye out for the next edition next month for further updates from the world of Apache Kafka.
Sources & Further Reading
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.