After the excitement of Current London July has seen 4.3.1 tidied up after 4.3.0, 4.4.0 planning underway, and the community busy filing new KIPs
At Current London last month, Stanislav Kozlovski hosted an impromptu fireside chat with Nahidul Kibria, Staff Engineer at HelloFresh, and Sion Smith, CTO at OSO, covering Kafka operations at scale, the current state of the streaming ecosystem, and proxies.
Watch the full recording here.
Why Apache Kafka Groups Are Not True Queues is an insightful dive into this new feature from my colleague at Aiven Olena Babenko. Olena explains that whilst Kafka 4.2 Share Groups bring elastic scaling and per-message control it does so without strict ordering. Worth a read if you want to understand when that trade-off is exactly what you need.
Apache Kafka 4.3.0: A guide for platform engineers from the wonderful team at Factor House is a solid, practical overview of the 4.3.0 release oriented specifically at people who have to decide whether and when to upgrade. It focuses on what actually changes for operators: the config defaults, deprecations, and the items you genuinely need to act on.
Also on the topic of Share Groups, Jack Vanlightly’s latest piece Kafka Share Groups - Pathological fetch waits with record_limit () documents a specific failure mode where fetch waits become pathological under certain record_limit configurations. This is the kind of edge case that won’t bite you in testing but will show up at production load. Worth reading alongside the earlier posts in the series if you’re evaluating share groups for anything approaching production use.
Apache Kafka 4.3.1 was released on June 25, with Bill Bejeck as release manager. The headline fix is a significant RocksDB native memory leak in Kafka Streams (KAFKA-20616) that can cause OutOfMemory errors in 4.3.0 if you’re running Streams—if you upgraded to 4.3.0 and haven’t pulled 4.3.1 yet, this one is worth prioritising.
Looking ahead, 4.4.0 planning has started and Omnia Ibrahim (who you’ll know from last month’s episode on KIP-1279 and cross-cluster replication!) has volunteered as release manager. KIP-1279 (Cluster Mirroring) remains the headline feature being targeted for 4.4 after missing the 4.3 cut.
KIP-1333, which proposes removing the current ~2GB log segment size limit, is progressing on the dev mailing list. The proposal would allow sizes up to Long.MAX_VALUE requiring a small but meaningful change in the API, potentially benefitting high-throughput topics and long retention periods.
KIP-1338, which would add Mapped Diagnostic Context (MDC) to all Kafka component logs, is also moving forward. Perhaps unglamorous in name this is an important KIP that makes a real difference in production affording correlated log analysis across a multi-broker cluster becomes significantly easier when every log line carries the same contextual metadata.
Thanks for reading, and keep an eye out for the next edition next month for further updates from the world of Apache Kafka.
Sources & Further Reading
• Apache Kafka Dev mailing list
• Apache Kafka User mailing list
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.