Recent content on Gunnar Morling

 Table of Contents 
 
 Parquet’s Dremel Encoding 
 Reading Effectively-Fixed-Length Lists Faster 
 Performance Gains 
 Summary 
 
 
 
 In its current form Apache Parquet isn’t a great fit for storing fixed-length lists, such as coordinates, RGB(A) colors, or—an increasingly common case—vector embeddings driving search and retrieval workloads.
A…

 Table of Contents 
 
 Why Hardwood 
 What’s in Hardwood 1.0 
 Performance 
 The Hardwood CLI 
 Building Open-Source With AI 
 A Big Thank You 
 What’s Ahead 
 
 
 
 Hardwood is a new Parquet library for the JVM, written from scratch to do one thing well: read (and soon, write) Apache Parquet files fast, with no mandatory dependencies.
It is…

 Table of Contents 
 
 Reworked ColumnReader API 
 Geospatial Support 
 Documentation Overhaul 
 Further Fixes and Improvements 
 
 
 
 I am happy to announce the release of Hardwood 1.0.0.CR1 ! 
 
 
 This first candidate release of Hardwood 1.0 brings a substantially improved API for columnar access to Apache Parquet files,
initial support…
I had the pleasure to do a few podcasts and interviews, e.g. talking about Debezium, change data capture, stream processing, my career, and software engineering in general.
Interview with InfoQ: Chasing Efficient Java Development: From 1BRC to Developing Hardwood AI Natively
Confluent Developer Podcast: Ep. 23 - The 1 Billion Row Challenge with Gunnar Morling
Unapologetically Technical…
Table of Contents 2026 2025 2024 2023 2022 2021 2020 2019 2018 2017 2016 2013 This page gives an overview over some talks I have done over the last years. I have spoken at large conferences such as QCon San Francisco, Devoxx and JavaOne, local meet-ups as well as company-internal events, covering topics such as Debezium and Change Data Capture, Bean Validation, NoSQL and more.
If you’d like to…

 Table of Contents 
 
 VARIANT Support 
 Hardwood CLI TUI 
 Unified Reader API 
 Performance Improvements 
 Wrapping Up 
 
 
 
 I am happy to announce the release of Hardwood 1.0.0.Beta2! 
 
 
 The latest version of this new parser for Apache Parquet comes with support for VARIANT columns,
an interactive text-based UI (TUI) for examining…

 Table of Contents 
 
 S3 Backend 
 Predicate Push-Down 
 Avro Bindings 
 hardwood-cli 
 Wrapping Up 
 
 
 
 I am pleased to announce the release of Hardwood 1.0.0.Beta1! 
 
 
 Hardwood is a new parser for Apache Parquet, optimized for minimal dependencies and great performance.
Since the project’s initial release just a few weeks back ,…
I have contributed to a wide range of open-source projects over the last years. Here’s a selection of projects I have been involved with.
1BRC 1️⃣🐝🏎️ The One Billion Row Challenge, or 1BRC for short, is a fun exploration of how quickly 1B rows from a text file can be aggregated with Java. It is a coding challenge I ran in January 2024, which provided an opportunity to learn about modern Java…

 Table of Contents 
 
 Why Hardwood? 
 Hello, Hardwood! 
 Parsing Performance 
 Built With AI, Not By AI 
 What’s Next? 
 
 
 
 Today, it’s my great pleasure to announce the first public release of Hardwood, a new parser for the Apache Parquet file format, optimized for minimal dependencies and great performance. 
 
 
 Hardwood is…

 Table of Contents 
 
 Materialized Views 
 Embracing Data Duplication 
 Streams for machines, tables for humans 
 
 
 
 Historically, data management systems have been built around the notion of pull queries : users query data which, for instance, is stored in tables in an RDBMS, Parquet files in a data lake, or a full-text index in Elasticsearch. When a user…

 Table of Contents 
 
 UUIDs 
 Monotonically Increasing Sequences 
 Deriving Idempotency Keys From the Transaction Log 
 Discussion 
 
 
 
 In distributed systems, there’s a common understanding that it is not possible to guarantee exactly-once delivery of messages.
 What is possible though is exactly-once processing . By adding a unique idempotency key…

 Table of Contents 
 
 Hello Persistasaurus! 
 Capturing Execution State 
 Delayed Executions 
 Human Interaction 
 Managing State 
 Wrapping Up 
 
 
 
 Lately, there has been a lot of excitement around Durable Execution (DE) engines.
The basic idea of DE is to take (potentially long-running) multi-step workflows,
such as processing a…

 Looking to make it to the front page of HackerNews? Then writing a post arguing that 'Postgres is enough', or why 'you don’t need Kafka at your scale' is a pretty failsafe way of achieving exactly that. No matter how often it has been discussed before, this topic is always doing well. And sure, what’s not to love about that? I mean, it has it all: Postgres, everybody’s most favorite…

 Table of Contents 
 
 ZGC Allocation Stalls 
 Summary 
 
 
 
 In the 'Let’s Take a Look at…!' blog series I am exploring interesting projects, developments and technologies in the data and streaming space. This can be KIPs and FLIPs, open-source projects, services, relevant improvements to Java and the JVM, and more. The idea is to get some hands-on experience,…

 Table of Contents 
 
 confirmed_flush_sn: Tracking Consumer Progress 
 restart_lsn: Handling Concurrent Transactions 
 Mid-Transaction Recovery 
 Looking Forward: Streaming In-Progress Transactions 
 
 
 
 Replication slots in Postgres keep track of how far consumers have read a replication stream.
After a restart, consumers—either Postgres read…

 This post explores how virtual threads in Java 21+ provide an elegant solution for converting legacy Future objects into CompletableFuture instances. 
 
 
 Since Java 8, the CompletableFuture API provides a convenient way for performing asynchronous operations in a functional, composable way.
This makes it very simple to call some long-running methods—for instance involving…

 Table of Contents 
 
 Use the pgoutput Logical Decoding Output Plug-in 
 Define a Maximum Replication Slot Size 
 Enable Heartbeats 
 Use Table-level Publications 
 Use Column and Row Filters 
 Enable Fail-Over Slots 
 Consider Using Replica Identity FULL 
 Monitor, Monitor, Monitor! 
 Drop Unused Replication Slots 
 Summary 
 
 
 
…
Table of Contents What I Use AI For What I Do Not Use AI For When you read a post on this site, you can be sure it is an original article, written by a human: me.
My goal is to share unique (in any case, personal), and hopefully innovative perspectives, based on my own work and experience. Large language models (LLMs) are not going to give you that. I don’t see much value in publishing…

 Table of Contents 
 
 Agents Need to Interact With LLMs 
 Agents Should Be Event-Driven 
 Agents Need Context 
 Agents Require Memory 
 When SQL Is Not Enough 
 Parting Thoughts 
 
 
 
 AI Agents have improved in leaps and bounds in recent times, moving beyond simple chatbots to sophisticated, autonomous systems. This post explores a novel approach…

 Table of Contents 
 
 Debezium Reselect Postprocessor 
 Flink DataStream API 
 Flink SQL With OVER Aggregation 
 Flink Process Table Functions 
 Summary and Discussion 
 
 
 
 Postgres logical replication, while powerful for capturing real-time data changes, presents challenges with TOAST columns,
whose values can be absent from data change events…

 Often times, 'Stream vs. Batch' is discussed as if it’s one or the other, but to me this does not make that much sense really. 

Table of Contents Speaker Information 🇬🇧 English 🇩🇪 Deutsch I am an open-source software engineer in the Java and data streaming space. I currently work as a Technologist for Confluent. In my past role at Decodable I focused on developer outreach and helped them build their stream processing platform based on Apache Flink. Prior to that, I spent ten years at Red Hat, where I led the Debezium…

 The last few days I spent some time digging into the recently announced KIP-1150 ('Diskless Kafka'), as well AutoMQ’s Kafka fork , tightly integrating Apache Kafka and object storage, such as S3. Following the example set by WarpStream, these projects aim to substantially improve the experience of using Kafka in cloud environments, providing better elasticity, drastically reducing cost, and…

 Table of Contents 
 
 Flink SQL Connectors for Apache Kafka 
 The Apache Kafka SQL Connector in Append-Only Mode 
 The Apache Kafka SQL Connector As a Changelog Source 
 The Upsert Kafka SQL Connector 
 Summary 
 
 
 
 Over the years, I’ve spoken quite a bit about the use cases for processing Debezium data change events with Apache Flink ,
such as…

 Table of Contents 
 
 KIP-974: Docker Image for GraalVM based Native Kafka Broker 
 
 
 
 With help of the GraalVM configuration developed for KIP-974 (Docker Image for GraalVM based Native Kafka Broker),
you can easily build a self-contained native binary for Apache Kafka.
Read on to learn how you can build a native Kafka executable yourself,
starting in…

 Table of Contents 
 
 Building an AOT Cache for Apache Kafka 
 AOT Caching With Apache Flink 
 Summary 
 
 
 
 In the 'Let’s Take a Look at…!' blog series I am exploring interesting projects, developments and technologies in the data and streaming space. This can be KIPs and FLIPs, open-source projects, services, relevant improvements to Java and the JVM, and…

 For building a system of distributed services, one concept I think is very valuable to keep in mind is what I call the synchrony budget :
as much as possible, a service should minimize the number of synchronous requests which it makes to other services. 


 Table of Contents 
 
 Towards Queue Support in Kafka—Introducing Share Groups 
 Share Groups in Action 
 Retry Behavior and State Management 
 Share Group State Persistence 
 Summary and Outlook 
 
 
 
 In the 'Let’s Take a Look at…!' blog series I am going to explore interesting projects, developments and technologies in the data and streaming…

 Table of Contents 
 
 Fault Tolerance and High Availability 
 Manually Triggering Savepoints 
 Observability 
 Bonus: Managing Flink Jobs With the Heimdall UI 
 Summary and Discussion 
 
 
 
 This post originally appeared on the Decodable blog . All rights reserved. 
 
 
 Welcome back to this two-part blog post series about running Apache…

 Table of Contents 
 
 Installation and Setup 
 Deployment Types 
 Deploying Your First Flink Job on Kubernetes 
 Building Custom Job Images 
 
 
 
 This post originally appeared on the Decodable blog . All rights reserved. 
 
 
 Kubernetes is a widely used deployment platform for Apache Flink.
While Flink has had native support for…

 Table of Contents 
 
 Hello, Failover Slots! 
 Failover Slots in Decodable 
 Wrapping Up 
 
 
 
 This post originally appeared on the Decodable blog . All rights reserved. 
 
 
 Postgres read replicas are commonly used not only to distribute query load amongst multiple nodes, but also to ensure high availability (HA) of the database.
If the…

 If you are following the news around Debezium—an open-source platform for Change Data Capture (CDC) for a variety of databases—you may have seen the announcement that the project is in the process of moving to the Commonhaus Foundation . I think this is excellent news for the Debezium project, its community, and open-source CDC at large. In this post I’d like to share some more context on…

 Every now and then, it can come in very handy to build OpenJDK from source yourself,
for instance if you want to explore a feature which is under development on a branch for which no builds are published.
For some reason I always thought that building OpenJDK is a very complex processing,
requiring the installation of arcane tool chains etc.
But as it turns out, this actually…

 Table of Contents 
 
 Recap: What’s The Outbox Pattern? 
 Implementation Considerations 
 
 Polling vs. Log-Based CDC 
 The Outbox Table 
 pg_logical_emit_message() 
 Format Considerations 
 Backfills 
 Idempotency for Consumers 
 
 
 Criticisms of the Outbox pattern 
 
 Database Overhead 
 Complexity 
 Latency 
…

 During and after my time as the lead of Debezium ,
a widely used open-source platform for Change Data Capture (CDC) for a variety of database,
I got repeatedly asked whether I’d be interested in creating a company around CDC.
VCs, including wellknown household names, did and do reach out to me,
pitching this idea. 


 Table of Contents 
 
 Creating Instances 
 Configuring SSH 
 Provisioning Software 
 Try It Out Yourself 
 
 
 
 Whenever I’ve need a Linux box for some testing or experimentation,
or projects like the One Billion Row Challenge a few months back,
my go-to solution is Hetzner Online , a data center operator here in Europe. 
 
 
 Their…

 Table of Contents 
 
 The Algorithm 
 Obtaining the Lock 
 Expiring a Lock 
 Lock Validity 
 Fencing Off Zombies 
 
 
 
 In distributed systems, for instance when scaling out some workload to multiple compute nodes,
it is a common requirement to select a leader for performing a given task:
only one of the nodes should process the records from a…

 In my day job at Decodable ,
I am currently working with Terraform to provision some cloud infrastructure for an upcoming hands-on lab.
Part of this set-up is a Postgres database on Amazon RDS,
which I am creating using the Terraform AWS modules .
Now, once my database was up and running,
I wanted to extract two dynamically generated values from Terraform:
the random…

 The other day, I was looking for means of zipping two Java streams :
connecting them element by element—essentially a join based on stream offset position—and emitting an output stream with the results.
Unfortunately, there is no zip() method offered by the Java Streams API itself.
While it was considered for inclusion in early preview versions,
the method was removed…

 Table of Contents 
 
 Full Events 
 Delta Events 
 Id-only Events 
 Change Event Metadata 
 Comparison 
 
 
 
 This post originally appeared on the Decodable blog . All rights reserved. 
 
 
 Data change events are at the core of Change Data Capture (CDC) solutions such as Debezium .
They describe the changes made to a specific record…

 In many applications it’s a requirement to keep track of when a record was created and updated the last time.
Often, this is implemented by having columns such as created_at and updated_at within each table.
To make things as simple as possible for application developers,
the database itself should take care of maintaining these values automatically when a record gets inserted or…

 Recently I ran into a situation where it was necessary to capture the output of a Java process on the stdout stream,
and at the same time a filtered subset of the output in a log file.
The former, so that the output gets picked up by the Kubernetes logging infrastructure.
The letter for further processing on our end:
we were looking to detect when the JVM stops due to an…

 Table of Contents 
 
 Results 
 Bonus Result: 32 Cores, 64 Threads 
 Bonus Result: 10K Key Set 
 Thank You! 
 Which Challenge Will Be Next? 
 
 
 
 Oh what a wild ride the last few weeks have been.
The One Billion Row Challenge (1BRC for short),
something I had expected to be interesting to a dozen folks or so at best,
has gone kinda viral,…

 Update Jan 4: Wow, this thing really took off! 
 1BRC is discussed at a couple of places on the internet, including Hacker News , lobste.rs , and Reddit . 
 
 
 For folks to show-case non-Java solutions, there is a 'Show & Tell' now, check that one out for 1BRC implementations in Rust, Go, C++, and others. 
 Some interesting related write-ups include 1BRC in SQL with…

 Table of Contents 
 
 Stand-By Logical Replication With Debezium 
 Towards Fail-Over Slots 
 Wrap-Up 
 
 
 
 This post originally appeared on the Decodable blog . All rights reserved. 
 
 
 Welcome back to this series about logical replication from Postgres 16 stand-by servers, in which we’ll discuss how to use this feature with Debezium—a popular…

 Table of Contents 
 
 What is Postgres Logical Replication? 
 Why Logical Replication On Stand-By Servers? 
 Provisioning a Testing Environment 
 Testing Things Out 
 
 
 
 This post originally appeared on the Decodable blog . All rights reserved. 
 
 
 For users of Change Data Capture (CDC), one of the most exciting features in Postgres…

 Table of Contents 
 
 An Example 
 Tracking RSS 
 
 
 
 As regular readers of this blog will now, JDK Flight Recorder (JFR) is one of my favorite tools of the Java platform.
This low-overhead event recording engine built into the JVM is invaluable for observing the runtime characteristics of Java applications and identifying any potential performance…

 Table of Contents 
 
 What Is PyFlink and Why Should You Care? 
 Prerequisites 
 
 Installing the Flink Kubernetes Operator 
 Installing Strimzi and Apache Kafka 
 
 
 A Simple PyFlink Job 
 Building a Container Image With Your PyFlink Job 
 Deploying a PyFlink Job On Kubernetes 
 
 
 
 This post originally appeared on the Decodable…

 Table of Contents 
 
 CDC—A Quick Primer 
 Does CDC Break Encapsulation? 
 Entering Data Contracts 
 Implementation Approaches For Data Contracts 
 
 The Outbox Pattern 
 Stream Processing 
 
 
 Streaming Data Contracts—Beyond the Basics 
 Handling Schema Changes 
 Summary 
 
 
 
 This post originally appeared on the…

 This question came up on the Data Engineering sub-reddit the other day:
 Can Debezium lose any events ?
I.e. can there be a situation where a record in a database get inserted, updated, or deleted, but Debezium fails to capture that event from the transaction log and propagate it to downstream consumers?