A Small Compiler for Explaining Delta Lake Pruning
How delta-explain turns one predicate and the Delta log into a trustworthy explanation of file pruning, through shared representation, safe rewrites, and conservative measurement.
cdelmonte.dev
Recent content on cdelmonte.dev
Latest posts
Unpacking Parquet: Explicit SIMD, Scalar Baselines, and What HotSpot Makes of Them
On the JVM, optimizing a hot kernel is not only about writing faster code: it is also about understanding how much the result depends on the machine code HotSpot derives from the scalar loop. Using Parquet bit-unpacking as a concrete case, the piece shows that a SIMD speedup depends on which scalar baseline C2 is handed, when explicit vectorization is actually justified, and why a more specialized…
Where Spark Changes Shape: UnsafeRow, the JVM, and the Relocation of Portability
Run df.explain on a Parquet scan and you find ColumnarToRow, an operator whose only job is to change the data’s shape. It is the seam where two architectural eras meet, and a record of how portability in analytical systems has relocated from the JVM runtime to the data format and the query plan.
Anti-patterns in Catalyst rules
Six concrete anti-patterns I encountered building a real Catalyst extension: from the wrong rule type for throws, to mutable state under AQE and Spark Connect, to JVM bootstrap traps in PySpark.
The Hidden DSL in Catalyst
How Spark’s internal rewriting framework, Catalyst, exposes an embedded DSL with a public extension surface, the same one Delta Lake and Iceberg use to plug into the optimizer pipeline.
delta-explain: Making Delta Lake Pruning Visible
Partition pruning and data skipping are invisible by default. delta-explain reads the Delta log directly and shows, step by step, how a WHERE predicate narrows down candidate files, with no engine required.
The Disaggregation of the Lakehouse Stack
How Delta Kernel, Arrow, and pluggable execution are disaggregating the lakehouse stack. The lakehouse stack is not converging on a new dominant engine. It is converging on a layered architecture in which protocol, data representation, and query execution are increasingly isolated behind stable interfaces.
Where Data System Abstractions Break: A Semiotic Reading
Many of the most surprising performance pathologies in modern data systems are semiotic failures: structural divergences between what an interface signifies and what the underlying system does.
Delta Lake MERGE Is Not a Simple Upsert. What Actually Happens at Scale.
At 10 TB, updating 200k rows can mean rewriting thousands of files. Here’s why, and what to do about it.
A Shared Kernel Is a Shared Trust Domain
Containers isolate processes, not trust boundaries. When your platform runs untrusted code, the architectural question is where you place the kernel boundary, and what that costs in memory, latency, and operational complexity.
Fixing Skewed Nested Joins in Spark with Asymmetric Salting
In large-scale Spark pipelines, skew can occur when a single key carries a disproportionately large nested payload. Asymmetric salting offers a targeted solution: explode, salt, join in parallel, and optionally re-aggregate.
Spark Is Not Just Lazy. Spark Compiles Dataflow.
Why calling Spark ’lazy’ is technically reductive, and how thinking of it as a dataflow compiler changes the way you design pipelines.
Two Algorithms, One Intuition: Shunting Yard and Pratt Parsing
Parsing arithmetic expressions looks simple… until precedence enters the picture. Two classic algorithms, Dijkstra’s Shunting Yard and Pratt’s Top-Down Operator Precedence, provide radically different answers that reveal the same underlying intuition.
Inside GKE Workload Identity: How Kubernetes Identities Become GCP Service Accounts
GKE Behind the Scenes: Understanding the Interaction Between Kubernetes and GCP Service Accounts Through The Metadata Server.