Why you might lease a blob: an intro to job locks
Use blob leases to build a reusable, globally coordinated job lock for data engineering workloads.
Miles Cole
Miles Cole's personal data engineering blog covering data systems, performance, and the details that matter.
Latest posts
How Incremental Liquid Clustering Works
Learn how Fabric Spark Runtime 2.0 incrementally clusters Delta tables, limits write amplification, and maintains file-skipping quality.
Mastering Spark: DataFrameWriterV2 vs. DataFrameWriterV1
Most Spark developers learn to write data with df.write long before they ever encounter df.writeTo. It is simple, familiar, and everywhere: choose a format, pick a mode, add a few options, and save the result to a table or path. For years, that mental model worked well enough. Spark was often writing files first and tables second.
Mastering Spark: Should You Infer Schema in Production?
Schema inference is convenient. In production or benchmarking, it is often a silent performance killer.
Creating your first Spark Job Definition
Coming from a notebook-first Spark background, I wanted to write the introduction to Spark Job Definitions (SJDs) that I wish I had when I first encountered them. If you are first interest in why you might want to use a Spark Job Definition over a Notebook, see my blog here.
Notebooks, Spark Jobs, and the Hidden Cost of Convenience
I’m guilty. I’ve peddled the #NotebookEverything tagline more than a few times.
Announcing: 🌊 LakeBench
I’m excited to formally announce LakeBench, now in version v0.3, the first Python-based multi-modal benchmarking library that supports multiple data processing engines on multiple benchmarks. You can find it on GitHub and PyPi.
The Small Data Showdown ‘25: Is it Time to Ditch Spark Yet??
Last December (2024) I published a blog seeking to explore the question of whether data engineers in Microsoft Fabric should ditch Spark for DuckDb or Polars. Six months have passed and all engines have gotten more mature. Where do things stand? Is it finally time to ditch Spark? Let The Small Data Showdown ‘25 begin!
Elevate Your Code: Creating Python Libraries Using Microsoft Fabric (Part 2 of 2: Packaging, Distribution, and Consumption)
This is part 2 of my prior post that continues where I left off. I previously showed how you can use Resource folders in either the Notebook or Environment in Microsoft Fabric to do some pretty agile development of Python modules/libraries.
Mastering Spark: The Art and Science of Table Compaction
If there anything that data engineers agree about, it’s that table compaction is important. Often one of the first big lessons that folks will learn early on is that not compacting tables can present serious performance issues: you’ve gotten your lakehouse pilot approved and it’s been running for a couple months in production and you find that both reads and writes are increasingly getting slower…