A Modern Data Benchmark
I benchmarked dbt against what an LLM would build from scratch. Across 7 models, warehouse+dbt had a 5% pass rate. App-unified architectures had 38-48%.
Matt Arderne
Data Engineering.
Latest posts
Notes on Building Agentic Tools Using Local LLMs
Over the Christmas break, I decided to explore code execution for AI agents, inspired by Anthropic's blog on the topic.
The Way of Ways
a Tour de Links that follows the journey of the Modern Data Stack.
Context, and the Lack Thereof
Enterprise data systems can often be quite distinct from their smaller startup cousins. This post takes a look at how our conversations around data systems and techniques need to be more sensitive to context, specifically when conversationalists have varying backgrounds. A puddle-deep dive into the world of enterprises and how they compare to startups.
The Ocean is Not Full
Seeking flow state on the foil in the first week of September, a reflection on the pursuit of an open-faced wave.
Data Navigators
Rich with historical trivia, linking to great practical insights, and sublime section headings. A must skim for any slow Friday - Data Workers Daily
Foils Have Hooks
Foils are pretty easy to ignore, ask me. I managed for nearly 8 years. I managed to miss the barbed hooks and swim on. Surfing, bouldering, drones, wave pools, offshore sailing for a seasick summer, you name it. This is a story about how you need patience to let the barbs catch.
The future history of Data Engineering
On Data Engineers and their place in a Data SaaS world. This is a narrative for the near future of Data Engineering in startups, and I think makes some interesting points. I do think the post has avenues for expansion, especially counter-arguments in the context of enterprise tech.
Getting into Data
Rapid fire thoughts on transitioning from a technical role into data. I’ve been asked frequently enough about making a transition into the Data Analytics space, aka my day job, that I thought it would be useful to combine my thoughts into a coherent post. This is a quick take on who this industry/job/role suits, what the skills required typically look like, and some background info on the industry…
dataform and dbt
A quick rundown on two of the “indicative-of-the-future” SQL tools in data analytics at the moment. Dataform and dbt. Welcome to my third post, one I have wanted to write from the beginning. Getting these posts done isn’t easy, and the time between publishing is a commitment that I undertook rather lightly. Like most good ideas, this one is late, irrelevant, and likely only to be marginally…