Top-ranked Data Engineering Blog with Genuine Insights into the Data Ecosystem with 1000s of Second Brain notes and an online book on Patterns of Data Engineering, all connected with interactive graphs.
It’s hard to keep up with the AI evolution; new AI tools drop every week, but how are experienced practitioners actually using them? Most of us are overwhelmed and unsure about the many possibilities, yet we need to keep going and do our work. You might use AI agents all day long, parallelize them with AI Orchestrators, tmux, git worktree, and so on, using AI IDEs, but in the end, you still…
These are my recently read books and some comments and notes from when I read them. As I love books and recommendations by others, I want to share them in a collective format so others can read gems of books, as I think books are still the best way to read new information these days. Even more with the fast pace we are going at, as books are well structured and made for the long term. Book…
Creation is the ultimate form of pursuing ourselves, giving to the world when shared, and using the power of our subconscious. It gives us joy, and to every artist, it is the ultimate (flow) state of happiness. The Act of Creation The act of creation is an outlet. It gives joy to us when we create something out of nothing, we block out anxiety or boredom. Creating should be done like: a kid in…
Grammars for languages or any other field are a beautiful thing. They compress complex systems into a language with a couple of rules. For the spoken language example, we know when to capitalize a letter or how to start a sentence. There are clear rules. Grammars also help us remember, as we do not need to recall every little rule, but apply them in a structured way. For text editing, we have Vim…
With ever-changing models, new and better ones coming out every few months, it’s great if we don’t have to rely on them too heavily. The better your tooling, the less dependent you become on any single model. That’s also why the deterministic harness matters: a correctness layer that lets you reproduce outputs and trace lineage regardless of which model you’re running…
I’ve taken notes over my whole life. As a Swiss, I’m very particular about organization and structured in filing my knowledge. I’ve tried and taken notes in many different ways, from forgetting everything, to noting down every detail, to only taking listings of the most important things. Today, I can say, there’s no right way, but I can see it’s working for me, as my…
Operating the data orchestration layer can be as critical as the source OLTP databases, when decisions are taken from the data assets produced by downstream data pipelines. When orchestration gets at the heart of triggering and observing most of the schedules or assets, there is even more importance involved. In Part 1 I argued that the open data platform makes orchestration data and its flow…
This series interviews real practitioners to extract the patterns behind how they actually use AI in their data work today. This is the second interview in ‘How to use AI with DE’, and this time we have none other than Wes McKinney. Creator of Pandas, probably the most widely used data analysis library for Python, Wes has shaped the era of data and is co-creator of Apache Arrow. He…
Writing SQL was never the hard part. Making it accurate and trustworthy against your warehouse always was. Point an AI agent like Claude or Codex at your data stack and ask a real analytics question, and the answer is usually mediocre: the agent can scrape some context from your git repos or whatever metadata it can find, but it doesn’t know your joins, your metric definitions, or the…
I have read the “Poor Charlie’s Almanack” by Charlie Munger and thought about what it would take to write one for Dagster. A complete guide with all the insights, tips, and some predictions for the data platform engineer, just like an Almanack provides, with practical information for daily life. My goal is to offer a collection of wisdom, insights, and principles gathered over…