# data stack (blogs) — RSS Amplifier Recent posts from the 4 feeds in the RSS Amplifier directory that cover data stack. Page: Feed: --- ## [Figma for Agents: How Airflow's Creator Coordinates AI ft. Maxime Beauchemin](https://www.ssp.sh/blog/figma-for-agents-max-beauchemin/) _2026-08-06 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ It’s hard to keep up with the AI evolution; new AI tools drop every week, but how are experienced practitioners actually using them? Most of us are overwhelmed and unsure about the many possibilities, yet we need to keep going and do our work. You might use AI agents all day long, parallelize them with AI Orchestrators, tmux, git worktree, and so on, using AI IDEs, but in the end, you still… ## [The Act and the Outcome of Creation](https://www.ssp.sh/blog/on-creation/) _2026-07-16 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Creation is the ultimate form of pursuing ourselves, giving to the world when shared, and using the power of our subconscious. It gives us joy, and to every artist, it is the ultimate (flow) state of happiness. The Act of Creation The act of creation is an outlet. It gives joy to us when we create something out of nothing, we block out anxiety or boredom. Creating should be done like: ## [The Grammar of Data: Define Once, Run Anywhere with Cross-Engine Expressions](https://www.ssp.sh/blog/grammar-data-engineering/) _2026-07-08 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Grammars for languages or any other field are a beautiful thing. They compress complex systems into a language with a couple of rules. For the spoken language example, we know when to capitalize a letter or how to start a sentence. There are clear rules. Grammars also help us remember, as we do not need to recall every little rule, but apply them in a structured way. For text editing, we have Vim… ## [Where AI Agents Belong in Data Engineering: The Correctness Layer](https://www.ssp.sh/blog/where-agents-belong-in-de/) _2026-07-07 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ With ever-changing models, new and better ones coming out every few months, it’s great if we don’t have to rely on them too heavily. The better your tooling, the less dependent you become on any single model. That’s also why the deterministic harness matters: a correctness layer that lets you reproduce outputs and trace lineage regardless of which model you’re running… ## [The Process of Smart Note-Taking](https://www.ssp.sh/blog/process-smart-note-taking/) _2026-06-27 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ I’ve taken notes over my whole life. As a Swiss, I’m very particular about organization and structured in filing my knowledge. I’ve tried and taken notes in many different ways, from forgetting everything, to noting down every detail, to only taking listings of the most important things. Today, I can say, there’s no right way, but I can see it’s working for me, as my… ## [Operationalizing Data Orchestration: Best Practices for DevOps, Infra, and Code Locations](https://www.ssp.sh/blog/operationalizing-data-orchestration/) _2026-06-27 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Operating the data orchestration layer can be as critical as the source OLTP databases, when decisions are taken from the data assets produced by downstream data pipelines. When orchestration gets at the heart of triggering and observing most of the schedules or assets, there is even more importance involved. In Part 1 I argued that the open data platform makes orchestration data and its flow… ## [How we build the ToFu LinkedIn CRM](https://nonamevc.substack.com/p/how-we-build-the-tofu-linkedin-crm) _2026-06-22 · Daniel Chepenko · nonamevc_ And no, it’s not “I vibe-coded my own CRM this weekend, I don’t need HubSpot anymore.” post. ## [Vibe Coding Is Dangerous, Agentic Engineering Isn't—Wes McKinney](https://www.ssp.sh/blog/how-to-use-ai-with-de-wes-mckinney/) _2026-06-11 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ This series interviews real practitioners to extract the patterns behind how they actually use AI in their data work today. This is the second interview in ‘How to use AI with DE’, and this time we have none other than Wes McKinney. Creator of Pandas, probably the most widely used data analysis library for Python, Wes has shaped the era of data and is co-creator of Apache Arrow. He… ## [Beyond the Semantic Layer: Building a Context Layer for the Agentic Era](https://www.ssp.sh/blog/agentic-context-layer/) _2026-06-11 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Writing SQL was never the hard part. Making it accurate and trustworthy against your warehouse always was. Point an AI agent like Claude or Codex at your data stack and ask a real analytics question, and the answer is usually mediocre: the agent can scrape some context from your git repos or whatever metadata it can find, but it doesn’t know your joins, your metric definitions, or the… ## [Plan Mode All the Time, Substrait over SQL, and the End of the DE Role ft. Chris Riccomini](https://www.ssp.sh/blog/how-to-use-ai-with-de-chris-riccomini/) _2026-05-26 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ This series interviews (see #1 with Mark Freeman ) real practitioners to extract the patterns behind how they actually use AI in their data work today. This is the second interview in ‘How to use AI with DE’, and this time we have none other than Chris Riccomini . Chris has seen the data stack evolve over the years. He thinks AI will soon handle the majority of data engineering work,… ## [Your 24/7 AI Media Buyer (Sponsored)](https://crawlproof.com/a/y5CM6TOTXGnl) _2026-05-26 · **Sponsored**_ Finds placement, geo, and funnel leaks and proposes fixes you approve ## [The Dagster Almanack: From Complexity to Composability](https://www.ssp.sh/blog/dagster-almanack-open-data-platform/) _2026-05-26 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ I have read the “Poor Charlie’s Almanack” by Charlie Munger and thought about what it would take to write one for Dagster. A complete guide with all the insights, tips, and some predictions for the data platform engineer, just like an Almanack provides, with practical information for daily life. My goal is to offer a collection of wisdom, insights, and principles gathered over… ## [Internal vs. External Storage? What's the Limit of External Tables](https://www.ssp.sh/blog/modern-external-tables-and-evolution/) _2026-05-13 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ When I started my career as a data warehouse engineer and business intelligence engineer in 2003, external tables with materialized views were the standard. We used external tables to integrate CSV files and other data not already in Oracle databases. Oracle External Tables have existed since 2001, and that’s where I first used them. If the Lindy Effect continues to hold, we’ll use… ## [AI Reveals Why BI Still Matters](https://www.ssp.sh/blog/bi-is-not-dead-2026/) _2026-04-21 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Ask a BI engineer what they actually spend their time on: it’s not building dashboards. More often: fixing the join that broke in the overnight pipeline, untangling the metric definition that means three different things to three different teams, or getting last week’s numbers into an Excel by Monday morning. The dashboard was always the easy part. This article looks at how BI evolved,… ## [Specs Over Vibes: Consistent AI Results ft. Mark Freeman](https://www.ssp.sh/blog/specs-over-vibes-interview-mark-freeman/) _2026-04-07 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ There’s so much going on in the AI space, and how to work with AI agents is changing every day. Everyone is overwhelmed and almost numb from so many possibilities, yet you need to find a way to work with AI, not to get left behind, right? You might use AI agents all day long, parallelizing them with AI orchestrators like Agent Teams, Gastown, tmux, git worktree, and AI-based IDEs, but in the… ## [Building an Agent-Friendly, Local-First Analytics Stack with MotherDuck and Rill](https://www.ssp.sh/blog/agentic-friendly-local-first-analytics-stack/) _2026-04-07 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Imagine going from a 100-million-row dataset to an interactive analytics app with just a few prompts. What used to take hours or days can now be done in minutes by combining local-first databases and BI tools with an agentic coding workflow. When Rill bet on YAML dashboards and CLI-first workflows in 2022, they weren’t thinking about AI agents. Neither was MotherDuck when they built… ## [Book review: Vibe Coding: Building production grade software with Gen AI, chat, agents, and beyond](https://www.pramodb.com/index.php/2026/03/26/book-review-vibe-coding-building-production-grade-software-with-gen-ai-chat-agents-and-beyond/) _2026-03-26 · pramod · Pramod Biligiri_ I recommend giving this book by Steve Yegge and Gene Kim a good skim even if not a thorough read. I found its best parts to be their advice about the day-to-day approach a software developer can take with respect to coding assistants (Part 3 of the book). But should you even invest time in \[ \] ## [Why I Still Blog — and Why the Future of Blogging Is Connected](https://www.ssp.sh/blog/why-i-still-blog/) _2026-03-06 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ I’ve been online twenty years, and blogging for ten of them. This is the story and lessons learned of blogging online for a decade. It goes beyond blogging topics and includes note-taking (workflow) , how to write well as well as the medium in which writing works best, and also the format in which writing works long-term such as writing in open formats and methods such as vim motions to… ## [Git for Data Applied: Comparing Git-like Tools That Separate Metadata from Data](https://www.ssp.sh/blog/git-for-data-tools/) _2026-03-03 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Continuing from Part 1 , where we learned what git for data is, how the architecture and use cases work, how you can achieve git-like functionality with different approaches, and how the key is to avoid moving data as much as possible to keep state that can be referenced and rolled back to, but at the same time saving cost by not duplicating all data every time you create a new branch. ## [Building an Obsidian RAG with DuckDB and MotherDuck](https://www.ssp.sh/blog/obsidian-rag-duckdb-sql/) _2026-02-12 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ I always wanted a personal knowledge assistant based on my notes. One that uses Obsidian’s backlinks and connections to surface ideas I’ve forgotten or never thought to link together. So I built one. A RAG system that runs locally with DuckDB as a vector database , then syncs to MotherDuck for a serverless web app running entirely in the browser via WASM. Think of it like J.A.R.V.I.S 1… ## [Arch Linux (Omarchy) — 8 Months Later: The Good, the Bad, and the Fixable](https://www.ssp.sh/blog/linux-omarchy-the-good-bad-and-fixable/) _2026-02-10 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ This is a follow-up to my part 1 of Switching macOS to Arch Linux with Omarchy , where I documented my first months with Arch Linux and \[\[Omarchy\]\], after switching from 15 years of using macOS and Windows on and off at work since 2003. Back then, I had a checklist of basics I needed before I could commit to Linux as a daily driver: Obsidian, a Raycast-like launcher for fuzzy finding files and… ## [Hire AI-Powered Freelancers Fast (Sponsored)](https://crawlproof.com/a/Ku04ZH2X78qe) _2026-02-10 · **Sponsored**_ Browse and hire skilled professionals using AI tools — free to post, free to apply. ## [Why Coinbase and Pinterest Chose StarRocks: Lakehouse-Native Design and Fast Joins at Terabyte Scale](https://www.ssp.sh/blog/starrocks-lakehouse-native-joins/) _2026-02-09 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Why is StarRocks gaining popularity among data engineers who need fast analytics on large-scale data? To find out, I did a deep dive on the companies actually using StarRocks in production, interviewing engineers and studying technical case studies from Coinbase, Pinterest, Fresha, Grab, TRM Labs, and Shopee. They all share a similar pattern: customer-facing analytics on Snowflake got too slow,… ## [A Diary of a Data Engineer](https://www.ssp.sh/blog/diary-of-a-data-engineer/) _2026-01-13 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ You ingest data. You model it. You transform it. You serve it. Someone asks for a change. Everything breaks. You rebuild. This is the loop. It was the loop in 2005 with SSIS and star schemas. It’s the loop in 2025 with dbt and Iceberg, or 2026 with prompting AI agents. The tools change. The loop doesn’t. The Invisible Plumbers When I started my career in 2003, there was no “data… ## [Well Being in Times of Algorithms](https://www.ssp.sh/blog/well-being-algorithms/) _2025-12-22 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ To be able to live well, to avoid anxiety and depression, it’s not that easy anymore in 2025. Everything is designed to grab our attention , and our attention span is declining and \[\[Learning with AI|disallowing people to think and learn\]\]. Algorithms know more about us than our spouses while grabbing our attention constantly. This is bad for our health. Bad for our spirit. Bad for our… ## [Opinionated Data Platforms vs. Open-Source: The Chef’s Choice for Your Data Platform](https://www.ssp.sh/blog/omakase-data-stack/) _2025-12-10 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ With the recent consolidation in the area of Modern Data Stack, the question arises: are open data stacks, a more flexible but labor-intensive way of building data platforms, out of fashion? Are we transitioning into an era where we use unified data platforms that do end-to-end analytics and are integrated with metadata and control plane? Data platforms that make the automatic choices provide a… ## [Simplicity of a Database, but the Speed of a Cache: OLAP Caches for DuckDB](https://www.ssp.sh/blog/olap-cache-for-duckdb/) _2025-12-09 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ A constant struggle in data is to make everything fast. This holds true for the ingest, for the data pipeline, but most certainly for the visualization part. When you use a BI dashboard and present data to users, you most always have a SQL query in the background that can be slightly complex when you have most logic in your data warehouse and persisted as tables so the query from the BI tool is… ## [Dlt+ClickHouse+Rill: Multi-Cloud Cost Analytics, Cloud-Ready](https://www.ssp.sh/blog/finops-dlt-clickhouse-rill/) _2025-12-04 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Following up on Part 1 , where we created an entire end-to-end FinOps project to analyze cloud costs from different hyperscalers with dlt and local parquet files. In this part 2, we set up a version that works cloud-native with ClickHouse Cloud and Rill Cloud. So you can share your dashboard and it scales to any cost data you might have. The work is done through a GitHub action job for… ## [Branch, Test, Deploy: A Git-Inspired Approach for Data](https://www.ssp.sh/blog/git-for-data-theory/) _2025-12-03 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Remember the 2 AM on-call duty when a recent data pipeline broke the production environment? A data pipeline you’ve never touched just corrupted customer records. You need to roll back, fast. Or you want to test a new transformation on real production data before deployment, but recreating a production-like state in dev would take all day. Sounds familiar? This is what a Git strategy for… ## [What I'm doing in GTM as B2B SaaS founder as of Dec 25](https://nonamevc.substack.com/p/what-im-doing-in-gtm-as-b2b-saas) _2025-12-01 · Daniel Chepenko · nonamevc_ This isn’t a post to showcase expertise, and it’s not a theoretical review either. ## [Multi-Cloud Cost Analytics: From Cost-Export to Parquet to Rill](https://www.ssp.sh/blog/cost-analyzer-aws-gcp/) _2025-11-21 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Companies often use multiple platforms simultaneously: Lovable for apps, OpenAI and Claude for AI, AWS and GCP for infrastructure. Tracking costs across these services becomes nearly impossible without a unified dashboard. You’re looking at AWS bills in one place, GCP invoices in another, AI API costs in a third spreadsheet. Combining cloud costs with Monthly Recurring Revenue (MRR) to… ## [Boredom is the New Luxury](https://www.ssp.sh/blog/owning-things-attention/) _2025-11-18 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Today, I sent out a newsletter that is distributed only via email. However, because I believe the content serves as a friendly reminder of how the pendulum is swinging back and offers nuggets of wisdom on how to live life, I’m sharing it here as a blog post as well. The Newsletter: I hope you are all well. It’s not an easy time. I don’t know what to write about, but I also had… ## [Transcript-Driven Stock Discovery (Sponsored)](https://crawlproof.com/a/rRpukfXrBpPd) _2025-11-18 · **Sponsored**_ Ranked, evidence-backed watchlists from indexed executive transcripts for 1–2 quarter horizons ## [How we use Attio for a new PLG startup](https://nonamevc.substack.com/p/how-we-use-attio-for-a-new-plg-startup) _2025-11-18 · Daniel Chepenko · nonamevc_ Engineering the product-led growth movement in Attio ## [I attended YourStory TechSparks 2025](https://www.pramodb.com/index.php/2025/11/07/i-attended-yourstory-techsparks-2025/) _2025-11-07 · pramod · Pramod Biligiri_ I attended Day 1 of the YourStory TechSparks 2025 conference and had a great time there! Some of the talks were very high quality, like for example the conversation with Mukesh Bansal (co-founder of Flipkart and currently heading Nurix.ai), a presentation by Mohit Saxena (CTO at InMobi) and the interview with Rajan Anandan (former head \[ \] ## [4 Senior Data Engineers Answer 10 Top Reddit Questions](https://www.ssp.sh/blog/data-engineers-roundtable-reddit/) _2025-11-03 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Every day, thousands of data engineers scroll through r/dataengineering (174K members strong) looking for answers to the same fundamental questions: How do I prepare for interviews in this market? What do I do about data quality? Should I use a data warehouse or jump on the lakehouse bandwagon? We analyzed the most-upvoted questions and concerns—the ones with hundreds of comments that capture the… ## [Data Modeling for the Agentic Era: Semantics, Speed, and Stewardship](https://www.ssp.sh/blog/agentic-data-modeling/) _2025-10-17 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ In data analytics, we’re facing a paradox. AI agents can theoretically analyze anything, but without the right foundations, they’re as likely to hallucinate a metric as to calculate it correctly. They can write SQL in seconds, but will it answer the right business question? They promise autonomous insights, but at what cost to trust and accuracy? These days, everyone is embedding AI… ## [Beyond Basic ETL: Enterprise Data Capabilities Without the Complexity](https://www.ssp.sh/blog/beyond-basic-etl/) _2025-10-13 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Most data teams spend 80% of their time wrestling with infrastructure—writing custom UPSERT logic, building incremental loading from scratch, or debugging why their Airflow DAGs failed again at 2 AM. Meanwhile, the business is still waiting for those critical data insights. This article is a follow-up to Part 1 , where we explored declarative data stacks. Here, we dive into the specific… ## [Why I Don't Research; and Write from Experience](https://www.ssp.sh/blog/why-i-dont-research/) _2025-09-17 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ When I \[\[Writing|write\]\], I deliberately don’t research. The fun of writing is to express myself and explain something in my head. Not researching something and losing focus. Recall what I read is not fun to me. I want to share my opinions. In a way, my research is taking notes on my Second Brain , all the time, everywhere. But not to one topic, about everything, not connected yet. When I… ## [Data Modeling Guide for Real-Time Analytics with ClickHouse](https://www.ssp.sh/blog/practical-data-modeling-clickhouse/) _2025-09-05 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Querying billions of weather records and getting results in under 200 milliseconds isn’t theory; it’s what real-time analytics solutions provide. Processing streaming IoT data from thousands of sensors while delivering real-time dashboards with no lag is what certain business domains need. That’s what you’ll learn at the end of this guide through building a… ## [Why Semantic Layers Matter—and How to Build One with DuckDB](https://www.ssp.sh/blog/semantic-layer-duckdb/) _2025-08-24 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Many ask themselves, “Why would I use a semantic layer? What is it anyway?” In this hands-on guide, we’ll build the simplest possible semantic layer using just a YAML file and a Python script—not as the goal itself, but as a way to understand the value of semantic layers. We’ll then query 20 million NYC taxi records with consistent business metrics executed using DuckDB and Ibis. By… ## [On building data-driven VC in Emerging Market firsthand](https://nonamevc.substack.com/p/on-building-data-driven-vc-in-emerging) _2025-08-13 · Daniel Chepenko · nonamevc_ When you’re managing investments across 12+ frontier markets, you don’t get the luxury of sending analysts to every city. You either build the data tools, or you fall behind. ## [My Journey from macOS to Arch Linux with Omarchy](https://www.ssp.sh/blog/macbook-to-arch-linux-omarchy/) _2025-08-07 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ I switched my five-year-old MacBook Pro M1 Max for a cheap (comparable) Lenovo ThinkBook 14 G7 ARP (AMD) laptop, running Linux (Arch btw, or better, Omarchy . And I am having a blast. But not everything is perfect. But let’s not get ahead of ourselves. This is a short recap after using it for one month on and off (due to repair 😅), and the last 2 weeks full time. I want to share what I… ## [Summer Data Engineering Roadmap](https://www.ssp.sh/blog/data-engineering-roadmap/) _2025-08-06 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ With this summer edition, you’ll have a roadmap for your vacation time to learn the basics of being a full-stack data engineer. Fill your knowledge gaps, refresh the basics, or learn with a curated list and path towards a full-time data engineer. After covering the essential toolkit in Part 1 (essential tools for your machine) and Part 2 (infrastructure and DevOps), this article teaches you… ## [Why Are We Here on Earth? True Happiness, Giving Up Control, or the Trap of Worshipping Earthly Things?](https://www.ssp.sh/blog/why-are-we-here-on-earth/) _2025-08-02 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ In the quest for happiness, it’s very important to know why we are on this earth, right? What’s our life’s purpose? What makes us truly happy, not superficially, but deep down? Is it having a good job, following a big career, having a family, buying a house — following the \[\[Default Path\]\]? Two years ago, when I started this article, I spent some quality time with my family, and… ## [The Data Engineering Toolkit: Infrastructure, DevOps, and Beyond](https://www.ssp.sh/blog/data-engineering-toolkit-devops-iac/) _2025-07-10 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Remember when data scientists spent 80% of their time wrestling with data wrangling instead of building models? I’d argue that today’s data engineers face similar challenges, but with the added complexity of infrastructure setup. We’re architects of entire data ecosystems, orchestrating everything from real-time pipelines to AI workflows. The secret? Infrastructure as Code and… ## [Has Self-Serve BI Finally Arrived Thanks to AI?](https://www.ssp.sh/blog/self-service-bi-ai/) _2025-07-09 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Two decades ago, we talked about self-serve business intelligence when we could filter dashboards compared to pixel-perfect reports. Today, twenty years later, does conversational BI through MCP and semantic context finally deliver on the true promise of self-serve by allowing us to prompt our way through data analysis? With the right setup, conversations can generate BI artifacts . When we chat… ## [Universal Data Orchestrator in Action: Enterprise Best Practices](https://www.ssp.sh/blog/enterprise-universal-data-orchestrator-in-action/) _2025-06-17 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Moving from orchestration theory to the enterprise level is a real challenge. How do you handle secrets across environments? Where does your business logic actually live? How do you make pipelines that work for both your senior engineers and the analysts who need to modify them? In Part 1, The Heartbeat of Data Engineering , we discussed the convergent orchestrator combining orchestration as code… ## [The Open Lakehouse Stack: DuckDB and the Rise of Table Formats](https://www.ssp.sh/blog/open-table-lakehouse-with-duckdb/) _2025-06-16 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Wouldn’t it be great to build a data warehouse on top of affordable storage and scattered files? SSDs and fast storage are expensive, but storing data in a data lake on S3 or R2 is significantly cheaper, allowing you to save a greater amount of essential data. However, the downside is that it quickly becomes messy or unorganized, lacking clear governance and rules. That’s where… ## [Self-Host & Tech Independence: The Joy of Building Your Own](https://www.ssp.sh/blog/self-host-self-independence/) _2025-06-07 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ After watching the two PewDiePie videos where he learned about installing Arch (something considered quite hard, even for Linux enthusiasts) and building three products (camera for the dog, weather/drinking/meditation device, and who knows what comes next) based on open-source, 3D-printed parts, I started wondering about building things yourself, self-hosting, and tech independence. Something dear… ## [From RPC to transactions and durable executions](https://www.pramodb.com/index.php/2025/05/21/from-rpc-to-transactions-and-durable-executions/) _2025-05-21 · pramod · Pramod Biligiri_ I spent some time reading about Durable Execution Engines (eg: Temporal) and explored possible connections to earlier concepts like database transactions, distributed transactions, and building RPC/Microservice based systems in a fault tolerant manner. In this post I ll try to summarize some of my learnings. How useful it is will depend on how much of this \[ \] ## [The Open Table Format Revolution: Why Hyperscalers Are Betting on Managed Iceberg](https://www.ssp.sh/blog/open-table-format-revolution/) _2025-05-20 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Wondering why open table formats are suddenly booming? Why is AWS investing heavily in making Iceberg tables on S3, and why did Databricks pay a reported $2B to acquire Tabular? The answers might change how we think about data architecture. Historically, object storage like Amazon S3 or R2 was used as inexpensive, scalable storage for unstructured files, while structured data typically went to… ## [Configure, Don't Code: How Declarative Data Stacks Enable Enterprise Scale](https://www.ssp.sh/blog/declarative-data-stack-enterprise/) _2025-05-20 · hello@sspaeti.com (Simon Späti) · Blogs on Data Engineering Blog & Second Brain_ Imagine building enterprise data infrastructure where you write 90% less code but deliver twice the value. This is the promise of declarative data stacks. The open and modern data stack freed us from vendor lock-in, allowing teams to select best-of-breed tools for ingestion, ETL, and orchestration. But this freedom comes at a cost: fragmented governance, security gaps, and potential technical debt…