Data Engineer Things Newsletter - Data Pulse Edition (Apr 2026)

Hello Folks,

Great to connect with you through this month’s newsletter.

I’m writing this from Coimbatore, India where summer is in full swing bright days, a bit of heat, and that constant push to slow down and pace yourself.

I’m Sri. I work in the data engineering and AI space, focusing on building scalable data platforms and Enterprise AI solutions enabling teams to make better use of data in real-world scenarios. Over time, I’ve come to appreciate that this field is more about how thoughtfully we design systems that people can rely on.

Outside of work, I enjoy reading, traveling, and watching movies small ways to recharge and stay curious. In many ways, this community feels similar: a space where we can learn from each other, share ideas, and keep growing together.

This edition brings together a set of interesting ideas and perspectives from across the data engineering world. I hope you find something here that resonates or sparks a new thought.

Happy reading, and thank you for being part of this journey.

– Sri

📖 Topic: Data & Context Engineering
🧠 Level: Intermediate

Summary: Modern data platforms are moving from managing raw data to managing context. The article introduces “context engineering” as a structured way to capture, enrich, and operationalize metadata so both humans and AI systems can understand data better. It focuses on building context layers like lineage, ownership, usage patterns, and semantics to improve discoverability, trust, and usability, especially in AI-driven workflows that need richer signals for accurate reasoning.

💡 Why is this relevant for DEs?

• AI-ready data platforms: As AI pipelines grow, systems need context along with raw data to interpret it correctly. Context engineering enables AI-native data platforms.

• Metadata as a core building block: Metadata like lineage, ownership, and quality signals becomes a first-class part of the system, not an afterthought.

• Improved data trust and discovery: Adding operational and semantic context helps users quickly understand what data means, where it comes from, and how reliable it is.

• Beyond static catalogs: Instead of passive catalogs, this approach enables dynamic, continuously evolving context that reflects real usage and system behavior.

📖 Topic: Data Engineering
🧠 Level: Intermediate

Summary: Spotify explains the engineering behind its 2025 Wrapped “Archive,” which goes beyond static summaries by turning listening behavior into narratives. It identifies meaningful moments like taste shifts and binge sessions and converts them into personalized stories. This required large-scale pipelines to precompute billions of user-specific insights, combined with AI-generated narratives to deliver a near real-time experience to hundreds of millions of users.

💡 Why is this relevant for DEs?

• From metrics to moments: Traditional analytics focuses on aggregates like counts and top items. This approach turns raw data into meaningful events and storytelling units.

• AI + data engineering convergence: It combines data pipelines with LLM-based narrative generation, showing how data systems are evolving for AI-driven experiences.

• Product thinking for data: Engineers move beyond pipelines to designing outputs users directly interact with, making data work more product-oriented.

• Serving at global scale: Delivering a consistent experience to hundreds of millions of users requires tight coordination across processing, storage, and serving layers.

📖 Topic: Databases
🧠 Level: Intermediate

Summary: ByteByteGo describes Nextdoor’s database evolution as incremental scaling steps. It starts with a single Postgres instance and grows by adding connection pooling, read replicas, caching, and reconciliation as needed. Each change fixes a bottleneck but adds complexity such as replication lag and consistency challenges. The key takeaway is that scaling is continuous and every improvement involves trade-offs.

💡 Why is this relevant for DEs?

• Scaling is incremental: Systems evolve step by step as scale increases, adding new layers as bottlenecks emerge.

• Read and write separation is essential: As traffic grows, reads often become the bottleneck, requiring a primary-replica model to split read and write workloads.

• Operational complexity grows with scale: Pooling, replication, and caching add complexity, making observability, alerting, and incident management critical for debugging and reliability.

• Pragmatic architecture decisions: Each change is driven by a specific need, favoring practical, context-driven engineering over “perfect” designs.

📖 Topic: Data Infrastructure and AI
🧠 Level: Advanced

Summary: Uber explains how it scaled its deep learning systems to support increasingly complex AI use cases. As it moved from traditional ML to deep learning and then generative AI, compute and workflow demands increased significantly. This led to investments in GPU efficiency, distributed training, and a streamlined ML lifecycle. The focus shifted toward reducing training time, improving efficiency, and standardizing processes so teams could reuse workflows instead of rebuilding them.

💡 Why is this relevant for DEs?

• Data infrastructure fuels AI: Deep learning performance depends heavily on the underlying data pipelines, making scalable, high-throughput infrastructure essential for training and inference.

• Compute-aware data design: With GPUs and distributed training, data engineering choices directly affect cost and speed, making formats, storage, and access patterns critical.

• End-to-end pipeline thinking: Data engineering extends beyond ingestion to include feature pipelines, training data preparation, and real-time serving within unified ML systems.

• Efficiency at scale: Small inefficiencies become expensive at scale, reinforcing the need to optimize data movement, reduce redundancy, and align pipelines with compute resources.

📖 Topic: Data Engineering & Leadership
🧠 Level: Intermediate

Summary: This article explores the transition from hands-on data engineering to people management. It highlights that skills like solving technical problems and building pipelines do not directly translate to leadership. The focus shifts to enabling others, setting direction, and creating an environment where teams can work effectively.

💡 Why is this relevant for DEs?

• New skill set required: Technical depth alone is not enough. Communication, decision-making, and empathy become essential for managing teams and aligning stakeholders.

• Shift from systems to people: Career growth in data engineering moves from building solutions to enabling and guiding others to build them.

• Leadership before title: Effective leaders often demonstrate leadership early by mentoring, unblocking teammates, and influencing decisions before moving into formal management.

Block the calendar for April 29: OpenXData is a free virtual conference packed with two tracks of expert-led sessions on open data architectures, AI-native platforms, and cost and performance at scale.

The lineup includes a keynote by Vinoth Chandar (Onehouse) on data platforms for autonomous AI, Maxime Beauchemin (Preset) breaking down the anatomy of their data agent, and engineers from Uber, Walmart, Booking.com, JD.com, Zalando, Anthropic, and Snowflake sharing how they’re running open lakehouse stacks in production. If you care about Iceberg, Hudi, Spark, Polaris, or where data engineering meets AI, this one’s worth your time.

📋 Agenda and more details HERE.

👉 RSVP

DuckLake v1.0: The Lakehouse Format Built on SQL Reaches Production-Readiness

DuckLake is a lakehouse format from the DuckDB team that simplifies metadata by moving it from scattered JSON and Avro files in object storage into a SQL database. Data still lives in Parquet files on object storage, similar to Iceberg or Delta Lake, but the catalog is stored in any transactional SQL database with primary key support.

👨‍💻 Quick taste

📰 What/s new in v1.0

Released April 13, it is now production-ready with backward compatibility guarantees. Key updates include default data inlining for updates and deletes to avoid small-file issues in streaming writes, support for sorted tables and murmur3-based bucket partitioning for Iceberg compatibility, native GEOMETRY and VARIANT types, and deletion vectors stored as Puffin files.

💡 Why is this useful for DEs?

• One less moving piece. Your catalog is just a db you already know how to run.

• Streaming into a lakehouse actually works. Inlining means you can do small, frequent writes without drowning in compaction jobs.

• Transactions are real. ACID guarantees come from the catalog DB.

• Escape hatch included. Iceberg-compatible at the data level, so no lock-in.

👉 GitHub: https://github.com/duckdb/ducklake

Stop Over-Partitioning Your Iceberg Tables

Partitioning can feel like free performance until it backfires. A common mistake is using a high-cardinality column like user_id or raw timestamps in an Iceberg table, which creates thousands of tiny partitions with only a few rows each. This shifts the cost to metadata, where query planning takes longer than the actual scan and effectively recreates the small-files problem through poor partition design.

📒 Rules of thumb

• Aim for partitions that hold at least ~1 GB of data. If most of yours are smaller, you’re over-partitioned.

• Partition on low-to-medium cardinality columns that actually appear in WHERE clauses, usually dates at day or month granularity, maybe region or tenant.

• Use Iceberg’s hidden partitioning transforms, not raw columns. PARTITIONED BY (days(ts)) beats partitioning on a raw timestamp, otherwise you’d get a new partition per millisecond. Transforms bucket values sensibly and Iceberg auto-prunes queries without extra derived columns.

• For high-cardinality columns you filter on, use bucket(N, col) instead. Hashes the value into N buckets, pruning without the cardinality explosion.

Quick gut check: SELECT COUNT(*), AVG(file_size_in_bytes) FROM my_table.files 👉 if average file size is in the MB range, rethink the partition spec.

📖 Iceberg Partitioning docs

Until next time, cheers!

Sri, Ananda, & Volker

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.