Data Engineering Weekly #282

We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams operating under a single platform.

In this book, you’ll learn:

- How composable architectures allow teams to ship faster
- Why data quality matters and how you can catch issues before they reach users
- What observability means, and how it will help you solve problems more quickly

Download your copy now

We did this with data platforms. Open table formats gave us a standard wheel. Then every vendor added its own catalog, governance model, access layer, and control plane — the proprietary bolts that make the wheel difficult to move anywhere else.

Semantic layer conversation is everywhere. The author raises a real alarming question. Are we going to repeat the same mistake we did with the OpenTable formats?

https://y4nnick.substack.com/p/the-semantic-layer-is-not-a-layer

Traditional data engineering requires you to define your schema (ontology) before extracting data. The author argues this doesn't work for unfamiliar domains. Instead, you should use LLMs to explore unstructured data and extract candidate entities and relationships first. A human then reviews these to create a "seed ontology," allowing the schema to evolve organically from the actual data.

https://medium.com/@dhilip.kumar/from-unstructured-chaos-to-queryable-knowledge-designing-ontology-backed-knowledge-graphs-for-9df031d97257

StarTree recently announced full text search support on top of Iceberg. We started seeing the Lakehouse evolving from simple columnar storage to support variants, text search, and vector indexing. Oracle terms this a converged database, differentiating it from multi-model databases.

https://blogs.oracle.com/developers/what-is-a-converged-database-definition-five-tests-and-ai-use-cases

AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.

Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.

Download the free guide

Making a remote, distributed graph feel like an in-memory data structure requires very specific trade-offs focused on I/O management, concurrency control, and early filtering. The author writes about optimization around breadth-first traversal, an async-first concurrency model, push-down streaming filters, and volatility-aware caching.

https://netflixtechblog.com/how-and-why-netflix-built-a-real-time-distributed-graph-part-3-querying-the-graph-with-grpc-0f3468349607

You cannot evaluate a product's impact just by looking at aggregate metrics, because user behavior (who chooses to use the product) skews the results. Olx writes about a classic study showing that comparing different categories of ads is often misleading, and the importance of adopting the Propensity Score Matching (PSM) method to compare similar products' performance.

https://tech.olx.com/correlation-lied-to-us-rethinking-product-impact-with-causal-inference-5ba47181f7c5

The author writes a battle-tested "survival guide" for engineers who are pushing Flink to its absolute limits in production environments. Scaling Flink is incredibly difficult and often involves a frustrating cycle of solving one configuration problem only to have it trigger another. To save other advanced engineers from this trial-and-error headache, the author outlines 13 specific optimizations, undocumented fixes, and pitfalls to avoid.

https://mohamedelzein.hashnode.dev/how-i-managed-to-build-and-scale-flink-to-300-million-rpm-in-a-stateful-exactly-once-workflow-and-how-you-could-too

Netflix writes about its architectural journey of solving a massive data engineering problem: how to store cost-effectively and query petabytes of historical time-series data without sacrificing performance or adding operational complexity. It is an interesting case study to think about turning static disaster recovery backups into an active, low-latency cold storage tier—can yield massive efficiency and cost savings at scale.

https://netflixtechblog.medium.com/timeseries-tiered-storage-journey-kafka-flink-streams-to-native-cassandra-cold-reads-e59d597c9d60

Flipkart writes about automating the tedious and expensive process of evaluating search result relevance by replacing human annotators with a specialized LLM. The most interesting part for me is that the pipeline now acts as a diagnostic tool; because every judgment includes a reasoning trace, engineering and catalog teams can pinpoint exactly why a search result failed.

https://blog.flipkart.tech/llms-for-relevance-automating-high-quality-product-relevance-labeling-in-flipkart-search-ddd5ca50b584

One of the fundamental shifts with LLMs is that the expert operator model is vanishing. The go-to guy for xyz tool is changing, and systems thinking is making a more and more prominent impact in organizations.

The AWS Spark agent tool simplifies the expert operator model and democratizes building data pipelines.

https://aws.amazon.com/blogs/big-data/introducing-apache-spark-troubleshooting-agent-for-amazon-emr-on-eks/

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.