The Context Gap: Why Data Products Anchor AI Success

Srinivasa Mathkur is a technology leader specialising in enterprise data engineering, data platforms, and AI-ready architectures. As a Principal Data Engineer at The Modern Data Company, he helps organisations build governed, scalable data foundations that accelerate AI adoption and business outcomes.

With over two decades of experience spanning financial services, retail, healthcare, and enterprise technology, Srinivasa has led large-scale initiatives in cloud modernisation, data governance, streaming, data products, and platform engineering.

A passionate advocate for automation-first engineering and modern data practices, he frequently writes about data products, enterprise AI, and the architectural foundations required to make AI trustworthy, efficient, and production-ready. We’re thrilled to feature his insights on Modern Data 101.

We actively collaborate with data experts to bring the best resources to a 20,000+ strong community of data leaders and practitioners. If you have something to share, reach out!
🫴🏻 Share your ideas and work: community@moderndata101.com

*Note: Opinions expressed in contributions are not our own and are only curated by us for broader access and discussion. All submissions are vetted for quality & relevance. We keep it information-first and do not support any promotions, paid or otherwise!

The moment when a model answers you, a model has a set of weights trained on a static snapshot of text, plus whatever you put in the prompt. Nothing else. No access to your systems or any sense of what changed yesterday, and no idea which of your fields are trustworthy and which are stale.

That fact explains most enterprise AI failure. The model isn’t weak. “Knowing a lot in general” and “knowing enough about this specific situation, right now” are different properties, and only the second one is what a real business decision needs.

A number in a column (net_revenue = 48213.55) is data and cannot be used as information without added context. Information requires a frame: what period, under what accounting rule, before or after returns, in which currency, from which system.

Two analysts can look at the same number and reach opposite conclusions if only one knows the frame. This is why data dictionaries and metadata catalogs have existed for decades.

Interestingly, an analyst can go ask someone while a model can’t, but it supplies a frame regardless, possibly the wrong one, with total fluency.

Say the model does know the frame. Is that enough? Not necessarily. Context sits above information: what’s normal right now, what’s changed recently, what this specific data point is unreliable about.

Two ER physicians can read the same borderline lab result with the same training. One has noticed a local cluster of atypical presentations the standard risk score misses, and knows the hospital’s assay is less precise near this threshold. She admits the patient. The other follows protocol and discharges him. Both used correct information. Only one used context.

This is the gap driving AI hallucination in production. Estimates put the direct cost to businesses in the tens of billions annually.

Industry estimates put the global cost of AI hallucinations at $67.4 billion in 2024. A Stanford HAI and RegLab study, peer-reviewed in the Journal of Empirical Legal Studies, found that legal AI tools built for a single domain still hallucinate between 17% and 34% of the time on challenging queries. Documented court cases involving AI-generated errors grew from around 10 in 2023 to 37 in 2024, and 73 from the first five months of 2025.
~ Source: Why AI gets it wrong and how data products fix that

Purpose-built legal AI tools still produce unsupported answers on a meaningful share of hard queries, even against peer-reviewed benchmarks. Medical AI models, even after mitigation prompting, still get a real share of answers wrong. None of that is a capability gap, but rather a context gap.

The instinctive fix of giving the model more information doesn’t scale, for two reasons.

Without a pre-built semantic layer, an agent reconstructs understanding at query time: exploratory queries, schema guessing, relationship inference, on every call. Every hop adds tokens, latency, and a new failure point.

Organisations that pre-define semantic context instead of discovering it live have cut input token consumption by up to 90%.

At production scale, that’s the line between an economically viable AI program and one that isn’t.

Researchers found accuracy can drop significantly simply because the relevant fact sat in the middle of a long context, buried in irrelevant material. Flooding a model with unfiltered data doesn’t average out to better-informed. Instead, it drowns the signal in noise the model can’t tell from signal. It still answers fluently, just incorrectly.

“Add more data” is not a bigger version of “add the right context.” It’s a different, worse strategy.

The resolving concept comes from domain-driven software design: bounded context. It’s a scope within which a term has one agreed meaning.

“Revenue” in billing and “revenue” in sales aren’t the same construct; treating them as interchangeable is where errors show up. Inside its boundary, though, the term is unambiguous.

Applied to AI: the fix for hallucination is a curated, governed, semantically consistent slice of data, scoped tightly to the problem at hand and no wider.

The ER physician isn’t more informed than a colleague who’s read every textbook. She’s differently informed, narrower but precisely calibrated to this hospital, this population, and this month. That precision is the entire source of her value. The target is to achieve bounded information instead of exhaustive information.

A bounded slice needs structure to hold up under use.

• Semantics: Not just that a column is called net_revenue, but the rule behind it: which adjustments apply, how returns are handled, what calendar it runs on. Operates at the field and metric level.

• Governance: Who’s accountable, who can access it, what regulatory constraints apply. In a regulated industry, this is the line between a defensible decision and a liability.

• Data quality: A field that’s 40% null or drifts in freshness can’t carry the same weight as one that passes rigorous checks. Quality metadata tells an agent how much to lean on a given field.

• Provenance and lineage: Provenance is origin. Where data came from, who collected it, under what conditions. While lineage is transformation: which jobs touched it, through which systems, when. Together they make an answer auditable.

  

• Domain taxonomy: Where semantics defines terms, taxonomy defines relationships. How a SKU rolls into a product line, a product line into a category. Without it, a model pattern-matches within one entity instead of generalising across them.

• Operational metadata: Descriptions, tags, usage signals, certification flags, SLA indicators. What makes the rest findable and consumable.

None of these six pillars is new. Metadata catalogs and MDM platforms have chased pieces of this for decades.

What’s different is packaging all six into a single, versioned unit with a defined interface, built to be consumed directly by an agent as well as browsed by an analyst.

That packaged unit is a data product: the buildable answer to the abstract idea of bounded context.

A demand-forecasting agent in retail has to distinguish shipment volume from sellout volume. These are constructs that look interchangeable but aren’t. Miss it, and the forecast is numerically clean and directionally wrong.

A clinical agent needs to know which fields carry consent restrictions, which codes have institution-specific overlays, which values were manually entered versus system-generated. Skip that, and the output looks certified but isn’t.

A risk agent in financial services needs verified provenance on counterparty data and a clear mapping between internal and external classification schemes. Without it, the output is fluent and wrong.

A maintenance agent watching equipment health needs calibration history and a sense of how shift patterns move the anomaly baseline. A raw sensor feed answers none of that.

In every case, the model isn’t underpowered but ungrounded.

Once the slice is built properly, the expensive work (freshness, null rates, schema drift, meaning) happens once, at build time, not per query, per agent, forever. Every agent drawing on a shared data product skips re-deriving what a prior agent already worked out.

A well-scoped data product is reusable, too: the same governed slice can feed a forecasting agent, a BI dashboard, and a compliance check at once.

Build cost amortises across every consumer, and every refinement made for one use case sharpens the context available to the rest: a fundamentally different cost curve than each agent rebuilding its own understanding from scratch.

Having a boundary isn’t the same as having the correct one. A data product encoding the wrong semantics doesn’t reduce hallucination, but industrialises it, wrapped in the false confidence of certified metadata.

Data products have to be built right-to-left: start from the business problem, work backwards through the data’s reality, and validate the semantic model before it ships to an agent, not after.

The boundary isn’t static, either. What made the ER physician valuable wasn’t generic experience, but the current experience, tied to the hospital’s specific quirks. A data product accurate six months ago, built on a source system that’s since changed schema underneath it, isn’t just stale, but dangerous, because it still answers with the same confidence it had when it was right.

One place the analogy breaks in the data product’s favour:
When the physician retires, her contextual understanding leaves with her. Institutional knowledge has always been fragile: carried in people, lost in turnover.

A data product doesn’t retire. Each governance update and quality certification layers into the product itself, not into someone’s memory.

Done well, the context compounds instead of resetting every time someone leaves.

What does a model have when it answers you? A static training snapshot and whatever’s in the prompt. This is the case for changing what’s in that second part deliberately, rather than just making it bigger.

Before an organisation argues over which model to deploy or which orchestration layer to buy, there are prior questions:

• Do we have data products that make those downstream choices work at all?

• Have we built the bounded contexts our agents need to be competent, not merely confident?

Models will keep getting better at reasoning over whatever they’re given. Whether what they’re given is trustworthy, scoped, and current, that’s still a data architecture problem.

If you have any queries about the piece, feel free to connect with the author(s). Or connect with the MD101 team directly at community@moderndata101.com 🧡

Got questions? Find Srinivasa on LinkedIn or drop a comment below. 💬

From the Modern Data 101 Team ❤

State of Data Products is a quarterly read for senior data and AI decision-makers that reports the industry reality of enterprise data and AI. This edition covers ontology as AI infrastructure, the agentic AI deployment gap, observability blind spots, and why lean AI is the economic reality check the industry needed.

Get your copy