Reminder: I’m offering a limited-time 50% discount on the annual plan:
Once you claim it, the discount will be applied forever.
Now, with only $5/month, you will have access to:
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues
Over the last 3–4 years, several key terms have gained popularity.
Some I’ve had the chance to learn and apply; others still leave me skeptical.
One of them is the “data contract”.
In this article, I want to look into this concept to understand what it is, its motivation, and how it would be applied.
Note that I don’t have any prior experience with data contracts, so I’m approaching the topic with a bit of skepticism, based purely on my experience and observation.
So, enjoy :d
I found the term “data contract” the same way you guys probably do, scrolling LinkedIn and seeing it stands next to words like “data mesh” or “data product”.
“Is it a thing? Is it legit? “
—
Software engineers have had API contracts for a long time.
It’s an agreement that hides complex, internal things and only exposes what the API consumer needs.
By doing it this way, the producer can change things behind the scenes without any worry about breaking the interface for anyone who depends on them.
Sound very like the object oriented huh?
—
Andrew Jones is widely recognized for introducing the “data contract” in 2021, while building the data platform at GoCardless, treating data as a product with explicit interfaces to manage data quality, ownership, and governance.
His point is that almost every data platform begins with pulling data out of an organization's OLTP databases, where the data was designed for the company applications. This means we (usually data engineers) have to put a lot of effort into adapting it into something useful for analytical reporting or modeling.
Disasters happen when changes are made to the source without the consuming team being fully aware. Breaking changes and data migrations are part of any backend application lifecycle.
Because the data is organized to serve the applications, the engineers responsible for making these changes may not understand how they affect the way the data is used in the OLAP realm.
He proposes to stop exposing the internal shape of the data inside the source system. Instead, design a separate, deliberate structure for what we hand downstream, one that stays stable even when the source changes underneath it.
So when something changes in the source, we don’t just expose the change as-is; we produce an interface designed for consumption, decoupled from how that data is represented internally.
Then treat that downstream-facing structure with the same discipline software engineers already apply to customer-facing APIs: document it, version it (and try not to break it)
Set aside the context; what is a data contract, at the most basic level?
A data contract is an agreement between data producers and data consumers.
It has details that link the business side (the logical meaning of the data) with the technical side (how it's physically implemented), plus extra metadata like data quality rules, SLAs, and expected behavior.
—
Instead of a contract between two services, a data contract sits between whoever generates the data and whoever consumes it.
It’s a promise about well-modeled, high-quality, trusted data, and the producers try hard to provide these properties.
—
Effective data contracts typically include four things:
Schema: field names, data types, and relationships between elements)
Ownership: which team or individual is responsible for maintaining it. This information helps streamline communication when something breaks or needs to be discussed. (or someone to blame)
Service level objectives: agreed-upon metrics like update frequency, acceptable error rates, and response times. This sets expectations for how data should be delivered.
Version information: how the contract has changed over time
It’s essentially a digital record — it could be a JSON or YAML file.
—
It's just a digital record to … record the promise; so, how could it be enforced?
Reminder: I’m offering a limited-time 50% discount on the annual plan:
Once you claim it, the discount will be applied forever.
Now, with only $5/month, you will have access to:
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues
The data contract itself doesn't enforce anything.
It's just the source of truth.
The enforcement happens in the actions around it.
It’s mostly found in:
CI checks that validate the contract before deployment: contract violations block the deployment.
Production validation that runs against the live dataset: confirming schema, quality, and SLA commitments are genuinely being met once data is actually flowing (what pre-deployment checks missed)
Consumer-side checks: critical consumers run their own validation on top of everything, confirming what they’re actually receiving matches what the contract promised.
Essentially, it’s schema checking, plus other promises. Remember from the previous section: a contract isn’t only a schema; it also carries ownership and service level objectives, like update frequency and acceptable error rates.
So enforcement has to check more than “are the field names and types right.” It has to check whether the data is actually showing up on time.
As discussed, a contract is just a record.
However, it must follow some structure to allow the validation engine to access it, and more importantly, I think it should have a standard to standardize the implementation.
When searching online, the dominant one I found right now is the Open Data Contract Standard (ODCS), governed by Bitol, a Linux Foundation project. ODCS leverages YAML as its format, which makes contracts easy to version and manage. So the contract itself, the actual document with the ID, status, description, schema, and everything else, is written as a .yaml file. It’s supported by many vendors.
That YAML file is then checked for validity using a separate JSON Schema that Bitol publishes alongside the standard: a JSON Schema for ODCS is provided specifically so it can be imported into an IDE or a validator to check whether a given YAML file is a valid contract (based on ODCS).
The YAML gets checked against the JSON Schema, which defines every required and optional field, its expected type, and the rules attached to it.
In the ODCS YAML file, there are:
You can view the ODCS YAML example here.
Identity fields:
a unique ID (to avoid naming collisions)
a name
a version number
a status: “proposed,” “draft,” “active,” “deprecated,” or “retired,” so a contract’s lifecycle stage is explicit.
Description fields
purpose: the intended use for the data
limitations: technical, compliance, or legal restrictions on how it can be used
usage: the recommended way to actually consume it
Dataset-related fields:
datasets and schema: the data’s shape
data quality: rules like completeness or accuracy checks on individual columns
stakeholders and roles: who’s accountable, and for what part
service-level agreements: what response time or freshness is promised
As discussed, a piece of YAML file doesn’t enforce anything. There must be a validation engine.
An ideal engine must validate those things:
The contract’s format: is in the correct format?
Schema validation: does the data still match the structure the contract promised? Field names and types are correct; nothing is renamed or dropped underneath the consumer without warning. There must be a central place to store and version the tables’ schema here.
Data quality: does the data “behave” the way it’s supposed to? This covers things like completeness, validity (does it fall within expected values), uniqueness (no unintended duplicates), volume (roughly the expected amount arrived), or freshness (it showed up when it was supposed to).
However, there are some gaps I believe the programmatic engine cannot validate. It is the dataset’s purpose, usage, and limitations. They are free-text description fields, written for a human to read. These are more like guidelines than measurable rules.
—
Based on my imagination, the validation engine must be observed as two parts: the rule engine and the compute engine.
Contract’s format and schema validation is lightweight, as it only touches a small amount of metadata.
However, data quality is a different story.
Although some rules can be checked using metadata (e.g., updated time to check freshness), most data rules require data scanning; compute engine resources must be a concern here. Ideally, this engine is also the one your company uses for data transformation to avoid syntax mismatch.
However, this does not rule out using different compute engines for different contract validation use cases. For example, near-real-time validation could use Spark Streaming or Flink.
—
Also, the validation engine should be flexible enough to allow the user to add or adjust rules. This is where the rule engine comes into play; it stores all the rules somewhere and exposes a method to let the user interact with the rule.
The crucial thing to note is that there is no single tool called the “validation engine“, it must be a solution where the technical component depends on the organization's requirements and context.
It could be Great Expectations, dbt test, or a self-development solution for the engine rule.
It could be Databricks, BigQuery, Snowflake, Trino, Spark, Flink, or DuckDB for the compute engine.
—
Although the idea of a data contract is attractive, implementing it is not easy. The contract recording is just the tip of the iceberg. You also need to build the validation engine for it, which requires as much effort as the main data pipelines.
However, the obstacle of data contracts does not stop at the technical side.
Data contracts are not a familiar concept like “data warehouse“, “ETL“, “data lake“ or “data quality“.
Not to mention that applying it requires time and resource investment.
Also, it can easily become something people feel like they have to do rather than something they want to do for the benefits it brings.
The tools can’t fix this; it requires mindset shifting.
How do you convince the CEO to invest time and resources in a data contract?
How do you convince the data producer to follow through on the “promises”?
How do you convince consumers to validate the dataset based on the contract?
These actions give people more work to do and add more time to the overall raw-to-insight process. That takes a lot of education and alignment, which involves many people. And whenever something involves a lot of people, it’s always a hard mission.
—
I’m writing this section to identify the culture obstacle, not to solve it. It’s more about bringing awareness. That said, I’d guess one of the first steps is starting with the organization’s current, specific problem, and elaborating on how a data contract would actually help with that.
An MVP built around a data contract can help catch issues that would have already happened before they reach production. That kind of concrete.
In this article, I tried to learn and research data contracts, a fairly new concept in data engineering. We started with its context, what it is, and how to implement it with the contract encoding and the validation engine. Finally, I laid out some thoughts about the obstacles to data contract implementation; besides the technical, mindset shifting is also a big one.
Thank you for reading this far; see you in my next article.
[1] Andrew Jones, Data Contracts (2021)
[2] Andrew Jones, A Guide to Data Contracts with Andrew Jones (2025)
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.