How Data Engineering Changes with AI

AI coding shifted around the turn of the year. AI went from fancy autocomplete to autonomous coder. The models got better, but more importantly, we figured out how to build coding agents that use harnesses to iteratively work toward solutions that actually run.

We’ve seen the same shift in data engineering. Users went from “Can you help me write this CASE WHEN statement?” to “Build me a pipeline that does XYZ.” What made this possible was using DataSQRL as a coding harness. That was a big aha moment for us.

Let me back up. We started DataSQRL to democratize data engineering: to let developers build complex pipelines that combine streaming and batch, spanning ingest, processing, storage, and serving layers, without needing a team of specialists. The framework made things easier and eliminated a ton of low-level plumbing. But data engineers still had to learn a new framework. That’s a real obstacle.

AI changed that picture completely. When AI writes the code using DataSQRL as a harness, users get over the learning curve quickly and produce end-to-end pipelines with speed and confidence. Seeing those results, we pivoted. DataSQRL is now an agentic data engineering harness focused on providing the guardrails and feedback loops that AI agents need to produce reliable data pipelines.

Over the last six months, we’ve watched this play out with users. It has fundamentally changed how we think about data engineering workflows. Here’s what we’ve learned.

In traditional data engineering, a huge chunk of time goes into translating business requirements into working pipelines that meet all the non-functional requirements: data quality, consistency, compliance, scalability, reliability. We’re seeing that effort move into the harness rather than being a project-by-project endeavor. That has two important implications.

First, you want your harness to encode the non-functional requirements that are general to data engineering and specific to your organization, like the regulatory or compliance landscape you operate in. Traditionally, organizations transferred this knowledge through training and then provided oversight and auditing to ensure proper implementation. With the harness, that process can be automated. But it requires upfront investment and clear ownership of how harnesses get built and maintained.

Second, the harness becomes the framework and guardrail for all data engineering initiatives inside your organization. That means change management, adoption planning, and helping users understand what the harness does and how to handle any gaps they discover. This is critical to unlocking large productivity gains. Without properly rolling out a shared data engineering harness, your teams will get stuck with small productivity gains from incremental tool adjustments.

DataSQRL’s compiler and testing framework provided some foundational components for this: the guardrails and feedback loops that coding agents need to iteratively produce complete pipelines. We’ve since extended DataSQRL to support deep introspection of the data flow DAG and SQL code, allowing organizations to customize their harness further.

Building data pipelines starts with understanding what data you need and how to access it. Traditionally, teams have relied on data catalogs for discovery. Catalogs work well for humans: they have familiar interfaces and search tools. But they’re hard to access for coding agents and often produce poor results. There are frequently discrepancies between what the catalog says and what’s actually true. And the catalog sources aren’t readily consumable by code.

Rather than building an access layer to make existing catalogs usable by agents, we stepped back and asked: what does the optimal data catalog look like from an agent’s perspective?

The answer: store the catalog in code. Coding agents already know how to navigate complex repositories. By making the data catalog a standard Git repo, we tap into workflows that both agents and humans understand — for managing data source evolution, versioning, and governance over time.

This “code is truth” approach has another benefit: a coding agent can include the data catalog as a Git submodule directly in its pipeline and manage evolution effectively. I’ll dig into this approach in a separate post, but it’s a foundational element to rethink. Most data engineering activities start with the data catalog.

When coding agents can discover source data and build high-quality pipelines that meet both functional and non-functional requirements, it dramatically changes what data engineers actually do. Their focus shifts to two things: requirements engineering (giving the agent the context it needs) and validation of the results.

In practice, we see most time spent on the latter. Engineers validate that pipelines meet requirements and understand what’s being produced. We believe it’s critical for humans to stay in the loop: to catch suboptimal results so that upstream agents and harnesses can be fixed, and to avoid surprises in production.

But here’s the rub: when you radically speed up pipeline creation, human validation becomes the bottleneck. Having engineers pore over thousands of lines of agent-generated code leads to fatigue and poor results. So a key goal has been to make validation easy. Not just by producing concise code, but by providing auxiliary artifacts that help humans quickly assess data flows, integrations, and pipeline behavior.

DataSQRL approaches this by using SQL as the primary implementation language. SQL is declarative, easy for most data engineers to read, and concise in how it represents data flow and transformations.

We’ve also added side outputs from the DataSQRL compiler: visualizations of the data flow DAG and artifacts for deep analysis of computations. This gives engineers multiple views of the pipeline to assess quality, reliability, and whether it meets requirements.

The shift to agentic AI is reshaping data engineering. The harness becomes the central investment: encoding best practices, compliance requirements, and organizational standards so that agents can produce reliable pipelines consistently. Data catalogs need to be rethought for agent consumption, and “code is truth” offers a compelling path forward. And human validation becomes the critical bottleneck to address through better tooling and multiple views into pipeline behavior.

We’re still early in this transition. But the patterns are clear: invest in your harness, rethink your catalog, and focus your engineers on validation rather than implementation. That’s the new workflow.

If you want to try this approach, check out the DataSQRL GitHub repo and our agentic data engineering harness documentation.