I Let Claude Code Build ELT Connectors. Here’s What Happened.

Development workflows have shifted fast these last few months.

Coding agents now sit between us and our tools—and that changes how we judge the ecosystem.

In data ingestion, my old heuristic was simple:

• Managed for standard sources (Fivetran, Airbyte)

• Python when you need flexibility (custom connectors)

Fivetran has an SDK, but it always felt too heavy—until agents entered the picture.

If an agent can absorb the SDK complexity, even proprietary interfaces become usable.

That collapses the old ELT spectrum: managed (Fivetran) vs code-driven (dlt).

So I tested it: can the same coding agent build custom connectors for both—using the Fivetran SDK on one side and dlt on the other?

Same API. Same LLM. Same goal.

Let’s see what happens.

AI copilots are a powerful tool for accelerating software development, and stand to commoditize (or even trivialize) data integration.

The goal of this post is simple: find the limits of AI-driven integration.

To test this properly, I built a virtual FastAPI server exposing three types of records:

• customers

• orders

• events

In the “realistic” version, the API looks like a typical demo:

• Auth: Bearer token

• Schema: consistent and complete

• Pagination: cursor (customers), offset (orders), Link header (events)

• Rate limiting: none

Pretty standard.

Then I made it deliberately painful:

• Broken records: null IDs, evolving schemas, random extra fields

• Type chaos: metadata as dict/str/list, amount as int/str/null

• Deep nesting: JSON 7 levels deep

• Auth upgrade: HMAC-SHA256 with X-Api-Key, X-Timestamp, X-Nonce, X-Signature

• Rate limiting: 5 requests / 10s, 429 + Retry-After

The FastAPI server exposes a complete OpenAPI spec—schemas and auth included—so an agent has, in theory, everything it needs.

To compare dlt and Fivetran, I ran both locally and loaded the results into DuckDB.

dlt is Python-native, so local runs are the default.

Fivetran’s Connector SDK can also run fully locally, which keeps the comparison fair.

For each tool, I launched an isolated Claude Code agent (Opus 4.6), gave it a minimal prompt, and let it build the connector.

I simulated incremental loads by adding new records after the first run, then rerunning the exact same pipeline.

At the end of each scenario, a separate agent compared the two DuckDB databases and flagged any differences.

Both agents did great: end-to-end in ~3–4 minutes.

The first run succeeded for both tools.

On the second run, both behaved as expected: they loaded only new incremental records and even detected a duplicate.

So far, so good—but a few type differences showed up:

• Dates: dlt uses TIMESTAMP, Fivetran stores ISO 8601 strings as VARCHAR

• Decimals: dlt uses DOUBLE, Fivetran uses DECIMAL

Fivetran treats the OpenAPI spec as the contract: if a field exists (even optional), it creates the column—whether or not the API ever returns it.

dlt is schema-on-observation: it creates columns only for fields it actually sees in the payloads.

In our case, a few fields were marked optional in the spec but never appeared in actual responses.

Fivetran still created the corresponding columns (all NULL/empty), while dlt never created them because it never observed those fields in the data.

Nothing dramatic—just different philosophies.

A solid warm-up: easy for both tools.

Now let’s go up a level.

Hard mode introduced some real friction: NULL primary keys, deeply nested structures (7 levels), mixed data types, and HMAC authentication—across the same three endpoints (customers, orders, events), each serving 1,000 records.

Both agents handled HMAC auth and rate limiting easily.

The outcome:

• dlt: 3,000 / 3,000 records loaded (100%). Created 5 tables, including child tables for nested arrays.

• Fivetran SDK: 2,815 / 3,000 records loaded (93.8%). 185 records dropped—all with NULL primary keys.

Here’s what happened.

dlt failed once the broken records appeared. A schema inferred from the clean run broke when corrupted variants showed up—specifically metadata changing types.

The agent had to patch the pipeline by:

• adding variant handling for metadata

• generating surrogate keys for NULL IDs

Fivetran ran end-to-end without code changes. Its “serialize to VARCHAR/JSON” approach avoided type conflicts, and its strict primary-key requirement filtered out invalid records.

The trade-off: 185 records never landed.

Orders returned metadata as a dict, a string, or an integer.

• dlt inferred a STRUCT from the first batch (all dicts). When strings/integers appeared later, the pipeline broke. After the agent fixed it, metadata was split into four columns: metadata__source, metadata__campaign, metadata__value (for non-dict values), and metadata__type (to track the variant).

• Fivetran serialized metadata into a single VARCHAR JSON column, so it never broke—but you lose the typed structure.

dlt flattened nested arrays into child tables—orders__line_items (2,534 rows) and events__tags (516 rows)—linked back to the parent via foreign keys.

Fivetran kept them inline by serializing the arrays as JSON strings in VARCHAR columns (_line_items_json, tags).

Both work, but the ergonomics differ: dlt is join-ready; Fivetran requires JSON parsing at query time (or an extra staging model).

Most of these edge cases could probably be handled with better prompting—and a bit more upfront guidance.

What’s interesting is how far you can get with a one-shot prompt, and how each framework’s defaults reveal its underlying philosophy.

In both cases, the agent navigated the tools smoothly and set up ingestion quickly. That’s what makes these tools compelling: they’re solid building blocks an agent can adapt for last-mile customization.

One more takeaway: maintenance. Agents aren’t just useful for bootstrapping pipelines—they can also keep them running. In this test, the agent diagnosed why the dlt pipeline failed after the first run and patched it into a more resilient shape.

ELT decisions traditionally come down to four things:

1. Cost model — MAR vs compute

2. Engineering time — setup + maintenance

3. Connector catalog — what you get out of the box

4. Custom connectors — how painful it is to build what’s missing

AI compresses #4 and a big chunk of #2.

Both dlt and the Fivetran SDK can get to a working custom connector fast, even against awkward APIs.

Once “can we build the connector?” stops being the differentiator, what matters most is the operating model:

• Fivetran SDK: your connector runs on Fivetran’s platform. You inherit hosting, scaling, scheduling, monitoring, retries, security, and general operational burden.

• dlt: you get full Python flexibility, but you own the whole stack: runtime, orchestration, scaling, observability, and on-call.

So the decision becomes a strategy choice.

For smaller teams (say, <400 people) with limited ops bandwidth, no desire to own ingestion, and highly commodified data models, Fivetran is a sensible default.

If you need maximum flexibility and control, and you’re willing to invest in running and maintaining ingestion, dlt is the better fit.

Thanks for reading,

Ju