Nobody Owns the Duplicates

The catalog I am working on right now holds a couple thousand products. A meaningful share of them are the same product listed more than once.

Not similar. The same. One seller uploads it with the manufacturer code in the title. A second one puts the code in a custom field and writes the description by hand. A third pastes the manufacturer’s marketing copy and appends a shipping note. This is a marketplace, so all three listings are legitimate, all three sellers are right, and the customer searching for that item gets three results, three prices, and a quiet doubt about whether this site knows what it sells.

Then there is the other half of the problem: products sitting in the catalog that nobody can actually deliver.

When I finally got the business intent stated plainly, it was two sentences:

1. Show customers what they can actually buy.

2. Stop showing them the same thing three times.

Everything else on that project is an implementation detail of those two sentences. Getting them written took several conversations. That delay is the point of this issue.

There is a standard for this. GS1 publishes the GTIN Management Standard, which defines when a change to a product requires a new identifier: new formulation, new functionality, a change the buyer is expected to notice. It is precise, public, and free to read.

It also does not survive contact with a few thousand sellers. Some assign a new code per colour. Some reuse one code across a range. Some have no code at all and upload whatever their ERP happens to contain. Marketplace rules on top of that vary by category and get reinterpreted twice a year.

So you cannot normalize a large catalog by implementing a specification. There is no specification you can trust end to end. You can only implement what this business means by “the same product,” and that meaning lives in people’s heads.

The question that stalled us was not technical. It was: whose job is this, engineering or sales?

Compressed, the two positions sound like this.

Engineering: “Give us the rule and we will apply it to the whole catalog tonight.”

Sales: “Show me any two listings and I will tell you in two seconds whether they are the same product. I cannot give you the rule.”

That exchange is tidier than the real one, which took two meetings and a spreadsheet. But both sides are stating something true. Sales holds a judgment that has never been written down. Engineering holds the only mechanism that can operate at this volume. Neither can finish the job alone, and either one, left to do it alone, does damage: engineering merges products that customers will receive as the wrong item, sales opens a queue of 300,000 rows and closes it after forty.

The answer is a mix. The useful part is the shape of the mix.

Three mechanisms, in this order:

Engineering proposes mass fixes. It does not apply them. The technical team produces batches: here is a group of listings we believe are one product, here is the evidence, approve, reject, or split. Sales decides at the level of the batch, not the row. That is the only ratio that works, because judgment is expensive and volume is cheap.

An exception interface, owned by the business. Every rule leaves a residue. The residue is not a bug report and it must not land in a developer’s backlog. It is a working queue for the people who know the answer, designed so that each case can be decided in seconds and the decision is recorded.

Then, and only then, inferred rules. We are now using AI and custom algorithms to propose deduplication rules. The training material is the corrections sales already made in that interface. Note the direction: the machine learns from business decisions. Building the interface first is what makes the model possible later.

Splink, the open source library most teams reach for when they need probabilistic record linkage, says in its own documentation that it is not designed for linking on a single “bag of words” column such as a company name. Product titles are exactly that column. To get anywhere you have to compare structured attributes, and choosing which two of forty fields actually decide identity in this category is a domain judgment, not a modelling one.

There is a second question the model cannot answer. Which error is more expensive here: merging two products that are genuinely different, or leaving a duplicate in the results? The costs are not symmetric and they vary by category. That threshold belongs to the business. If you set it yourself, you have made a commercial decision in a config file.

Both of those need volume of data and depth of domain knowledge. Most teams have the first.

It is a small development team. The highest-leverage thing I do with them is not architecture review and not code review. It is building their capacity to hold a real conversation with the colleagues who sell the products, and to keep going until they understand the market and the product line properly.

Eric Evans called the result a ubiquitous language: the words in the code and the words in the sales meeting are the same words. When they diverge, every requirement passes through a translation step, and translation is where intent is lost. In a catalog project, intent is the entire product.

A developer who understands why two listings are the same product writes different code. Not tidier code. Different code: fewer branches, better defaults, an exception queue that surfaces cases a human can decide rather than cases a human has to research.

It also makes them better developers, permanently, in a way no course does.

1. Write the business intent of your current data project in two sentences, in the words your commercial colleagues use. If you cannot, you are not ready to write the matcher.

2. Identify the person who can judge the ambiguous case in two seconds. Put them in the loop as an approver of batches, not a reviewer of rows.

3. Ask which error is more expensive. Write the answer down and put it in the ticket, not in a threshold.

4. Send one developer to a customer or sales conversation. This week, not next quarter.

The catalog will never be clean. It can be trustworthy, which is what the customer was asking for in the first place. Nobody owns the duplicates. Both teams own the conversation about them, and the software is where that conversation gets recorded.

• GS1 GTIN Management Standard - the rules for when a product counts as a new product. Read it to see how precise a standard can be, then look at your own catalog.

• Splink documentation - open source probabilistic record linkage in Python, based on the Fellegi-Sunter model. Start with what it says it is not good at.

• Ubiquitous Language - Martin Fowler’s short summary of Evans’ idea. Two minutes, and it explains most failed requirements you have seen.

• Domain-Driven Design Reference - Evans’ own free pattern summary, if you want the source without the 560 pages.

Cleaning up a product catalog, or any dataset where the rules live in people’s heads rather than in a document? Find me on LinkedIn, I am happy to compare notes.

Ciao,

Luca