I run a side project called Switch Scores. It pulls in game reviews from partner sites through their RSS feeds, matches each review to the right game, and builds up a score.
The matching part is the fiddly bit. A feed doesn’t include a clean game title, it hands over a headline like “Nintendo Switch 2 Review: Donkey Kong Bananza”. Each site writes those differently, so each one has its own rule for pulling the game name out of the headline before I can look it up. When a site changes its headline format, the rule breaks, and I have to manually link the review to the game.
What I actually see day to day isn’t a broken rule. It’s a review sitting there with no game attached, waiting for me to link it up. There are always a few of those, and some are unavoidable. Games with subtitles, or with a colon or a hyphen in the name, get written slightly differently from site to site, and that difference is often enough to stop a match. But in small numbers, they don’t point to a pattern.
What changed was the volume, and which sites it was coming from. Nintendo Life is one of the most reliable sites I pull from. When I found myself hand-matching most of its reviews, that didn’t seem right. I checked the rule, fixed it, and then ran into the same thing on a second site I’d never had trouble with.
When the issue kept on happening, I knew I needed a better approach.
A partial failure looks exactly like success
Nothing had broken. No error, no alert, nothing red on a dashboard.
Worse than that, one feed was showing that its last run was successful, but the feed had been dead for eight months. That status only ever got written after the feed loaded, and this one was failing before it got that far. So it kept reporting the last thing that went well.
A system that fails completely is easy to spot. It stops, something errors, and someone chases it that day.
A system that keeps running while writing incomplete or incorrect data is easy to miss. Every page still loads. Every job still finishes. The gap exists only in the data, and unless something is checking the data, nobody is checking at all.
The failures that cost you the most are rarely the noisy ones.
Waiting for the next failure
Once I could see the pattern, fixing the rules for the two sites I’d spotted was the obvious move. But there’s a catch with fixing a rule that way: the only way to know it works is to wait for the next review to come in. Some sites publish infrequently, so there may be nothing recent to check against.
There was a bigger problem underneath that: I only knew about the sites I’d happened to notice. So rather than fix them one at a time and wait to find out, I built a tool that could test any rule against the titles I already had. That let me work through all of them proactively, instead of waiting for the next failure to surface.
The tool took longer to build than the first two fixes took to make. Once it existed, repairing the rest took about an hour. Nintendo Life went from 10% to 73%. Rapid Reviews went from 0% to 63%.
The rules were never the hard part. Being able to check them all was.
Like this post? Product Notes is a free weekly newsletter for product people. Subscribe to get it in your inbox.
The workaround was doing the hiding
The reviews still got created. The site still worked. Readers saw scores.
They saw scores because I was matching the unmatched ones by hand, without ever really registering that I was doing it. The bug didn’t stop the outcome, so the fault had nowhere to show up. I’d absorbed the failure and taken on the matching myself. Same end result, but wrong process.
This isn’t a technical problem, but a human one. Whenever a person is manually bridging a gap in a system, the gap stops being measurable. In larger organisations, the manual work is invisible to everyone except the person doing it.
Where this shows up in product work
I don’t think this is a story about RSS feeds.
It’s like shipping a feature with no tracking and then wondering if it worked. Six weeks later someone asks how adoption is going and the answer is a story about two customers who mentioned it on a call. That’s not a measure of adoption, it’s the two people you happened to hear from.
It’s like a support team building a spreadsheet to work around a broken flow, and that spreadsheet becoming the reason nobody escalates it.
It’s like a metric that only moves when something is catastrophically wrong, so a slow decline reads as stable.
In all three cases the problem isn’t that people are ignoring bad news. It’s that the bad news has no way of reaching them.
The better question
When you find something broken, the natural question is “how do I fix this?”
The more useful one is “why didn’t I already know about this?”
If something flagged the issue and you found it quickly, nothing needs to change. More often, the answer is that nothing was checking. If nothing was checking for this, there’s a good chance something else is going unnoticed too.
That’s the point to stop fixing and start building a way to know. It doesn’t have to be much. Mine was a single page showing a match rate per site, and a number saved each night so I could watch it move.
One page, every site, including the ones I haven’t added yet.
What I’d take from it
Fixing the broken thing feels like progress because it is progress. It’s just usually the smaller half.
The bigger half is asking what else looks fine right now for exactly the same reason this did, and building the thing that would tell you.
What are you fixing by hand at the moment, and how would you know if it got worse?
Product Notes is free and lands every week. Subscribe to get it in your inbox.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.