Internet Advice is Luck and Survival Bias in a Trenchcoat

I’ve apparently been thinking about this topic for awhile:

What percentage of business advice is survivorship bias masquerading as universal truth? — July 24, 2018

I’m exhausted by survivorship bias and luck masquerading as globally-applicable advice. — April 11, 2024

When it comes to success and failure, there’s a lot of “anecdotal evidence skewed by context and luck” getting sold as the solution to all your problems. Obviously plenty of advice has merit, but it’s not always applicable because the sample size — one person who hit the perfect combo of work, talent, timing, and luck — is too small to be unbiased, and too specific to be universally repeatable.

We all fall victim to various fallacies in our reasoning. They’re like compounding errors in thinking. Survivorship Bias is the one where we disregard failed attempts and only draw conclusions from successes. “I did X, and Y was the result, therefor anyone who does X will also get Y result.” If only it were that simple!

To put it more statistically, survivorship bias is when a sample is non-representative because we tossed out data before we took the sample. The most famous example is researcher Abraham Wald’s proposal for reinforcing World War II Navy planes after examining damage patterns on returning planes. His team’s conclusion was the Navy was focusing on the wrong thing — reinforcing where returning planes were shot. Instead, Wald and team suggested reinforcing the places where the returning planes didn’t get shot, reasoning the one who didn’t make it were likely shot in those places.

You have to measure the right thing, in the right way, to get a useful result.

You can’t trust the Internet

I see survivorship bias manifest itself in business advice on the web and social media. (Hint: you should absolutely not take business advice from the internet. It lies for attention.) A few bits and bobs I’ve come across this week:

• Good design sells itself!

• Just post daily content and clients will find you

• Signal enough competence and you'll never have to ‘sell’

These all feel like variations on the quote “advertising is the tax you pay for being unremarkable” which sounds great on a bumpersticker and plays out precisely no where in the market. All of the above statements could be true, given the right mix of context, timing, personalities, and luck. They’ve seemingly been true for me at certain times. But given a different mix of those elements, or even a different order of them, they could be false.

If “good design sells itself” is a universal truth, but my design work still requires selling, then it logically follows I must not be good at design. But I don’t believe that to be true. My own anecdotal evidence in a 20+ year career invalidates it. So there must be something else going on here. Perhaps “good design sells itself” is not a universal truth, no matter how much certain people might believe it.

If the advice is only applicable to the winners, and handed down from a position post-success, it’s not universal. We’re buying into our own mythologies.

So what can we trust?

I heard a standup comedian do a bit about a Boomer on an airplane vocally decrying peanut allergies as a modern invention — “back in my day, people didn’t have…” — and the comedian saying “no, but how many people did you know that just dropped dead for mysterious reasons?” Before we knew what to measure, the anecdotal evidence is a poor sample, and certainly not enough to draw conclusions with certainty.

We think of pre-historic humans as cavepeople because caves are excellent for preserving evidence. Stuff left outside exposed to the elements weathers away much faster. — Erika Hall

The only people that get invited to speak at design conferences have already been successful. (Editor’s note: except when they invite me.) We don’t tend to want to hear inspirational stories from failures (which is a shame! There’s value, empathy, and things to learn from failure.). But trying to follow someone else’s path and expecting their results rarely works out, despite what all the folks pivoting from Doing The Thing to Selling A Course On How To Do The Thing may be telling you.

Now, I have nothing against online courses. I’m sure I’ll get around to make one eventually, and some trustworthy friends have great courses and programs worth following. But everybody’s selling something. The oft misused aphorism — “Those who can, do; those who can’t, teach” — was a cynical take about class and social ability versus ability in the 1903 play Man and Superman. In our modern online context, it might be more apt to say “Those who could, did (however briefly), but pivoted to teaching how to do because the margins were better.” How’s that for cynicism?

All advice is a mixed bag of anecdote, bias, experience, context, timing, and so on. We have to parse it that way, and then do a bit of “chew the meat, spit out the bones” to find relevant takeaways for our context. There’s good stuff out there. You just have to work to find it.

I trust we’ll figure it out, carve our own paths, and find a measure of success we can live with. Just… buyer beware. This is the internet — here be grifters. And they’re coming for every minute of your attention.