RSS Amplifier

Experimental Mind · Jun 1, 2026

'Value of evidence-based decision making' — Digest 22, 2026

Read inOriginalenesdefrptjaittr
0
Sign in to vote or save

Kevin Anderson · Experimental Mind

This week: how to value evidence-based decision making? A new paper has an answer. Also: LLM evals meet A/B testing, and why strong ideas don't need to ship immediately.

You'll also find 914 open roles, upcoming events, and a cartoon that made me smile.

Convert—A/B tests & personalization for growth teams

Experimentation Jobs—Find your next role

[Paper] Estimating the value of Evidence-Based Decision Making
By Abadie, Agarwal, Imbens et al. (MIT/Stanford/Amazon) — How much is running experiments actually worth? This paper builds a framework to calculate it. Using empirical Bayes on 4,8k+ Upworthy A/B tests, they show that standard p-value decision rules leave ~27–30% of attainable value on the table. READ

____

[Paper] Two-sided online review systems drive desirable customer behaviors
By Marder et al. — Eight experiments confirm it: when Airbnb or Uber can rate you back, you clean more, tip more, and talk more politely. Simply because your score is at stake, not because you’re kind. People fear being judged by someone who controls your future access. And it only kicks in when you plan to use the platform again. READ

My reflection: this research reminds me of the Black Mirror episode Nosedive. Here they depict a world where every social interaction is rated, and people perform relentless niceness to protect their score. This paper is the empirical evidence that we are already there. Not sure I like this state of things.

____

Stop building innovation labs
By Jack Strachan — The first wave of public sector labs worked because of the conditions around them: protected authority, proximity to decisions, evidence through building. When those conditions disappeared, so did the impact. The lab was never the innovation. Experimenters need to understand: a faster feedback loop only matters if it’s attached to something consequential. READ

____

Better experiments with LLM evals — a funnel, not a fork
By Matilda Ankargren and Mårten Schultzberg (Spotify) — LLM evals verify quality before an experiment; A/B tests validate real user impact after. Use evals to raise your experiment hit rate, then run evals on A/B data to calibrate the judge. Each cycle makes both smarter. READ

____

The website specification
By Joost de Valk — Great reference for what a good website should do, regardless of the stack. Every topic has a status: required, recommended, optional or avoid. CHECK

____

Why I don't write every day
By Mike Fisher — Not all work benefits from daily output. Strong ideas don't fade, they accumulate tension until writing (or shipping) becomes inevitable. In the context of experimentation: not every hypothesis deserves immediate execution. Time is a filter, and too early convergence is not good. READ

____

Last week’s favourite:

Tips for buying a new A/B testing tool
Ruben de Boer created three short videos packed with tips on how to approach selecting a new A/B testing vendor. Take your time and make a good, solid decision. LEARN

Find 914 open roles on ExperimentationJobs.com. This week’s featured roles:

A running list of upcoming events. Subscribe here. (👋= join me, 🎁= discount)

If this newsletter sparked something and you want to talk it through, a few ways I help:

  • Scale experimentation
    Strategy, setup, metrics, and ways of working for teams that want more impact.

  • Careers and hiring
    Support with your next role, or help finding and hiring strong experimenters.

  • Quick sparring
    A fresh outside perspective on your ideas, roadmap, or experimentation setup.

Interested?
Just hit reply and tell me what you’re thinking about.

Have a great week — and keep experimenting.

Thanks, Kevin

Share Experimental Mind

Read the original on kevinanderson.substack.com ↗

Comments

Nothing yet. Say the first thing.

    Sign in to join the conversation.