#199 | How do you choose an MDE?

Follow us: Slack | LinkedIn | YouTube | Spotify | Substack

• Community thread: How do you choose an MDE?

• Host our podcast: We’re looking for hosts, guests and presenters!

• ICYMI: Jon and Tracy discuss ethical CRO

• News from Vendorland: Target and Croct

• Jobs: New job opportunities at Adobe and Notion

• Save 50%: Off our Amazon Canada Bestseller, Prove It or Lose It

Photo credit: Adobe Firefly

• Flip the MDE question: Instead of asking “What’s a meaningful lift?”, ask “What lift can we realistically detect within our traffic and timeframe?”

• Tie MDE to business value: Choose detectable effects based on the revenue, conversions, or customer impact they represent.

• Let historical data guide expectations: Use past variance and performance to set realistic MDEs instead of relying on guesswork or default calculator values.

• Rommil Santiago: Prior to running an A/B test, how do you decide what percentage change is meaningful?

• Eddie Aguilar: 100 % catches errors faster than 1%, just saying. Professional answer: I calculate it based on what different uplift thresholds represent. I.e, a 5% uplift means n more conversions, meaning £n additional revenue, 10% means zyx etc... Real answer: I put 5% in the ‘expected uplift’ box

• Ishan Goel: I personally think a reverse MDE calculator should be made popular. Rather than thinking, what change is meaningful which is very hard to think. You should think about what change is detectable in a meaningful time-frame. And then you should decide whether to test the idea or not. A calculator I and Haley Carpenter built for this is here and it can be copied and used: https://docs.google.com/spreadsheets/d/1pwk7--j9QRcU2VGvUKHbL6vHK16BXASUUjcIDjoT3D4/edit?usp=sharing

• Rommil Santiago: Other than sorting out what matters to the business, another approach I’ve used is looking back at historical, figuring out what values would be a % of a stdev beyond the average so we could actually detect it - which i guess is an ungraceful version of reverse MDE

• Tally Keller: Thanks for the question, @Rommil Santiago and that worksheet, @Ishan Goel . Good topic of discussion to bring to my execs.

If you’re looking for a place where you can connect with other CROs either to talk shop, or simply shoot the sh*t - consider signing up to join our Slack.

Our Slack is a place to:

• Get answers in a judgement-free zone

• Share learnings

• Talk shop with other CROs

Request a Slack invite

You can join by signing via this form and we will send you an invite within 48 hours.

Experiment Nation is looking for passionate hosts, guests, and presenters to join our YouTube channel. If you love experimentation, CRO, product growth, analytics, UX, or data-driven decision making, we’d love to feature your expertise.

Whether you’re interested in hosting discussions, sharing case studies, interviewing industry leaders, bringing fresh perspectives to the community, or presenting a topic you’re passionate about, there are plenty of ways to get involved. Presentations can also be prerecorded, similar to a conference talk, so if you have an idea, lesson, case study, framework, or story you’d like to share with the CRO community, pitch it to us!

It’s a great opportunity to help educate thousands of practitioners while sharing your expertise and growing your profile. If you’re interested, get in touch. We’d love to hear from you.

Appear on our YouTube channel

• Adobe Target - Bayesian statistics for A/B Test (Manual) activities
  A/B Test (Manual) activities now support Bayesian statistics as an alternative to Welch’s t-test. Choose the statistical methodology in Goals & Settings: a Bayesian activity’s report shows probability-based decision metrics, such as Chance to Beat Control, and credible intervals instead of the frequentist confidence interval.

• Croct - Product update: estimate your audience size
  The audience estimator helps you understand how many users match an audience criteria. It estimates what percentage of your user base would qualify for the audience at the moment you generate the estimate. By providing immediate feedback while you refine your criteria, the segmentation process becomes much faster and more informed.

Jon Crowder joins Tracy Laranjo to explain why ethical experimentation is not just better for customers—it is better for long-term growth. They unpack fake urgency, misleading pricing and subscription tactics, short-term A/B test wins that erode trust, and practical ways to design experiments that improve retention, lifetime value, and revenue.

• Jon’s LinkedIn: https://www.linkedin.com/in/jc-awip

• Jon’s agency website: https://www.anotherwebispossible.co.uk

Chapters:
1:29 Why ethical experimentation matters
2:17 The pressure to drive results at any cost
7:17 When winning an A/B test loses money long term
8:34 Reframing urgency into trust-building experiences
13:38 Choosing clients and setting ethical boundaries
16:37 The ethics of price testing
19:21 Fake sales, subscriptions & agency pressure
20:46 How to steer unethical tests into ethical ones
23:09 Have urgency and scarcity gone too far?
24:02 Why legacy businesses cling to outdated tactics
27:36 Gen Z, trust, and changing expectations
29:48 What an ethical future for digital experiences looks like
38:27 Why ethical experiences convert better
40:15 A simple test for ethical experimentation

Watch the full episode

As a THANK YOU for being an active subscriber of Experiment Nation’s newsletter, you get access to Optimization Jobs, a CRO-focused job board with 2000+ open roles from the last 30 days alone.

From Optimization Jobs by Experiment Nation

1. Product Designer, Acrobat Mobile Growth
   📍 Adobe – 3 Locations

2. Growth Creative Producer
   📍 Claylabs – San Francisco

3. Growth Marketing Manager, Web & CRO
   📍 Cresta – United States (Remote)

4. Outcomes Growth Strategy and Programs Lead
   📍 Notion – San Francisco, California

5. Senior Marketing Manager, Growth
   📍 Webflow – U.S. Remote

To see more jobs, visit the Optimization Jobs board and explore 3000+ jobs from 400 companies from just the past 30 days that other boards may have missed.

View 3000+ jobs

Are you ready to prove the value of your Experimentation Program? In this actionable guide, Rommil Santiago distills over 15 years of experience and shares all you need to know to survive managing an Experimentation Program. Inside, you’ll learn:

• Frameworks for communicating value effectively

• Strategies to win over stakeholders

• How to build a brand for your program that lasts

Prove it or Lose it is laser-focused on delivering only the actionable advice you need. Your Experimentation Program is only as strong as the value you can demonstrate.

Get your PDF copy for 50% off today!

Learn more here