Follow us: Slack | LinkedIn | YouTube | Spotify | Substack
Our latest episode: Ethical CRO: How Better Experiments Build Trust, Retention & Revenue
Community thread: How Do You Experiment Offline?
Host our podcast: We’re looking for hosts (and guests)!
Jobs: New job opportunities at Enavi and Gympass
Save 50%: Off our Amazon Canada Bestseller, Prove It or Lose It
Jon Crowder joins Tracy Laranjo to explain why ethical experimentation is not just better for customers—it is better for long-term growth. They unpack fake urgency, misleading pricing and subscription tactics, short-term A/B test wins that erode trust, and practical ways to design experiments that improve retention, lifetime value, and revenue.
Jon’s LinkedIn: https://www.linkedin.com/in/jc-awip
Jon’s agency website: https://www.anotherwebispossible.co.uk
Chapters:
0:00 Welcome & introducing Jon Crowder
1:29 Why ethical experimentation matters
2:17 The pressure to drive results at any cost
7:17 When winning an A/B test loses money long term
8:34 Reframing urgency into trust-building experiences
13:38 Choosing clients and setting ethical boundaries
16:37 The ethics of price testing
19:21 Fake sales, subscriptions & agency pressure
20:46 How to steer unethical tests into ethical ones
23:09 Have urgency and scarcity gone too far?
24:02 Why legacy businesses cling to outdated tactics
27:36 Gen Z, trust, and changing expectations
29:48 What an ethical future for digital experiences looks like
34:30 Brands that earn loyalty through values
38:27 Why ethical experiences convert better
40:15 A simple test for ethical experimentation
41:37 Where to find Jon
Photo credit: Adobe Firefly
Offline experiments require different statistical thinking. Unlike digital tests, in-store experiments often suffer from small sample sizes and require careful randomization at the store level to avoid misleading results.
Measurement is more complicated than it seems. Customers don’t always follow the intended path. Someone may see an in-store promotion but install the app later without scanning a QR code, so combining direct attribution with regional or store-level lift analysis can provide a more complete picture.
The same experimentation principles apply beyond digital. Organizations like Mastercard routinely test store layouts and other offline experiences, proving that rigorous experimentation can drive decisions in physical environments just as effectively as online.
Andrei S: Did anyone here run in-store (offline) experiments? I’m currently planning to take experimentation offline and I’m curious to learn how you went about it and if you found what you were looking for. Thank you!
Johan: It’s in a sense a bit like B2B experimentation, you run into sample size limitations (so you mostly learn about that there were no negative effects) and that you must be careful about randomization (if you randomize on a store level, either calculate metrics on the store level or do some clever clustering without tripping yourself) What kind of experimentation did you have in mind?
Andrei: Not final, but something along the lines of X% of stores to feature banners at the entrance with QR codes guiding people to download an app to see offers. The link would feature some UTMs so we know who downloaded the app via this channel & the main metric is.. total non-direct App downloads. I’m unsure of how to measure this in the best manner but we’re aiming to leverage stores to get more online customers. Any advice would be helpful!
Rommil Santiago: I have yes. Tested in-store SOPs for grocery stores - it was tricky because we had to pick comparable stores across many dimensions so that we could have a synthetic control. You ought to be mindful that folks may see the banner and download without scanning. so if you have a way to track installs by city/region that could give you some insight into the bigger impact of the effort.
Ishan Goel: Hey Andrei, I recently met Michael Paolini, VP of Test & Learn at Mastercard. They run a lot of offline experiments like testing out different store layouts. Also, Stefan Thomke’s Experimentation Works have a lot of stories of offline experiments that you can learn from.
If you’re looking for a place where you can connect with other CROs either to talk shop, or simply shoot the sh*t - consider signing up to join our Slack.
Our Slack is a place to:
Get answers in a judgement-free zone
Share learnings
Talk shop with other CROs
You can join by signing via this form and we will send you an invite within 48 hours.
Experiment Nation is looking for passionate hosts, guests, and presenters to join our YouTube channel. If you love experimentation, CRO, product growth, analytics, UX, or data-driven decision making, we’d love to feature your expertise.
Whether you’re interested in hosting discussions, sharing case studies, interviewing industry leaders, bringing fresh perspectives to the community, or presenting a topic you’re passionate about, there are plenty of ways to get involved. Presentations can also be prerecorded, similar to a conference talk, so if you have an idea, lesson, case study, framework, or story you’d like to share with the CRO community, pitch it to us!
It’s a great opportunity to help educate thousands of practitioners while sharing your expertise and growing your profile. If you’re interested, get in touch. We’d love to hear from you.
As a THANK YOU for being an active subscriber of Experiment Nation’s newsletter, you get access to Optimization Jobs, a CRO-focused job board with 2000+ open roles from the last 30 days alone.
From the Community:
Senior Growth Experimentation specialist
📍 Enavi- Remote
From Optimization Jobs by Experiment Nation
Manager, Retention and Growth
📍 Razorpay – MumbaiSenior Product Manager, B2B Growth
📍 Grammarly / Superhuman – Hub - San FranciscoSenior Full-Stack Software Engineer, Growth
📍 Discord – San Francisco Bay AreaDirector of Growth Marketing (Partner Acquisition)
📍 Gympass – Brazil (São Paulo - Hybrid)Senior CRO Designer, Paywalls and Funnels
📍 RevenueCat – Americas
To see more jobs, visit the Optimization Jobs board and explore 3000+ jobs from 400 companies from just the past 30 days that other boards may have missed.
Are you ready to prove the value of your Experimentation Program? In this actionable guide, Rommil Santiago distills over 15 years of experience and shares all you need to know to survive managing an Experimentation Program. Inside, you’ll learn:
Frameworks for communicating value effectively
Strategies to win over stakeholders
How to build a brand for your program that lasts
Prove it or Lose it is laser-focused on delivering only the actionable advice you need. Your Experimentation Program is only as strong as the value you can demonstrate.
Get your PDF copy for 50% off today!
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.