Follow us: Slack | LinkedIn | YouTube | Spotify | Substack
Community thread: How do you decide when an experiment isn’t worth running?
Host our podcast: We’re looking for hosts, guests and presenters!
ICYMI: How to Maximize Hypotheses Testing with Eduardo Marconi Pinheiro Lima
Throwback: Learn how to speed up your CRO program
Jobs: New job opportunities at Lovable and Sony
Save 50%: Off our Amazon Canada Bestseller, Prove It or Lose It
Effort is more than development time. Teams often estimate effort by not only considering engineering time, but also available resources, tooling, dependencies, stakeholder availability, and how long the experiment will need to run.
Not every experiment should be judged the same way. Classifying ideas as iterations, investigations, or moonshots helps set the right expectations and justify larger investments when strategic learning is the goal.
Some changes challenge traditional experimentation. Brand new features or experiences with no true baseline can make A/B testing difficult, leading teams to debate whether pre/post analysis or alternative evaluation methods are more appropriate.
Eddie Aguilar: For me it’s all about effort then effort = time + resources, then break that down further to what time I have available now vs what time the “client” is looking for, and length it may potentially take for the experiment to run. Then resources is broken down by what tooling is available, what people have time available, access barriers and potentially any cross collaboration needed.
Sometimes I look at it and know immediately the effort. Those tend to move the fastest. Sometimes it’s a 6 month build effort, because it’s across many teams and touch points.
Rommil Santiago: Beyond the prioritization frameworks, simply put, I decide whether a test is an iteration, a moonshot, or an investigation. Because I find each has a different bar for whether something is worth running. Something could be very important to know strategically - so it could be worth the extra time. While others are at the tail end of a fruitless roadmap. I also tend to factor in how hard-pressed the program is to show a win.
Gerda Thomas: Maybe when you can’t really measure it? like adding a completely new feature that didnt exist before - meaning like a pop up vs before there was no pop up what so ever. I’ve heard some debate around this where people still measure conversion before pop up vs after but its not really a true comparison
If you’re looking for a place where you can connect with other CROs either to talk shop, or simply shoot the sh*t - consider signing up to join our Slack.
Our Slack is a place to:
Get answers in a judgement-free zone
Share learnings
Talk shop with other CROs
You can join by signing via this form and we will send you an invite within 48 hours.
Experiment Nation is looking for passionate hosts, guests, and presenters to join our YouTube channel. If you love experimentation, CRO, product growth, analytics, UX, or data-driven decision making, we’d love to feature your expertise.
Whether you’re interested in hosting discussions, sharing case studies, interviewing industry leaders, bringing fresh perspectives to the community, or presenting a topic you’re passionate about, there are plenty of ways to get involved. Presentations can also be prerecorded, similar to a conference talk, so if you have an idea, lesson, case study, framework, or story you’d like to share with the CRO community, pitch it to us!
It’s a great opportunity to help educate thousands of practitioners while sharing your expertise and growing your profile. If you’re interested, get in touch. We’d love to hear from you.
One of the biggest mistakes experimentation teams make isn't choosing the wrong ideas, it's abandoning the right ones too soon. In this week's podcast, Eduardo Marconi, argues that prioritization shouldn't always mean adding new hypotheses to the roadmap. Instead, treat each hypothesis like a piece of fruit: if the first experiment produces a strong positive or negative result, keep "squeezing" it. Dig into segments, revisit the research, uncover why it worked (or didn't), and use those learnings to build version two and version three. Only when a hypothesis consistently produces neutral results should you retire it and move on. It's a refreshing reminder that the highest ROI often comes from doubling down on proven customer problems rather than constantly chasing new ideas.
Interesting moments:
38:18 • Why no one gets it right the first time
39:00 • A hypothesis is like a fruit: squeeze it before moving on
40:40 • Why your biggest wins usually come from version 2 or 3
42:40 • How to find the next iteration after a winning (or losing) test
44:20 • Always end every experiment with “What’s next?”
45:20 • How to prioritize follow-up experiments over new ideas
47:30 • Case study: Turning one search filter test into multiple wins
51:15 • The importance of experience in identifying what comes next
57:00 • Better data leads to better prioritization
58:00 • Stop prioritizing button colors over strategic business decisions
Implementing prioritization scores and structured frameworks streamlines CRO decision-making and boosts transparency to significantly accelerate your experimentation program.
As a THANK YOU for being an active subscriber of Experiment Nation’s newsletter, you get access to Optimization Jobs, a CRO-focused job board with 2000+ open roles from the last 30 days alone.
From our Slack community
The founder of Quasi, a US skincare brand, is currently looking for a CRO Lead. It’s a great opportunity for someone with strong eCommerce and conversion optimization experience. You can contact James on LinkedIn directly and send him your CV.
From Optimization Jobs by Experiment Nation
Data Scientist, Product
📍 Lovable – StockholmGlobal Growth Lead
📍 Passage – IndiaSenior SEO and Lifecycle Growth Manager
📍 Colibri Group – 1 RemoteManager, Growth Marketing - Go-To-Market
📍 Sony Interactive Entertainment – United Kingdom, LondonSenior Growth Engagement Manager, International
📍Robinhood – Luxembourg, Luxembourg
To see more jobs, visit the Optimization Jobs board and explore 3000+ jobs from 400 companies from just the past 30 days that other boards may have missed.
Are you ready to prove the value of your Experimentation Program? In this actionable guide, Rommil Santiago distills over 15 years of experience and shares all you need to know to survive managing an Experimentation Program. Inside, you’ll learn:
Frameworks for communicating value effectively
Strategies to win over stakeholders
How to build a brand for your program that lasts
Prove it or Lose it is laser-focused on delivering only the actionable advice you need. Your Experimentation Program is only as strong as the value you can demonstrate.
Get your PDF copy for 50% off today!
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.