#201 | What If Your CRO Program Stops Winning?

Follow us: Slack | LinkedIn | YouTube | Spotify | Substack

• Support Colombia: Donate here

• Community thread: What If Your CRO Program Stops Winning?

• Related videos:

  • Building a mature CRO Program

  • Managing failed experiments without losing stakeholder confidence

• Own a Substack?: Let us know!

• Jobs: New job opportunities at Monks.com and Rho

• Save 50%: Off our Amazon Canada Bestseller, Prove It or Lose It

Support Colombia

Carlos Trujillo recently posted on LinkedIn:

“Two days ago, a magnitude 7.4 earthquake hit us in Colombia 🇨🇴

We managed to quickly grab our kids and evacuate to a relatively safe spot (we're on the first floor) before the ground started shaking to the point of not being able to stand up, while hearing screams, explosions and glasses breaking.

Longest minute of my life.

We were very lucky, though. Escaping with just some minor cracks in the walls and a couple of days without power and comms was the absolute best scenario for us and our closest relatives.

Many people weren't that lucky. Dozens of buildings collapsed, thousands lost their homes, the death toll is in the hundreds, and by the time I write this, people are still trapped under the rubble 15 minutes away from where I'm at.

It's incredibly heartbreaking.

Within the last couple of days, dozens of friends from all over the world have written to ask about the situation and how they can help. First: thank you from the bottom of my heart. It means everything to me.

In terms of how to help (which is why I'm writing this post): donations are the go-to approach.

That said, I know it's hard to donate to local organizations with international bank cards. So I've done my research, and this GoFundMe is the one that makes the most sense. It's easy to donate to from anywhere, and it's getting essentials (water, food, hygiene, medical supplies) straight to communities in Chocó, one of the Pacific regions where the need is particularly critical.

DONATE HERE >

If you have issues after trying, let me know and we'll find an alternative. If you can't donate money, interact with this post to increase its reach.

Your help is invaluable in these times of need 💛💛💙♥️”

Donate here

• No wins? Don’t just run more tests. Diagnose the program.
  Before increasing velocity, check whether experiments are properly powered, running long enough, and aligned to meaningful customer problems. Also look for value beyond “wins,” such as avoiding costly launches or uncovering what doesn’t work.

• The quality of your evidence shapes your win rate.
  Experiment ideas grounded in multiple evidence sources tend to perform better. Start with quantitative data, layer in qualitative signals like session recordings and surveys, and ideally add user research or interviews.

• An inconclusive test isn’t necessarily a dead hypothesis.
  If a test shows a meaningful directional effect but misses statistical significance, look deeper before abandoning the idea. Revisit test duration and power, then consider testing a different design, placement, or execution. Sometimes the hypothesis is right and the treatment simply missed the mark.

• Edd Saunders: (Assuming this problem is from a poor win rate, as opposed to the absence of testing). Based on both Optimizely data, and HBR article about online experimentation, around 10-12% of experiments will be winners - So if you throw a bunch of ideas at the wall, every 10 you’ll have a win.
  To get more wins, you either ramp up velocity, or improve that win rate by making higher confidence bets. Ramping up velocity will be the tougher sell if your stakeholders are getting frustrated with the lack of results, so opt for confidence. How I approach this:

  • Step 1: Make a joke about how the real winners are the friends we made along the way

  • Step 2: Now that stakeholders are happy with your hilarious joke, do a analysis of your programme:

    • You might not have any wins, but have you identified cost savings from testing a feature that was ready to roll out?

    • What are you trying to achieve with the programme?

    • What’s blocking customers from achieving their JTBD. What problems have you identified?

    • What sources of evidence do you have to back up that those are legit problems? Grab a good range, ideally blending quant, qual, user research + meta analysis.

    • What have you learnt from the tests to date and how are you iterating on them?

    • Some basic health checks - are the experiments running to term, powered properly etc

  • Step 3: New set of rolling hypotheses, prioritising the problems which have the most evidence, balanced alongside what’s most important for the business and what’s feasible to build within your resource constraints. In an ideal world, you’ve been selling the benefits of experimentation throughout the programme without needing to rely on ‘wins’, but that doesn’t always happen...

• Eline Schepers:

  • 1. Do you have enough data, running time to get a potential win? Calculate this with MDE

  • 2. if 1. is OK; I would check if the way I’m coming up with the test ideas reflects what users are telling me via quant and qual data; Were it mostly ideas that came from opinions in the company, best practice articles, other websites in the same branch rather then actual funnel pain points, try user testing to find friction, survey surveys too (if you don’t have surveys running on your biggest dropping off points I highly recommend you do that; users will tell you why they quit and you can experiment how to provide an answer to that)

  • 3. if 1. and 2. are OK: Is the test showing some potential uplift? Try a different design, different placing; Inconclusive result doesn’t mean you have to stop or are on the wrong track; If you see some potential in your data you should keep experimenting that hypothesis and see that maybe you were on the right track but the design was not what users wanted

  • 4. if 1. and 2. are OK and you tried out step 3. where you tested out multiple variations and still don’t have any wins you could try option 4: Cry

  Oh maybe before starting to cry you could also plan in a meeting with customer care to see what users often need help with to get some new experiment ideas with a high chance of winning

• Florent Buisson: AFAIK, the 10% success rate is for pretty mature companies. In my experience it has been closer to 20% or even 40%. If you have run 10 experiments and none worked out, I would take a look back to how many had a positive measured effect that was not stat sig. If it’s a meaningful portion, I’d reassess my test duration. If you’re genuinely not having winning ideas, I would ideally take a step back and do some deeper research with UX Research. I know, it’s hard to do when stakeholders are breathing down your neck, but throwing more desperate spaghetti at the wall is unlikely to work better than your past ideas.

• Edd Saunders: You can roughly plot your win rates based on the evidence sources you’re using. A lot of variables of course, but this was my experience:

  • Testing random ideas and hoping for the best = 10%

  • Quant data only (e.g just a funnel drop off analysis) = ~20%

  • Quant + qual (session recordings, heatmaps etc) = ~25%

  • Quant, qual + user interviews = ~30-35%

Achille Vaulot explains how turning inconclusive and losing tests into strategic business learnings helps you build a mature CRO program that delivers lasting value beyond short-term wins.

When your CRO program stalls, keep stakeholder confidence high by tracking guardrail metrics, sharing learnings from failed tests, and treating negative results as valuable insights instead of outright losses.

Own a Substack?

Calling Substack writers in digital marketing, growth, experimentation, and product management: Experiment Nation is looking to connect with other newsletter creators to exchange recommendations and introduce our audiences to each other’s work. If you run a Substack in this space and are interested in exchanging recommendations, reach out to us.

Contact Us

As a THANK YOU for being an active subscriber of Experiment Nation’s newsletter, you get access to Optimization Jobs, a CRO-focused job board with 2000+ open roles from the last 30 days alone.

From Optimization Jobs by Experiment Nation

1. Growth Lead
   📍 GlossGenius – New York, NY

2. Growth Lead, Product & Virality
   📍 Rho – NYC

3. Growth Director
   📍 Monks.com – Shanghai

4. Product Growth Strategist
   📍 Fueledcareers – Remote

5. Engineering Manager, Global Growth
   📍 Lyft – San Francisco, CA

To see more jobs, visit the Optimization Jobs board and explore 3000+ jobs from 400 companies from just the past 30 days that other boards may have missed.

View 3000+ jobs

Get your copy of Prove It or Lose It for $10 USD

The (mostly) no-nonsense guide to surviving experimentation program drama

Are you ready to prove the value of your Experimentation Program? In this actionable guide, Rommil Santiago distills over 15 years of experience and shares all you need to know to survive managing an Experimentation Program. Inside, you’ll learn:

• Frameworks for communicating value effectively

• Strategies to win over stakeholders

• How to build a brand for your program that lasts

Prove it or Lose it is laser-focused on delivering only the actionable advice you need. Your Experimentation Program is only as strong as the value you can demonstrate.

Get your PDF copy for 50% off today!

Learn more here