Hi TLCers!
Since last summer, I’ve been digging into how AI is shifting our work, and with some recent reading, it’s becoming increasingly clear to me that we might not be probing deep enough or asking the right questions.
The piece that really pushed me from “we need to start learning” to “people are underreacting” is what Sani shared recently about AI-referred traffic. Twelve months ago, that traffic converted significantly worse than regular (non AI-referred) traffic. That made sense: new channel, low trust, early adopters. Of course, it would convert worse.
That is no longer true.
In less than a year, AI-referred traffic performance flipped. Not slowly. Not incrementally. AI-referred traffic is now outperforming non-AI-referred traffic, and not by a rounding error.
That doesn’t happen in our world. Not like that.
And data suggests that optimization is falling behind. Why? Because we’re facing a structural shift in how buying decisions are being made, and these changes aren’t evolving slowly; they’re accelerating. Fast. Here’s what’s happened;
The sign has flipped on AI-referred traffic. Last year: negative. This year: positive. AI-referred traffic now converts better than non-AI-referred traffic.
A website-as-discovery model is out. A website-as-confirmation model is in.
If your site’s illegible to AI, it’s invisible to your customer.
Review sites can now be the foundation of a site’s visibility.
AI is now the shortlist-generator for vendor discovery and selection.
All of this means we need to stop treating AI’s effect on our work as a side conversation and start asking what it actually means for how we work. Because the way people are discovering, evaluating, and deciding has changed this quickly, the way we test, measure, and optimize must change with it.
I published a deeper breakdown of all of this, but I want to summarize the part that matters most for this community.
The funnel didn’t disappear. It moved.
A meaningful portion of the decision-making process is now happening before a user ever reaches your site. It’s happening inside AI chat & synthesized answers, and within systems that perform the comparison, filtering, and shortlisting for the user.
That one shift explains a lot of what people are starting to see in their data, but can’t quite reconcile. The data looks like shorter sessions paired with higher conversion rates, less browsing, and more decisive behavior. We’re seeing traffic that looks “worse” by traditional engagement metrics, but that drives more revenue.
That’s not your site suddenly getting better at persuasion. It’s your user arriving already persuaded.
This creates a very different job for optimizers.
For years, we’ve been optimizing for discovery and influence. We built journeys, shaped narratives, and tested ways to move people from curiosity to conviction.
Now, in many cases, the AI is doing that work upstream. By the time someone lands on your site, they’re not there to explore. They’re there to confirm what they already learned from our AI overlords.
If what they see doesn’t match what the AI told them, trust breaks immediately. And when trust breaks, conversion drops just as quickly.
That shift alone would be enough to rethink what we’re doing. But there’s a second layer that’s even more uncomfortable.
We are no longer observing the full funnel.
We are observing the survivors.
The people who make it to your site are the ones the model surfaced correctly (the ones who found what they were looking for - a good match of user to product) and didn’t filter out. The ones who never made it through the AI filters—because they saw a shitty review from 2007, or were steered elsewhere based on outdated information, etc. —are completely invisible to your analytics, so you cannot see or fix the problem keeping them away.
This means a lot of what we’re measuring right now is incomplete in a way we’re not accounting for and cannot account for.
And that has real implications for experimentation:
Your audiences are being pre-qualified upstream in ways you didn’t design.
Your baselines are shifting as traffic composition changes.
Your test results are mixing fundamentally different types of visitors without you realizing it.
And we’re still largely running the same kinds of tests, in the same places, using the same mental models.
That’s the part that worries me.
If the decision is increasingly being made elsewhere, we need to start asking: What are we actually testing? Where should we be testing? And what does “optimization” even mean when part of the system lives outside our control?
I go much deeper on this here:
The Funnel Didn’t Break. It Just Moved Somewhere You’re Not Measuring
If you want to go straight to the source material:
And I strongly recommend reading and following the people who are actively pushing this conversation forward, because they’re the ones helping make sense of what’s happening in real time. People like:
Once you start looking at the world this way, the rest of this month’s articles hit differently.
They’re not just interesting reads. They’re pieces of a much bigger shift in how our discipline is evolving.
It’s easy for an experimentation program to slowly devolve into glorified bug-testing. When we only track transactional metrics, we lose sight of user intent. True optimization requires an architecture that can natively integrate direct customer feedback into the experiment itself.
To celebrate 15 years of independent building and the launch of the new Conductrics.com, Matt Gershoff and Nate Weiss wrote a piece about how the industry must pivot toward true, customer-aligned experimentation. Read the full post here.
Quick note for transparency: Yes, Conductrics sponsors the TLC, and has from the very beginning. This isn’t that. No quid pro quo here.
This is me being very happy that one of the most thoughtful teams in experimentation is finally showing up and saying what they actually believe about how this work should be done.
It’s worth your attention when or if you’re in the market.
My $0.02 on the new Conductrics website. It’s about damn time. 😘
When the buyer is a model, conversion rate optimization becomes token-space optimization.
Across five Knowledge Distillation episodes, Sani Manić, Josh Silverbauer, Simo Ahava, Tim Wilson, and John Lovett trace what agentic commerce is doing to the web stack, measurement, the data layer, and traffic detection — and what it means for the experimentation discipline. Semantic HTML is back on the critical path. Log files are the new session recordings. Your test cells are quietly mixing humans and agents into the same cohort, flattening every lift you ship. The old skills still work; the canvas is just larger. Read the whole article on the Ask-Y blog site.
A team of 4.
500+ experiments.
$1M → $35M ARR in a single year.
No, that’s not a typo.
This behind-the-scenes look at Fyxer’s growth engine shows what happens when experimentation isn’t a “program” ... it’s the operating system. By combining an AI-first workflow with a relentless test-and-learn mindset, this team ran experiments at a pace most companies can’t even imagine, turning constant iteration into compounding growth.
This is worth your time. Read the full breakdown.
Our monthly Convert-sations with Convert are back and better than ever. This community-powered series is still all about sharing real perspectives on experimentation, whether you’re just getting started or have years under your belt.
The questions are all about getting to know YOU.
It’s easier than ever to share your thinking, learn from others, and join the conversation.
Even better, you’ll have more opportunities to be featured in Convert content and win monthly prizes. Win prizes for:
🐣 First-Time Contributor
💡 Most Actionable Insight
👋 Getting to Know You
🚀 Convert Contribution
Head to the #convert-sations Slack channel, pick a question, and jump in.
🐣 First-Time Contributor
Jason Packer (TLC: @Jason Packer)
We are all excited to welcome THE Jason Packer to Convert-sations and look forward to many more contributions in the coming months. It was a pleasure to see his name pop up - and it was funny - because we realized it was his “first-time”, so we had to wait on “most actionable insight”. Good thing there were many to choose from!!
💡 Most Actionable Insight
Elizabeth Gilbert (TLC: @Elizabeth Gilbert ) answered the Community question, “What is something new or something you will be improving in your experimentation strategies?”
“I’ve recently been thinking about getting stakeholders involved earlier in the process, showing them how experimentation can support their needs by involving them from the beginning.
I took a lot of guidance here from Ruben de Boer’s episode of Lucia van den Brink’s podcast, The Initial @ 22:40”
We love this podcast!! And we love that our TLC members can immediately take action by listening to this podcast and learning from fellow TLCers!!
👋 Getting to Know You: “What’s your favourite way to connect with others at live events?”
Dylan Lewis (TLC: @dylanlewis )
A bit more tactical, but having a set of questions that you are confident in asking, and can hold a conversation with someone for a few minutes worked well…I tried an experiment at experimentation island, and these questions seemed to be most useful:
What is your favorite sunset?
What brings you joy?
What is your favorite animal?
They could be done in roundtables, at dinner, or on a plane traveling to the event. I try to stay away from:
Which button color test was your favorite (TLC: @Shiva Manjunath)
We loved the practical advice; we loved that you experimented with it at a live event. And we love YOU and your sunsets. 💜
We urge you to hop on to the #convert-sations Slack channel to see how other TLC members weighed in — and share your thoughts for next month’s questions!
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.