Day #10 - Ohh, you mean backed by THAT data...

“I want all decisions to be backed by data”, declared my boss.

She was pumped up. I guess this was the first time she had ever heard of the concept? Apparently, she has talked to the marketing director of quite successful brand in town who told her how they tested everything.

Numbers she wanted. Numbers she got.

In the next meeting, I gave a lengthy breakdown of the ads and their performance. The ads that were doing best were one that the boss didn’t like. They were boring or off-brand or whatever. She had already voiced her dislike of the ads, so it wasn’t a surprise to me… but she was forced to say out loud in front of all the managers that she wanted to kill that ad group.

Sometimes leadership needs to be bold.

Like 6 months later when she boldly laid off everyone.

Bold.

The ads in question were product photos on white background. They were supposed to sell the product, not the brand. However, she wanted to sell the brand.

Her intention and vision was not wrong. That was how she built that business up from the ground to well-known, successful retail stores. But when it came to the goal of these ads, her wires were crossed. She assumed brand ads would sell product and “lame” product ads would hurt the brand.

What did she get wrong?

She was wrong that there are winners and losers.

You’ll occasionally here a CRO talk about having a win-rate. Like, an agency boasting a 75% win-rate or a freelance CRO humbly declaring that they only have a 75% win-rate.

Win-rates are the same as “likes” on a social media post. It’s something to spout off to people who don’t know the nitty-gritty of your industry, but it’s… just gas.

Inconclusive Tests

I don’t want to throw too much shade at inconclusive tests, because they happen. Maybe you thought a change of the headline would emphasize a point you kept hearing from customer feedback, and so you tested it… and it made 0 difference.

That is ok. That is about as close to a “losing” test as you can get. but you still get something. You still learned the change didnt matter. It’s minimal, but not nothing.

This is not a thing. I have never said it in my life, and I never will. If you were talking to a CRO and they said “I don’t track win-rate, but I have a 100% learn-rate,” you should… I don’t know… I don’t condone violence… but I condone batman.

The point is that you learn something from a test.

On my post for Day #9, we started the Collection page test… and guess what?

Flip the results, the control is the variant with the new changes. There is a reason I set it up backwards, which I can go into later, but that doesn’t matter. Regardless, this test is not looking good for me if I look at tests in terms of “winning vs losing.” The original is kicking the shit out of the new one.

If you read the provious post, I mentioned that the reason we added a blurb to the product cards is because customers have told us numerous times that they don’t know the difference between the use cases of the 3 main products… that is something to address.

But clearly, this is not the place to address that.

The purpose of CRO is to learn about your customers — their needs, their psychology, and their behavior. Not to win all the time. Increasing the conversion rate is actually the byproduct of the entire process.

So at the end of each test, if you can answer what’re you going to do about it?, then you have won.

- Shep

Share