Implementing an experimentation tool? Here's what you need to be checking

An experimentation tool (especially a client-side one) is, by nature, an agent of chaos for your site. I mean, the tool is there to mess with the user interface! Even if you’re using server-side testing, it’s still adding an element of chaos. What’s more, after adding all that chaos, you need super accurate measurements from analytics!

I’d also argue that tracking accuracy becomes more critical if you’re analysing an A/B test (I’m using A/B as shorthand for all types of tests, by the way).

My point is this: you need to run validation checks after installing or modifying an experimentation tool.

But what checks do you need to run? And why? Matt and I thought we’d have a quick chat to discuss the topic. Choice takeaway from the chat:

“There’s always some bleed between variations. The world is messy — people clear cookies, switch devices, and wear Spider-Man costumes during podcast recordings.”

The bleed mentioned here is about A/B test data contamination, where the same user is counted in both groups of an experiment. Some “bleed” is normal. We need to worry, though, if the numbers are crazy. What this crazy number is will likely vary from site to site, but we need a sense of a baseline. And that’s why it’s a good idea to run these checks as early as possible.

More stuff like that in the video above.