The peeking problem: why sneaking a look at an A/B test inflates false positives

On 21 January 2015 Optimizely — one of the most widely used A/B testing platforms in the world — switched on a completely new statistical engine for all of its customers, the New Stats Engine. It wasn t a technical whim: the old engine, built around a classic fixed-horizon t-test (Fixed Horizon) and developed with statisticians Continue reading "The peeking problem: why sneaking a look at an A/B…