Stop Calling the First Significant Day a Win

Checking an A/B test until it crosses p 0.05 can turn a nominal 5 percent false-positive rate into almost 28 percent. I use a seeded simulation to show how large the damage gets and compare the fixes that keep early stopping honest.