Better Priors for Bayesian Analysis of A/B Testing

Motivation When estimating a treatment effect (typically the lift), , using the Bayesian methodology, Datadog uses a default prior of Do I really believe that true effects are normally distributed? No, not really. Under such a model, we should see true lifts larger than 10% in absolute value less than 5% of the time, and lifts less than 20% less than 1% of the time. Roughly, that means 1 in every…