Reading · Dr. Juan Camilo Orduz

This page cannot be shown here. You can still read it on the original site — the toolbar below keeps your place in the directory.

This notebook implements a simulation-based Bayesian power analysis for conversion-rate A/B tests and compares it against the classical frequentist
two-proportion z-test. We follow the Kruschke HDI+ROPE framework from The Bayesian New Statistics (Section “Planning for precision and other goals: Better done Bayesian”), in the same way as in the introductory example: Introduction to Bayesian…

This notebook implements a simulation-based Bayesian power analysis for conversion-rate A/B tests and compares it against the classical frequentist two-proportion z-test. We follow the Kruschke HDI+ROPE framework from The Bayesian New Statistics (Section “Planning for precision and other goals: Better done Bayesian”), in the same way as in the introductory example: Introduction to Bayesian Power Analysis: Exclude a Null Value. The process is as follows:

Generate synthetic A/B test data across a grid of sample sizes and true relative lifts \(\omega\) (distributions).
For each cell, fit a Bayesian model via MCMC and compute the \(95\%\) HDI of the relative_lift posterior.
Power = fraction of simulations where the \(95\%\) HDI falls entirely outside a Region of Practical Equivalence (ROPE).

We compare three prior specifications motivated by Prior Predictive Modeling in Bayesian AB Testing (see also The Bet Test):

Bayesian Power Analysis for A/B Testing

Comments