Exchangeability in AB Testing

Earlier this year, I worked on our cumulative impact feature ; an implementation of the hierarchical normal normal model to enable customers to estimate the cumulative impact of past experiments while adjusting for the Winner’s Curse. In my opinion, the largest difficulty with using this feature is going to be defining an exchangeable set of experiments, and so I thought I would collect my…