A More Complete Post on Decision Theory and EVSI in A/B Testing

We’re gearing up for a really cool decision-theory feature at Datadog. In anticipation, I thought it would be useful to revisit some fundamental material on the subject. This post revisits three crucial topics in decision theory in the context of A/B testing: loss, the expected value of perfect information (EVPI), and the expected value of sample information (EVSI). We’ll discuss how to interpret…