We’re gearing up for a really cool decision-theory feature at Datadog. In anticipation, I thought it would be useful to revisit some fundamental material on the subject. This post revisits three crucial topics in decision theory in the context of A/B testing: loss, the expected value of perfect information (EVPI), and the expected value of sample information (EVSI). We’ll discuss how to interpret…
Earlier this year, I worked on our cumulative impact feature ; an implementation of the hierarchical normal normal model to enable customers to estimate the cumulative impact of past experiments while adjusting for the Winner’s Curse. In my opinion, the largest difficulty with using this feature is going to be defining an exchangeable set of experiments, and so I thought I would collect my…
Before we get started, I want to say that here may be good reasons to use a normal prior over a Student’s t prior. Statistics may be the science of defaults , but I am not recommending the Student’s t as the default prior for anyone. Rather, this post is going to take a look at some of the properties of the normal prior as they relate to prior data conflict and we’ll see how the Student’s t’s tail…
The “Text” section of my “How I Use AI” reads All text on this blog, and in publications in which I am listed as an author , is written by me […] I will never copy-and-paste from an LLM, nor will I ask an LLM to write a blog post, section of a paper, or email for me. and I am beginning to think about changing this. The fact of the matter is that I have never really been good at scientific writing,…
Motivation When estimating a treatment effect (typically the lift), , using the Bayesian methodology, Datadog uses a default prior of Do I really believe that true effects are normally distributed? No, not really. Under such a model, we should see true lifts larger than 10% in absolute value less than 5% of the time, and lifts less than 20% less than 1% of the time. Roughly, that means 1 in every…
Let be a sample of dimensional random variables with mean . Their covariance is defined as I don’t blame you for not really grokking what this means. Should you want to know, then we will need to revisit some fundamental concepts in linear algebra, starting with projections, moving on to outer products, and finally landing on covariances before briefly touching on PCA. Projections Consider a…
I work 10 hours per week at my University’s medical school consulting on statistics for residents and physicians. I’ve noticed far fewer requests for work, and I think AI is the reason. I have some thoughts on this. Economics, Statistics, and Tukey Tukey once said The best thing about being a statistician is that you get to play in everyone’s backyard. I take the phrase to mean that because…
Important In accordance with my AI policy, I have written this blog post myself. However, all code is written Claude and a lot of the mathematics is done in conjunction with Gemini. Introduction I’ve been thinking about the hierarchical normal-normal model a lot recently. We’re planning to build some sort of empirical Bayes estimation service for a variety of reasons (I won’t expound on the…
This post is a spiritual successor to an early post on this blog and a fast follow here from this post about A/B test run times for Bayesians . From that post, we are able to obtain an analytic approximation to the expected impact from a single experiment using a Bayesian analysis and a given decision rule. Below is a plot like the one found in my other post, but this time perhaps with a more…
Important This blog post is inspired, almost entirely, by Tyler Buffington’s excellent statistics reading group presentation today at Datadog on the value of information in A/B testing. The ideas presented here are mainly his, I’ve just gone ahead and done a little more math. I’ve also relied a lot on AI to help me get through some of the math. I can’t remember all the tricks there are for…
I asked claude code to read through all the posts I’ve made in this blog in order to pull out some of my principles. Here is what claude thinks I believe Empathy First, Math Second I think this is largely true, as evidenced by a previous post . I’ve said, multiple times in the past week, that “experimentation is not a math problem, it is a people problem”. I think this also explains why I like…
I still clean data from time to time, and I find myself writing patterns that looks like raw_data %>% mutate ( smoking = case_match ( str_to_lower (smoking), "y" ~ "Y" , "n" ~ "N" , .default = "N" ), hypertension = case_match ( str_to_lower (hypertension), "y" ~ "Y" , "n" ~ "N" , .default = "N" ), diabetes = case_match ( str_to_lower (diabetes), "y" ~ "Y" , "n" ~ "N" , .default = "N" ) ) # A…
On The Validity of a Normal Approximation to The Ratio of Means A/B tests often target the “lift” in the metric as an estimand. As a reminder, the lift is and is interpreted as the percent improvement treatment had over control. Clearly, the lift can be problematic when – the expected value of the outcome under no treatment – is too close to 0. This is not a revolutionary insight and has been a…
To the Economists, I am sorry . What is a Fixed Effect Model? A model with fixed effects is the following Subjects (indexed by ) are followed over time (indexed by ) and outcomes are recorded. Covariates are also recorded and we fit the model using OLS. Each subject has an “idiosyncratic” effect and there are shared coefficients , Now, we don’t need to follow subjects over time to use a fixed…
I’ve been climbing this hill for what feels like a long time. I can look back and see how high I’ve climbed, the rocks I’ve had to overcome, and the gullies I nearly was trapped in. But as I look forward, all I see is more hill. A lot of people are on this hill with me. Some are climbing with me, some are on other paths and we are just meeting either by accident or because this climb was more…
Take a look at this interaction I had with user jginestet on Cross Validated. In short, jginestet suggests that a randomized experiment assessing percent changes in tumor volume should adjust for pre-experiment differences (at this point, I’m nodding yes) and therefore the most appropriate method is DiD – difference in differences – (at this point, I’m shaking my head no). User jginestet is right,…
I keep coming back to A New Look at P Values for Randomized Clinical Trials – an instant classic paper on RCTs, which could be extended to online experimentation as well. The authors, who include Gelman and Greenland, take an empirical Bayes approach to infer the joint distribution of z statistics and the Signal to Noise Ratio (SNR, the true effect divided by the standard error) from the Cochrane…
I recently wrote a blog post for my employer in which I made some claims regarding Bayesian statistics. Claim 1: Flat Priors in Heirarchal Models Can Lead to Pulling Apart and Implausible Estimates On flat priors in hierarchical models, Gelman says A noninformative uniform prior on the coefficients is equivalent to a hierarchical N(0,tau^2) model with tau set to a very large value. This is a very…
Introduction Randomized experiments have a rich history, almost none of which is the concern of this post. Rather, I want to focus on a tiny slice of randomized experiment literature: CUPED (Controlled-experiment Using Pre-Experiment Data). The technique has been impactful in the online randomized experiment (a.k.a. A/B Testing) community, with many vendors – Eppo included – offering the…
Introduction Bayesian decision making with loss functions was a very popular with stakeholders and data scientists when I was working in AB testing. In short, eschew p values instead ask “If I made this decision, and I was wrong, what do I stand to lose?” and then make the decision which minimizes your expected loss. In my opinion, and from what I’ve heard from others, this is an attractive method…
Introduction Thinking back to my Introduction to Epidemiology class, confounding was one of the topics of central interest. However, it took the remainder of my PhD (and then some) to really get what confounding was . Confounding was introduced as exposure and outcome having common cause. You’ve probably seen the idea motivated with the following DAG: Here, S is a common cause of and , and we’re…