Things to look out for when running CUPED
CUPED is a great method to reduce variance in an AB test. However, there are some things to look out for when running CUPED.
Rezka Leonandya
Personal site and blog by Rezka Leonandya — data science, machine learning, statistics, causal inference, and AB testing.
Latest posts
Non-compliance issue in an AB test
Non compliance issue can make your test underpowered. Here is how to address it.
Notes on doing AB test properly
AB testing is a common method used in tech companies to evaluate the effectiveness of different strategies. In this post I try to go over the concept and visualize it to make it easier to remember.
How to interpret confidence intervals (correctly)
Notes on how to interpret confidence intervals.
Lexical match in a search engine retrieval
Have you ever wondered how search engine works? How can it match a query with millions of documents super fast and return them to the users?
Target rate for class imbalance
Class imbalance is a common problem in machine learning, where the negative class greatly outnumbers the positive class (or vice versa). I recently watch a talk from stripe where they share their techniques in addressing class imbalance in a credit card fraud detection system. I decided to create a summary here and try it out for myself on a credit card fraud public dataset.
Uncertainty in the parameters of linear regression
I recently found out that linear regression assumed that the output variable comes from normal distribution, which consequently turns the coefficient to have probabilistic interpretation. I was confused at first because I was taught linear regression from a machine learning model perspective: input some features and linear regression will fit a line that minimizes a certain cost function such as…
Hypothesis testing with binomial distribution in an AB test
In the previous post we talked about how does frequentist hypothesis testing work in an AB test using a normal distribution. In this post we are going to look at the hypothesis testing if your variable of interest is binary using a binomial distribution. Let’s get started!
Hypothesis testing with normal distribution in an AB test
AB test is one of the integral parts that a data scientist need to master. One of the goals of doing AB test is to better inform the team to help them make decision. How can we do that? In a hypothetical scenario, let’s say that we have done an AB test and we have gathered two experiment data: one for variant A (baseline) and one for variant B. One of the things that we had to consider is how to…
Causal model with bayesian network
In this previous post, I wrote about the language of causal inference via counterfactuals. I briefly mentioned that there are at least two ways to capture the causal effect: counterfactuals and structural causal model. In this post, we are going to look at structural causal model and how we can use it to simulate what would happen if we do an intervention to a particular variable of interest.…