Every day you see a headline about a celebrity official in the Trump administration. One day Robert F. Kennedy is promoting junk science. The next day, Pete Hegseth is retweeting Christian nationalists. But the one celebrity administrator you don’t hear much about is Dr. Oz, the peddler of dubious remedies who now heads the CMS (Centers for Medicare and Medicaid Services). What ever happened to…
With the rise of Claude Code, the prospect of widespread white collar unemployment feels increasingly real, and it seems like a bad outcome. Unemployment causes a profound reduction in life satisfaction, larger than the reduction caused by divorce and 3 times larger than that of bad health . The US alone has 25 million white collar workers. If AI displaces a significant share of them, the result…
At some level, most people know that factory farming is an indefensible way to treat animals. At the same time, meat is delicious, and most people aren’t willing to adopt a fully vegan lifestyle. If that resonates with you, I’d like to propose a new diet. For a while now, I’ve been a beef-o-tuna-tarian . As a beefotunatarian I can eat beef, tuna, and dairy, although in practice I only have beef or…
This weekend the New York Times published an essay by Emma Goldberg titled, What if Charity Shouldn’t Be Optimized? . It argued that effective altruism (EA) – which uses cost/benefit analysis to fund easily quantified causes like malaria prevention and lead abatement – might detract from “squishier” causes like museums and community groups. In the comments, Goldberg praised readers who choose to…
Last year we learned that 20 papers from Hoau-Yan Wang, an influential Alzheimer’s researcher, were marred by doctored images and other scientific misconduct . Shockingly, his research led to the development of a drug that was tested on 2,000 patients . A colleague described the situation as “embarrassing beyond words” . We are told that science is self-correcting. But what’s interesting about…
Science has a fraud problem. Highly cited research is often based on faked data , which causes other researchers to pursue false leads. In medical research, the time wasted by followup studies can delay the discovery of effective treatments for serious diseases, potentially causing millions of lives to be lost. Unfortunately, researchers who witness this type of fraud are often reluctant to speak…
[ Notebook ] In 2006, Sylvain Lesné published an influential Nature paper showing how amyloid oligomers could cause Alzheimer’s disease. With over 2,300 citations, the study was the 4th most cited paper in Alzheimer’s basic research since 2006, helping spur up to $287 million of research into the oligomer hypothesis, according to the NIH. Sixteen years later, Science reported that key images of…
[ Notebook ] The most basic question one might have about respiratory infections like Covid is when are you infectious ? Some researchers believe that people with low viral RNA counts like \(10^3\) cp/mL are meaningfully infectious\(^1\). Others believe that almost all spread occurs during the brief peak of viral count, at around \(10^9\) cp/mL. The answer will determine whether you typically stop…
Have you ever felt a little hazy on what the Air Quality Index (AQI) actually means? I have, which is why I decided to deep dive on it. What I found was a bizarre Frankensteinian metric emerging from a web of federal judges, EPA employees, interest groups, and random members of the public. The first thing to know about AQI is that it is different for every country. I’ll focus on the US version.…
When people first hear about the risk of AI running amok, they sometimes ask how so much damage could come from a piece of software that is trapped inside a computer. A good way to respond to this question is to talk about the many drug lords and gang leaders who have controlled their empires from inside prison walls. The mob boss Lucky Luciano controlled the New York Harbor from his prison cell.…
In machine learning, double descent is a surprising phenomenon where increasing the number of model parameters causes test performance to get better, then worse, and then better again. It refutes the classical overfitting finding that if you have too many parameters in your model, your test error will always keep getting worse with more parameters. For a surprisingly wide range of models and…
Across the world, thousands of labs perform research on infectious pathogens. How risky is this research? The American Biological Safety Association hosts a database of all published laboratory-acquired infections through 2016, which represents only a fraction of all infections, many of which were never published. To make it easier to see, I have plotted the data on an interactive map. Figure 1 .…
[ Code ] Major depressive episodes among teen girls have increased by 52% since 2005 ( Twenge, et al. 2020 ). Many reseachers believe social media is the primary cause ( Haidt & Twenge ). Other researchers do not believe that social media is the cause, since the correlations between social media use and well-being are thought to be small. The correlation between overall screen time and well-being…
Think of a view you hold politically. Bear with me, and ask yourself a few questions. How well do you really understand the issue? Have you thought through all of the second and third order consequences? Did you consider perverse incentives ? What will happen when your plan is played out for multiple generations ? Did you apply a discount factor? Do you know what the discount factor should be? Are…
[ Code ] Last week I published a blog post on whether infectiousness was a linear or logarithmic function of viral RNA count, as measured by PCR. The answer has significant implications for when people can return to school or work after an infection. At the start of infection, viral RNA count increases exponentially, reaching about a billion copies per mL, before dropping back down to zero. If the…
[ Code ] Here’s a plot of SARS-CoV-2 “viral load” over the course of an infection, where “viral load” is measured by PCR. At the start of infection the number of viral RNA copies is close to zero, but it quickly and exponentially shoots up to about a billion cp/mL before dropping back down again. Figure 1. Viral load over time for a typical infection, as measured by PCR in RNA copies/mL ( Kissler,…
Disclaimer: I’m not an epidemiologist, so adjust your priors accordingly. Code available here . A classic finding from epidemiology is that in a single-wave epidemic, reducing R0 lowers the number of cases. To reduce R0, most countries have adopted some combination of Non-Pharmaceutical Interventions (NPIs) like masks and testing. But in a multi-wave epidemic with waning immunity, a contrarian…
At this moment, a growing number of scientists are studying viruses that have not yet infected human populations, seeking to understand and publish information about how deadly they are. Some of these viruses are found in animal populations. Others are partially engineered by scientists, often specifically to make them more deadly or more transmissible. Scientists who conduct this type of…
Starting around 2012, teen loneliness and depression increased dramatically . A good explanation is social media , which became more ubiquitous and addictive around that time. Teens now spend an average of seven hours per day online, not including time spent doing homework. Social media is simply a massive intervention in teen culture, and no other development has so profoundly changed the way…
Covid-19 vaccinations are a triumph of science. Every major public health official in America recommends them. Despite widespread expert support for vaccination, the FDA is taking its time with vaccine approvals. While the FDA granted its first Emergency Use Authorization (EUA) in December 2020 , full approval could come as late as January or February 2022 . Matthew Yglesias and Eric Topol argue…
What are some ways to reduce the risk of autism? According to published research, one of the most powerful interventions is for mothers to take folic acid around the time of conception. A recent meta-analysis of 10 studies found that folic acid supplementation reduced the odds of autism by about 42% . A large cohort study from Norway found an odds reduction of 49% , and some studies have even…
A special thanks to Scott Cunningham and Dean Eckles for helping me clear up my own confusion and inspiring me to write this blog post, and to Scott for his excellent book , which taught me a lot of background for this. You’ve developed a new health program and you want to measure its impact. You run a randomized experiment where half of participants are allowed to get treatment, and half are not.…
Instrumental Variables estimation is one of the most popular techniques in causal inference. But it can be a little hard to understand, especially when it is presented with terms mostly familiar to economists, such as “endogeneity” and “correlated with the error term”. In this blog post, I share some visualizations that gave me better intuitions about it. The post is designed for scientists and…
Every once in a while, I try to better understand cross-entropy by skimming over some Medium posts and StackExchange answers. I always come away only half-understanding it. A major cause of confusion is that different sources use different notations and conventions. Here are some that tripped me up. p and q vs y and p In information theory, the cross-entropy for an event with \(M\) discrete…
Disclaimer: I’m not a medical professional, but I’ve been working closely with testing experts since July. Until vaccines are widely adopted, rapid antigen tests are one of our best hopes for controlling the COVID-19 pandemic. If used widely, these inexpensive tests could massively drive down infections. Antigen tests work because they can identify people who are contagious. Knowing you are…
If you run a Github Pages blog that uses MathJax for formulas, you’ve probably had some problems recently. Most likely, you pushed some changes and now your math isn’t rendering correctly. The math looks fine locally, but it’s all messed up remotely. What happened? The cause of your problem is likely that in early August 2020 Github Pages updated Kramdown to 2.3.0, which as far as I can tell does…
UPDATE: After I posted this, I came across a recent preprint from Larremore et al. that makes this point much better than I do. The preprint emphasizes that antigen tests have high sensitivity when viral load is high, corresponding to when people are contagious. See rapidtests.org for more info. If we want to reopen schools and offices, we need to do so safely. According to Nobel Prize winning…
A few months ago, I came across one of the most extraordinary papers I have ever read . In testimony before a Congressional Committee, it has been asserted that a prolonged collapse of this nation’s electrical grid—through starvation, disease, and societal collapse—could result in the death of up to 90% of the American population. According to the paper, the grid could be knocked out either by…
Anybody looking at coronavirus data right now must feel very confused. The UK has a daily case count 60 times higher than Australia. Italy has a case fatality rate 3 times higher than nearby Greece and 12 times higher than Pakistan. These heterogeneities seem massive and have the potential to teach us critical insights about the disease. But as any epidemiologist will readily acknowledge , these…
Attention mathematicians and computer scientists: I’ve got a problem for you, and I don’t know the solution. Here’s the setup: You’re at your friend’s place and you need to take a shower. The shower knob is unlabeled. One direction is hot and the other direction is cold, and you don’t know which is which. You turn it to the left. It’s cold. You wait. At what point do you switch over to the right?…
This is Part III of a three part blog post on how to optimize your sample size in A/B testing. Make sure to read Part I and Part II if you haven’t already. In Part II , we learned how before the experiment starts we can estimate \(\hat{L}\), the expected post-experiment lift, a probability weighted average of outcomes. In Part III, we’ll discuss how to estimate what is perhaps the most important…
This is Part II of a three-part blog post on how to optimize your sample size in A/B testing. Make sure to read Part I if you haven’t already. In this blog post (Part II), I describe what I think is an incredibly cool business-focused formula that quantifies how much you can benefit from increasing your sample size. It is, in short, an average of the value of all possible outcomes of the…
A special thanks to John McDonnell , who came up with the idea for this post. Thanks also to Marika Inhoff and Nelson Ray for comments on an earlier draft. If you’re a data scientist, you’ve surely encountered the question, “How big should this A/B test be?” The standard answer is to do a power analysis, typically aiming for 80% power at \(\alpha\)=5%. But if you think about it, this advice is…
What do \(R^2\), laboratory error analysis, ensemble learning, meta-analysis, and financial portfolio risk all have in common? The answer is that they all depend on a fundamental principle of statistics that is not as widely known as it should be. Once this principle is understood, a lot of stuff starts to make more sense. Here’s a sneak peek at what the principle is. \[\sigma_{p}^{2} =…
One of coolest things I ever learned about sensory physiology is how the auditory system is able to locate sounds. To determine whether sound is coming from the right or left, the brain uses inter-ear differences in amplitude and timing. As shown in the figure below, if the sound is louder in the right ear compared to the left ear, it’s probably coming from the right side. The smaller that…
When I was in grad school I occasionally overheard people talk about how humans do something called “hyperbolic discounting”. Apparently, hyperbolic discounting was considered irrational under standard economic theory. I recently decided to learn what hyperbolic discounting was all about, so I set out to write this blog post. I have to admit that hyperbolic discounting has been pretty hard for me…
I recently came across a magazine that helps pastors manage the financial and operational challenges of church management. The magazine is called Church Executive . Readers concerned about seasonal effects on tithing can learn how to “ sustain generosity ” during the weaker summer months. Technology like push notifications and text messages is encouraged as a way to remind people to tithe. There…
This isn’t news to anybody, but the internet is changing our culture. Recently, I’ve been thinking about how it has changed our moral culture, and I realized that most of our beliefs on this topic are weirdly in tension with one another. Below are three questions that I feel are very much unresolved. I don’t have any good answers to them, and so I think they might be good topics for social science…
Keyboard shortcuts are interesting. Even though I know they are almost always worth learning, I often find myself shying away from the uncomfortable task of actually learning them. But after years of clumsily reaching for the mouse while my colleagues looked at me with a kindly sense of pity, I have slowly accumulated enough keyboard tricks that I’d like to share them. This set is probably far…
I recently came across Artir ’s Pyramid of Economic Insight and Virtue . It’s not actually a pyramid, but is instead a riff on the Expanding Brain meme. Check it out : What’s interesting about Artir’s Pyramid is that at every step, the position flip-flops from the previous step. This isn’t just a dialogue between two sides. It is a description of the belief sequence that people traverse as they…
Here’s a data problem I encounter all the time . Let’s say I’m running a website where users can submit movie ratings on a continuous 1-10 scale. For the sake of argument, let’s say that the users who rate each movie are an unbiased random sample from the population of users. I’d like to compute the average rating for each movie so that I can create a ranked list of the best movies. Take a look at…
As a data scientist, a big part of my job involves picking metrics to optimize and thinking about how to do things as efficiently as possible. With these types of questions on my mind, I recently discovered a totally fascinating book about about economic problems in the USSR and the team of data-driven economists and computer scientists who wanted to solve them. The book is called Red Plenty .…
Last week on Marginal Revolution, there was a link to a wonderful paper comparing the policy opinions of economic experts to those of the general public. The paper , by Paola Sapienza and Luigi Zingales , found some pretty significant discrepancies between the two groups. The authors attributed this difference to the degree of trust each group put in the implicit assumptions embedded into the…
One of the great developments in product design has been the adoption of A/B testing. Instead of just guessing what is best for your customers, you can offer a product variant to a subset of customers and measure how well it works. While undeniably useful, A/B testing is sometimes said to encourage too much “hill climbing”, an incremental and short-sighted style of product development that…
Jupyter notebooks are great because they allow you to easily present interactive figures. In addition, these notebooks include the figures and code in a single file, making it easy for others to reproduce your results. Sometimes though, you may want to present a cleaner report to an audience who may not care about the code. This blog post shows how to make code visibility optional, and how to…
Welcome to the new location for The File Drawer! This blog is now hosted on Github Pages and powered by Jekyll . My old blog at filedrawer.wordpress.com will be shutting down soon. I actually really liked WordPress, but I wanted to have a little bit more control over what I can put in my posts. In particular, I wanted to be able to insert my own JavaScript animations, for example in this post on…
Some videos on the internet are so good that I’ve watched them twice. Below is a list of 10 of my favorite interviews and dialogues. Obviously this isn’t an endorsement of all the positions taken. I just think they are very well done and fun to watch. The last four are best watched on 1.4x speed. 1971 Michael Parkinson interviews Muhammad Ali. 1974 Michael Parkinson interviews Muhammad Ali again,…
Everywhere you look, people are optimizing bad metrics. Sometimes people optimize metrics that aren’t in their self interest, like when startups focus entirely on signup counts while forgetting about retention rates. In other cases, people optimize metrics that serve their immediate short term interest but which are bad for social welfare, like when California corrections officers lobby for longer…
Like most people who have analyzed data using frequentist statistics, I have often found myself staring at error bars and trying to guess whether my results are significant. When comparing two independent sample means, this practice is confusing and difficult. The conventions that we use for testing differences between sample means are not aligned with the conventions we use for plotting error…
I often need to jump between different directories with very deep paths, like this: $ cd some/very/deep/directory/project1 $ # do stuff in Project 1 $ cd different/very/deep/directory/project2 $ # do stuff in Project 2 While it only takes a handful of seconds to switch directories, the extra mental effort often derails my train of thought. Some solutions exist, but they all have their limitations.…