Because it’s what the cool kids are doing.
WordPress is too sluggish: it took >20 seconds to get this editor window to load (slow Chinese internet, but still). Since Substack is now also doing RSS & all the other stuff, I think it’s a good platform and I am moving there.
Heavy on Twitter this week
Good Tweets
Fresh off the press in Nature Communications, an update on the mOTUs (marker gene-based operational taxonomic units) concept.
This concept was introduced in the first mOTUs paper, which itself built upon the specI concept:
First of all, it updates the database with newer data, which is valuable for the tool, but this version includes a few conceptual improvements as well:
You can get the tool from https://motu-tool.org/, through bioconda, and there is a module for running it through NGLess.
From the ongoing series “how do statistics feel to me” (previous episode)
Vaccines create adults
This is a strong slogan, but it’s trivially false and, eventually, stupid. There were plenty of adults before vaccines. Almost every kid that gets one of the diseases that we vaccinate against will survive just fine. Many other pro-vaccination slogans are equally alarmist, equally false, and (I believe) equally counter-productive.
Mind you, I agree with the idea that vaccines are good and I think they should be mandatory, due to the fact that children cannot make informed decisions and vaccination has public health consequences (it’s not just your body, your choice, it’s our herd immunity, our choice). But, even if all vaccines were to disappear, most of the time, the kids would be fine. In fact, the total mortality rate might be not more than 1-2% (deaths before the age of 5).
Without vaccines, most families would not have to explain to the older brother why their sister died, they would not have to mourn a dead child. Society would not materially change that much. There would be roughly as many adults around.
However, 1-2% of children dying would be horrible. There is no need to exaggerate to make it sound even worse: it would be bad enough. If you take your kids to kindergarten, and your kindergarten has an average size, that would be one child funeral per year (or every other year).
Without vaccines, your child would most likely be fine, the odds are in their favour (they’d have a 98% chance of making it). However, you would probably have to explain to them that someone in their school of a horrible disease at some point in their early childhood. You’d probably be saying “Good Morning” to at least one parent who lost a child on a regular basis. If you are a early-education teacher, you’d have to expect to attend at least half-a-dozen funerals in your career, mourning little children.
This is a tremendous amount of pain and grief.
Most kids would be fine, most kids were fine before vaccines. But 1-2% is way too much. We’d see people developing defense mechanisms (Detachment parenting: don’t get too close before you know they’ll make it, a New York Times best seller for the post-vaccine era). We don’t need vaccines to create adults, but a world without them is a significantly worse world than the one we live in today.
It’s not “if we don’t vaccinate, all children will die”: that’s flashy, but false.
Rather, it’s “when you walk into kindergarten with your toddler tomorrow, think that, without vaccines, one child in there would die every other year. It’s probably not going to be yours, but, please, vaccinate them.”
One of the nice things about blogging about my work is that I can use the blog to dig deep into a technical aspect that may would otherwise be buried in a little note in the Methods section. This is exactly such a post.
In our recent paper on the dog microbiome (see previous blog summary), we had a simple experimental design: All dogs ate the same food (Base diet) for 4 weeks. At the end of this period, we obtained a first sample from each dog. Then, dogs were randomly assigned to one of two diets (HPLC or LPHC) for 4 weeks, and we obtained a second sample.
Thus, for each dog, we had two samples: one baseline sample and one post-intervention sample. We can use tools like MOCAT2 or NGLess to obtain taxonomic (or functional) profiles of each of these samples (this project was done when we were still using MOCAT2; nowadays, we would use NGLess, but I digress). Half of these samples are from the HPLC diet, the other half from LPHC.
Now, the question is: how do we detect features (taxa) that are differentially affected by diet?
Attempt #1: Just use the endpoint values
The simplest way is to ignore the first (baseline) timepoint and test the effects on the second (post-intervention) sample. For example, we can use the Wilcoxon test and compare relative abundances (there are other methods too, but Mann-Whitney/Wilcoxon is still great).
In pseudo-code, this looks like:
data = data[timepoint == 'post-intervention'] for g in data.columns: data = data[g] print(stats.mannwhitneyu(g[diet == 'LPHC'], g[diet == 'HPLC'])
Attempt #2: Test on the ratios
Still, intuitively, it should be more powerful (in the technical sense of statistical power) to use the baseline samples in addition to the just the end point. Thus, let’s do a simple transformation: for each dog and each feature, we compute the difference in abundance between the initial timepoint and the end point.
normed = data[timepoint == 'post-intervention'] - data[timepoint == 'baseline']
for g in normed.columns:
g = normed[g]
print(stats.mannwhitneyu(g[diet == 'LPHC'], g[diet == 'HPLC'])
Most often, in biology, we care about ratios rather than differences. We report things as “10-fold more abundant.” So, it is a good idea to convert the data to logs before running the test.
The problem is (as usual): How do you handle the zeros? For many problems, there is a trivial solution, namely adding a pseudo count:
data10 = log10(pseudo_count + data)
normed = data10[timepoint == 'post-intervention'] - \
data10[timepoint == 'baseline']
However, this does not work well in our dataset! We observed a loss of power (many fewer taxa were considered differentially abundant) and a little bit of investigation and thought revealed the issue:
Consider a subject where the abundance of a feature goes from 10⁻² to 10⁻⁴ when comparing baseline to post-intervention. On this subject, the intervention had an effect of -2 log-units. We can say that this was a more negative effect than a subject where the effect was 10⁻⁴ to 10⁻⁵. What about a subject where the abundance of that feature went from 10⁻⁴ to undetected? Well, if your pseudo-count is 10⁻⁶, then you would be claiming that the effect was -1 log unit, whilst if you used a lower pseudo-count, you would be claiming a larger effect.
Here is a visual representation of the effect (this is real data, but let’s abstract from the details by just saying that it’s Diet 1 and Diet 2 and we are comparing the effect on a particular taxon).
You can probably intuit that the effect of Diet 2 is much stronger. However, if you test on the differences between post and baseline, you would find a non-significant effect (after multiple hypothesis correction). Comparing just the end-point, though, gives a strong signal. The problematic profiles are coloured blue, as they involve points below the detection limit on the post-intervention time point, which we set to 10⁻⁷.
Attempt #3 (solution): Use Gehan’s test
Before introducing Gehan’s Test, let’s review how Wilcoxon works:
The Wilcoxon test can be seen in the following way: take two samples A and B and compare all of the elements of A to all of the elements of B. Every time the object in A is greater than the object in B, you count +1 (you count 0.5 for draws). Add all of these values together and you get a score, U. If A and B are drawn from the same population, then U is roughly normally distributed with the mean being roughly half the comparisons.
Now, we can easily generalize this to take into account the fact that our data is only partially observed. Now, whenever we have zero in our data, we take that into account. If we estimate our detection limit to be 10⁻⁵, then when a subject goes from 10⁻⁴ to undetected, we know that the effect was at least -1 log units, but may be even more negative. The result is that we can claim that this effect is more negative than a subject where the abundance went up (or just slightly down). However, when we compare the effect to another subject where the effect was -2 units, then the comparison is indetermined.
Thus, we can adapt Wilcoxon to work in a slightly different way. We still compare all the objects in A to all the objects in B, however, the result can now be one of three options: (1) greater-than, (2) equal-to, (3) smaller-than, (4) indeterminate. If you work through this intuition to develop a proper statistical test, you get the Gehan Test (Gehan 1965a & Gehan 1965b)! Now, the test is empirically much more powerful when you have many samples close to detection limit!
2016 made me reassess an important component of my view of the world. No, not Brexit or Trump becoming President (although, it’s not unrelated).
At the end of 2016, I realized that almost all psychology is pseudo-science. Not hyperbole, not oversold, but pseudo-science.
People used to joke that Parapsychology is the control group for science: i.e., a bunch of people ostentatiously following the scientific method in a situation where every result should come out negative. It’s a null field: the null hypothesis (that there is no effect) is true. Thus, the fact that you can still get positive effects should be worrisome. Turns out the true joke was that psychology is the true control group. Parapsychology was a bad control as most scientists were already predisposed to disbelieve them. Psychology is a much better control.
I had heard of the “Replication Crisis” before, but had not delved into the details. I thought psychology was like microbiome studies: over-hyped but, fundamentally, correct. We may see reports that the microbiome makes you be rude to your uber driver or whatever silly effect. We often read about the effects of the microbiome on obesity, as if it didn’t matter that our diets are not as healthy as they should be and it was all down to microbes. Jonathan Eisen collects these as overselling the microbiome.
Still, to say that people oversell the microbiome is not to say that there is no effect. The microbes do not single-handedly cause obesity, but they have an impact on the margin (a few BMI points up or down), which is enough to be significant for the population. They may not cause nor cure cancer, but they seem to influence the effect of immunotherapy enough that we may need to adjust dosages/drug combinations. And so on…
I thought that when it came to psychology, the same was true: sure, a lot of hype, but I thought there was a there there. There isn’t.
My basic mistake was that I had shared Daniel Kahneman’s view of the situation:
My position […] was that if a large body of evidence published in reputable journals supports an initially implausible conclusion, then scientific norms require us to believe that conclusion. Implausibility is not sufficient to justify disbelief, and belief in well-supported scientific conclusions is not optional. This position still seems reasonable to me – it is why I think people should believe in climate change.
This was exactly my position until I read this long Andrew Gelman post. Since then, I started to read up on this and find that psychology (as a field) has sold us a bill of goods.
Yesterday, the Guardian published a long essay by Tim Harford on the dangers of automation. The argument is not new (I first heard it on the econtalk episode with David Mindell), and the characteristic example is that of the Air France flight that crashed in the middle of the ocean after the autopilot handed control back to the human pilots who immediately proceeded to crash the plane. As I read it the argument runs as follows: (a) full automation is impossible, (b) partial automation erodes skills, therefore (c) we should be wary of over-automating.
On twitter, I responded with the snark that that medium encourages:
tl;dr An Air France crew screwed up massively one time and killed everybody on board. Therefore, computers are bad https://t.co/qF7HQ5IxxU
— Luis Pedro Coelho (@luispedrocoelho) October 11, 2016
But I feel I should make a longer counter-argument.
1. Despite being a good visual (a plane crash is dramatic), the example of an airplane crash in 2009 is a terrible one. Commercial civil aviation is incredibly safe. Commercial aviation is so safe, I wouldn’t be surprised to read a contrarian Marginal Revolution post arguing it’s now too safe and we need more dangerous planes. I would be very careful in arguing that somehow whatever the aviation community does, is not working based on a single incident that happened 7 years ago. If this was happening every 7 weeks, then it would be a very worrying problem, but it doesn’t.
2. Everything I’ve heard and read about that Air France accident seems to agree that the pilots were deeply incompetent. I have also gotten the distinct impression that if the system had not handed back control to the humans, they would not have crashed the plane. It is simply asserted that we cannot have completely autonomous planes, but without evidence. Perhaps at the very least, it should be harder for the humans to override the automated control. Fully automated planes would also not be hijackable in a 9/11 way nor by their own pilots committing suicide (which given how safe planes are, may now be a significant fraction of airplane deaths!).
3. Even granting the premise of the article, that (a) full automation is impossible and (b) partial automation can lead to skill erosion, the conclusion that “the database and the algorithm, like the autopilot, should be there to support human decision-making” is a non sequitor. It assumes that the human is always a better decision maker, which is completely unproven. In fact, I rather feel that the conclusion is the opposite: the pilot should be there (if a pilot is needed, but let’s grant that) to support the autopilot. Now, we should ask: what’s the best way for pilots to support automated systems? If it is to intervene in times of rare crisis, then pilots should perhaps train like other professionals who are there for crises: a lot of simulations and war games for the cases that we hope never happen. Perhaps, we’ll get to a world where success is measured by having pilots spend their whole careers without ever flying a plane, much like a Secret Service agent trains for the worst, but hopes to never have to throw themselves in front of a bullet.
4. Throughout the essay, it is taken as a given that humans are better and computers are there to save on effort. There is another example, that of meteorologists who now trust the computer instead of being able to intuit when the computer has screwed up, which is what used to happen, but I don’t see an argument that their intuition is better than the computer. If you tell me that the veteran meteorologists can beat the modern systems, I’ll buy that, but I would also think that maybe it’s because the veteran meteorologists were working when the automated systems weren’t as good as the modern ones.
5. The essay as a whole needs to be more quantitative. Even if computers do cause different types of accident, we need to have at least an estimate of whether the number of deaths is larger or smaller than using other systems (humans). I understand that authors do not always choose their titles, but I wouldn’t have responded if title of the essay had been “It won’t be perfect: how automated systems will still have accidents”.
6. The skill erosion effect is interesting per se and there is some value in discussing it and being aware of it. However, I see no evidence that it completely erases the gains from automation (rather than being a small “tax” or clawback on the benefits of automation) and that the solution involves less automation rather than either more automation or a different kind of human training.
7. My horse riding skills are awful.
Meta Rabbit 