Engineers vs RCTs

Following my recent article on an RCT of air filters in care homes which failed to demonstrate a meaningful reduction in infections, I read an opposing viewpoint from a scientist from a chemical engineering background1.

The key point was that RCTs are meaningless for these technologies, and what really matters is measuring levels of infectious pathogens in the air in the rooms around the residents.

I have seen similar viewpoints echoed from several people among the engineering community stating RCTs are the wrong way of assessing technology such as air filters. There are essentially two approaches here, and they are viewing it through the wrong lens. Let me address some specific points.

This was the main point of the first essay. It is argued that the appropriate outcome for such a study is simply to measure how much infectious material was in the room where the filter was placed.

The cornerstone of the argument as to why the RCT was flawed, is that the filters would only be effective if the residents were forced to stay inside the room with the filter on for the duration of the experiment. Since they did not, there was no point measuring how many infections they had and the RCT was a waste of time.

This argument is completely backwards.

Measuring infectious pathogens in the air is not the thing we are actually interested in. This only matters insofar as it leads to symptomatic infections. Symptomatic infections are the primary outcome of interest, and infectious pathogens in the air are only a surrogate marker - something we would anticipate to be correlated with the thing of interest, but not the thing of interest itself. There is no point in measuring a surrogate if you can measure the main outcome of interest. In fact, this is always the wrong thing to do.

If the residents spend almost no time in the rooms, and therefore the filters have no opportunity to have any effect, this has given you your answer! There is no point putting the devices in the residents rooms. If the residents constantly turn the machine off because they are too noisy, this has given you your answer! The machine is useless because it is intolerable.

The conclusion here is precisely the opposite of what is argued. It is excellent that we did not just measure the surrogate outcome, or else we might have concluded the residents were benefitting from a measured reduction in pathogen load. As it turns out, they couldn’t tolerate having them turned on and didn’t spend any time near them, and so investing in them would not be a good idea. We only know this because we did the RCT.

Finding a negative outcome because the machines were not used correctly is a feature and not a bug of the RCT.

Human beings are complex, and when exposed to technologies can behave in unexpected (or in this case, perhaps expected) ways. When you add distance between this mechanical effect and the desired outcome, such as patient adherence or complicated biological pathways, RCTs become even more important.

The case was exactly the same for community masks. Of course we shouldn’t expect huge benefits when they have to be removed to eat and socialise, as these are the highest risk encounters. Imagine a condom that you are made to wear all the time, but for practical reasons has to be removed during sex. We would not expect good protection when the barrier has to be removed during the most high risk encounters. These types of questions cannot be answered by simply testing machines in a lab, or measuring levels of pathogens either side of a barrier.

When done correctly, a clinical trial should always measure the final clinical endpoint that we are most interested in, or at least get as close to it as possible. If the thing you really care about is the number of infections, measure those. Measuring anything else makes it a worse study. We should never care about surrogate outcomes when we can measure the thing that really matters, and whether it changes or does not, it renders the surrogate outcomes meaningless.

The first comparison made is with fridges. We do not do RCTs of fridges stopping people getting sick, we simply measure pathogen loads at different temperatures.

This argument misses that the point of fridges isn’t actually to stop people getting sick, but to make foods last longer. This is directly measurable by pathogen load; it is not a surrogate.

It would actually be very easy to do an RCT of fridges stopping people getting sick; give two groups of people an uncooked chicken for 7d, with one group in the fridge and the other out in the garden. Then force both groups to eat the chicken. We would not struggle to find a meaningful difference! The causal chain here is so short that the RCT becomes unnecessary. The outcome is inevitable. This has been shown to be far from the case in trying to implement masks or air filters.

A different counter-example was handwashing. The author points to some RCTs that were done for handwashing, and failed to find an important difference.

This doesn’t surprise me at all.

At modern levels of hygiene, handwashing almost certainly makes a trivial, if any difference to disease transmission in most settings. The scenario in which it was originally discovered to be beneficial involved doctors moving immediately from examining dead, festering corpses, to delivering babies. Enormous inoculations of highly pathogenic bacteria directly into open wounds. This is hardly the scenario we would expect to find ourselves in today.

Why then do we still recommend it? Again, there is a very short causal chain; albeit at very low risk. But importantly, it is a behavioural intervention with effectively zero cost. Almost any conceivable benefit at all would make it worthwhile. Simply making people to stop and think about cleanliness of their hands is probably worth it, because it costs nothing. It is similar with asking people to sneeze into their elbows. Does it work? Possibly. But it costs nothing, so might as well ask people to do it. This is very much unlike the cost of retroactively installing hundreds of millions of pounds worth of air filtering devices into institutions around the country.

We need to understand there are two valid approaches to using interventions like this technology.

The first is to decide there is likely a large, measurable benefit which is worth rapid investment of very large sums of money. Such a proposal requires an RCT to evaluate it, otherwise huge sums are spent on a wasted endeavour. During and post-pandemic, many advocates of clean air framed the debate in this manner, hence the RCTs were performed. I’m sure they are disappointed to see the negative results, as we all are. However, it does show that doing RCTs to evaluate such claims was the right decision.

If you want people to rapidly, and at great cost, install your technology, then they will expect effect sizes which should be easy to measure in an RCT.

The alternative, and in my opinion, correct approach, is to suggest that such technology probably provides marginal benefits which would be difficult to measure in the short term, but over many years and over large populations probably provide a worthwhile benefit.

Such claims are extremely difficult to evaluate in an RCT, but can be evaluated on the merit of existing mechanical evidence and such surrogate markers, and the decision point is different. There is no huge urgency in such a case to retroactively fit devices wherever we can squeeze them in. Instead, we look at things like reforming building codes to include more emphasis on ventilation and clean air, and provide incentives for institutions to proactively upgrade systems over time when the opportunities arise.

The well intentioned, but misguided advocates for clean air during the pandemic have perhaps had their actions backfire. Messages like, “air is the new poop”, or “clean air like clean water”, or that they could effectively end the pandemic, have set impossible expectations for these technologies. That should not lead us to believe they are worthless. Clean air for health is deserving of much more attention. We simply need to refocus our efforts on policies and methods of implementation which are befitting of the magnitude and timelines over which we would expect to see benefits.