AI And The Merchants of Doubt

I explore how the historical manufacture of scientific doubt used by the tobacco and climate industries has evolved into a modern bureaucratization of risk within artificial intelligence development. I argue for an adult politics that distinguishes between corrupted doubt intended to delay oversight and useful doubt that establishes clear lines of institutional responsibility.

In June, 2010, Naomi Oreskes and Erik M. Conway published Merchants of Doubt. The book followed a group of prominent scientists, several with Cold War credentials, who intervened in public disputes over tobacco smoke, acid rain, the ozone layer, DDT (a synthetic insecticide), and climate change. Their method was not always to prove the opposite case. It was often enough to prevent the existing case from becoming usable. The approach was simple. Ask for another study. Question the sample. Call the models uncertain. Praise better evidence as a civic duty. Describe regulation as premature. Hint that the experts are not of one mind. Almost any phrasing will do, provided it buys time.

“‘Doubt is our product,’ ran the infamous memo written by one tobacco industry executive in 1969, ‘since it is the best means of competing with the ‘body of fact’ that exists in the minds of the general public.’”

Time was the product.

That is the part of the book I cannot stop thinking about now. Not the villainy, though there is plenty of that. Not the think tanks, the memoranda, the expert letters, the solemn men talking about scientific freedom while lung cancer moved through the population, in other words general society who would not make the time or effort to read the small print and footnotes. What stays with me is the discovery that doubt can be manufactured with the same discipline as steel. It can be funded, distributed, franchised, laundered through respectable institutions, and placed in front of the public with an expression of injured seriousness.

Artificial intelligence has brought the word back into circulation, but not in the old form. That is why the analogy is tempting and treacherous. The tobacco companies sold doubt after enough was already known. The AI argument is happening while the machine is still under construction, still being tuned, still being praised by investors, still being feared by its own inventors, still being handed to children doing homework, doctors checking notes, soldiers reading briefings, and people at kitchen tables who have begun to treat a machine’s patience as a kind of kindness. No one is standing over a deceased person. The decision, or lack of one, is being made while the thing itself is still changing.

So I read Oreskes and Conway with a pencil in one hand and a certain unease in the other. Geoffrey Hinton warns of catastrophic risk. Yoshua Bengio calls for stronger governance, independent evaluation, and liability. Max Tegmark asks whether we are building powers we cannot control. Dario Amodei, while running one of the companies pushing the field forward, writes that the risks may be the only obstacle between us and a future of extraordinary human benefit. Demis Hassabis, meanwhile, can point not to a promise but to a result: AlphaFold, protein structures, biology accelerated, the long patience of science suddenly assisted by a machine that did not need a grant extension, a sabbatical, or a second cup of coffee.

This is not a clean courtroom. It is not the cigarette executive versus the epidemiologist. It is stranger than that, and more morally awkward. The same field has produced the warning and the cure, the alarm bell and the instrument, the engineer who says slow down and the scientist who says look what we can now do. The age is not asking us to choose between reason and delusion. That would be easy, and public life no longer grants such luxuries. It is asking us to distinguish between different kinds of doubt before the consequences have finished arriving.

The scientists in Merchants of Doubt are a useful comparison. One can look at Hinton, Bengio, Tegmark, and the other AI pessimists and say: there they are, the new obstructionists, the men of delay, the professional worriers standing in the doorway of abundance. The accusation has an appealing briskness. It makes the accelerator feel brave. It allows the product launch to stand in for moral seriousness. It turns caution into a personality defect. One can almost hear the impatient AI executive saying, with the smoothness of a man who has never had to answer for a downstream consequence, that humanity cannot be allowed to miss the future because a few academics have become gloomy.

I do not buy it.

Hinton is not a retired village scold objecting to the railway because the horses will be embarrassed. Bengio is not defending candlelight against electricity. These are not men who woke one morning and discovered, to their horror, that young people were using tools. They built the intellectual foundation on which the present industry stands. Their warnings come with the authority, and the discomfort, of proximity. It is one thing to be denounced by an enemy. It is quite another to hear the alarm from the person who helped design the product.

Hinton’s numbers have become famous because they are so resistant to ordinary mental filing. A ten to twenty percent chance of catastrophic outcomes. What is one to do with such a figure? Too low to cancel the build too high to place beside the canapés. If a surgeon gave those odds before an elective procedure, the patient would not ask whether the operating theatre had a strong innovation culture. He would put his trousers back on. Yet in AI we have learned to discuss such probabilities with the poise of men comparing airport lounges.

Bengio’s case is less theatrical and for that reason more useful. He asks for evaluations, liability, public interest research, and governance that can survive contact with firms whose incentives are not identical to civilization’s. There is no melodrama in this. It is almost dull. That is its strength. He is not asking us to worship fear. He is asking us to give responsibility an address.

Tegmark presses the harder question, the one that makes practical people suddenly reach for procedural language. If systems become more capable than their makers in domains that affect power, biology, persuasion, cyber operations, or autonomous planning, what exactly will keep them within human purposes? Not vibes. Not brand values. Not a launch blog with soft lighting. Not a safety card written by the same institution whose valuation depends on being first. The question may be overstated in some tellings, but it is not stupid. The future has an unfortunate habit of punishing people who confuse discomfort with irrelevance.

And then comes Hassabis, ruining the simplicity of the indictment.

AlphaFold is not a marketing adjective. It is one of the great scientific achievements of the last decade. To predict protein structures at scale is not to add a feature to a subscription plan. It is to solve a hard scientific problem in public, with results other scientists can use. The 2024 Nobel recognition of Hassabis and John Jumper, alongside David Baker, made the point harder to evade. AI was not only producing bogus citations and helping consultants rename their slide decks. It was helping science do science.

The old anti-technology reflex becomes unserious. I have no patience for the easy sneer that treats every claim of AI benefit as public relations. Some of it is. Much of it is. Perhaps an indecent amount of it is. But not all. When a protein structure prediction changes what a laboratory can attempt next week, the promise has left the brochure. It has entered someone’s experiment. That does not make the surrounding industry virtuous. It makes the governance problem harder, because the benefit is no longer imaginary.

Shakespeare was the master writer on doubt and uncertainty. His tragedies are full of people trapped between partial knowledge and irreversible action. Hamlet knows enough to be sickened, not enough to be free. Macbeth receives a forecast and mistakes it for authorization. Othello demands certainty and accepts counterfeit certainty from Iago, a man who understands that suggestion can do the work of proof. Lear asks for love in the wrong format and discovers, rather late, that metrics are not always measurement. One can imagine the old king in a board meeting, demanding sentiment scores from Cordelia.

AI places us in this Shakespearean interval. We know enough to be uneasy. We do not know enough to be finished. The systems work, but not always reliably. They reason, or appear to reason, or perform something near enough to reasoning to make the difference unclear. They help programmers and mislead auditors. They summarize documents and invent sources with the same helpful posture. A chemist can use them. So can a cyber attacker. A lonely person may feel comforted by one, which is not the same as being cared for. A manager may call them productivity tools while using them to watch workers more closely. The difficulty is not that the claims are false. The difficulty is that so many of them are true in opposite directions.

The old merchants of doubt understood this condition perfectly. They knew that democracy needs argument, and they turned that need into a delaying tactic. They did not have to persuade the public that smoking was healthy. They merely had to persuade enough people that the question was not ready for decision. Delay became policy. Uncertainty became profit. Scientific modesty became a hiding place for commercial power.

The line I would draw now: useful doubt assigns responsibility; corrupted doubt disperses it. Useful doubt asks for an evidentiary threshold, a testing authority, a veto, and a name on the invoice when the system fails. Corrupted doubt avoids such particulars. It multiplies venues until the original question loses force.

That is the danger now, but it comes by a less obvious path. The danger is not simply that AI companies will deny risk. Some will do that. Some will do it with impressive sincerity, which is always the most efficient kind. The deeper danger is that uncertainty itself will be administered by those with the strongest interest in speed. The public will be told, usually in patient tones, that judgment requires access it does not have. The model is complicated. The weights are private. The race is international against China. The opportunity is large. The documentation is coming. The independent audit is being discussed. Nothing in that sequence is necessarily false. That is precisely the problem. This is not the denial of science. It is the capture of uncertainty.

Captured uncertainty is harder to confront because it never presents itself as evasion. It arrives as prudence, process, further review. Captured uncertainty keeps changing rooms. The old merchants practiced external doubt. They sent confusion outward, toward the citizen, the voter, the journalist, the legislator. The new form is more intimate. It turns uncertainty into procedure. The risk is not denied. It is routed.

A question that begins in public soon acquires a room, then a review process, then an advisory board, then a policy team trained in the art of grammatical evaporation. By the time anyone asks who decided, the answer has become a calendar invitation with twelve attendees and no owner. This is not the manufacture of ignorance in the old style. It is the bureaucratization of risk.

This is why the old book still bites. Oreskes and Conway were not merely writing about bad arguments. They were writing about the political uses of delay. Tobacco, acid rain, the ozone layer, climate change: in each case, the same pattern appeared. Enough doubt was kept alive to make collective action look premature. The result was not intellectual humility. It was organized postponement.

AI requires something more difficult than denunciation. It requires us to save doubt from the merchants who would exploit it and from the enthusiasts who would abolish it. Doubt is not the enemy of progress. Doubt is one of the ways progress remains answerable. A scientist without doubt is a salesman with a lab coat. A regulator without doubt is a clerk with a stamp. A founder without doubt is, depending on the funding round, either a visionary or a public hazard.

The bad doubt of the old merchants did the opposite. It removed responsibility from reach. It asked for perfect proof where practical proof already existed. It used scientific modesty as a hiding place. Today the hiding place is technical complexity. The citizen is told that the system is too complicated to judge, the model too opaque to inspect, the competitive stakes too high to disclose, the risk too specialized for democratic argument. This is not skepticism. It is anesthesia with better vocabulary.

The same anesthesia is available now in finer packaging. The public hears that alignment has improved. It hears that evaluations are under way. It hears that the company takes safety seriously, which is the sort of sentence that should make a citizen contact their Member of Parliament or Congress. Some of these claims may be true. That is what makes them useful. The most durable evasions are rarely pure falsehoods. They are truths arranged so that no one has to stop.

Dario Amodei’s phrase about risk standing between us and a deeply positive future is therefore worth taking seriously. It does not fit the cartoon. His argument is not that AI is a monster waiting to be named, nor that salvation will arrive by subscription. It is closer to the older scientific temperament: the promise is large enough to make care obligatory. In his version, optimism is not the denial of danger. It is the refusal to let danger disfigure the promise before the public has had a chance to benefit from it.

That position is more interesting than either corporate triumphalism or professional despair. Amodei writes from inside the machine room, not from a monastery. He knows that a model can help a laboratory move from guesswork to experiment faster than before. He also knows that the same kind of acceleration, once tied to money or power, has a nasty habit of turning public goods into private weapons. This is the bridge between Hinton’s alarm and Hassabis’s achievement. The serious optimist does not say, trust us. He says, build the institutions that would make trust less necessary.

Humanity has invented machines that can help fold proteins, draft legal arguments, and keep a lonely person talking long after a human friend would have gone to bed. We have then asked the public to trust safety statements issued by the institutions most eager to deploy them. The fox is no longer guarding the henhouse. The fox has published a governance framework and invited the hens to a webinar.

Still, sneering is too easy. One must not become the kind of critic who mistakes contempt for judgment. Hassabis and the AlphaFold story force a more honest discipline. The good is real. The risks are real. The institutional weakness is real. The question is whether our public systems can grow quickly enough to hold together those three facts without sacrificing one to the convenience of the others.

What would that require? Not a sermon against machines. Not a festival of new acronyms. Not another declaration of principles printed on expensive paper and forgotten before the coffee cools. Start with one unfashionable rule: no institution should be allowed to deploy a high-stakes AI system that no outside authority can meaningfully inspect. A hospital should not be asked to trust a vendor’s private reassurance when a model influences diagnosis. A court should not accept opacity as the price of efficiency. A school should not buy a tutoring system whose failures become visible only in the child.

Responsibility needs machinery. Without machinery, every ethical commitment becomes atmospheric. It hangs above the enterprise, fragrant and useless, while the product ships.

The old merchants of doubt succeeded because public institutions were slower than private incentives. That asymmetry has not disappeared. A company can release an AI model on Tuesday. A regulator may still be trying to define the relevant harm the following spring. In between those two dates, habits form. Workflows harden. Contracts renew. The system becomes ordinary before it becomes understood.

The cost often first appears as a small administrative fact. A benefits office accepts the model’s recommendation. A caseworker, busy and underpaid, does not challenge it. The rejected applicant receives an AI generated formal explanation that explains nothing. No one has lied. No one has intended cruelty. Responsibility has simply passed through too many hands to leave fingerprints.

I think that is the real terror behind the AI debate. Not that the machines will suddenly become demons. Demons are at least legible. The terror is that responsibility will become increasingly difficult to locate. The AI model suggests, the manager accepts, the vendor updates the documentation, the regulator asks for more information, and the user is said to have consented somewhere along the way. When the harm appears, everyone has participated just enough to deny authorship.

Against that, the serious doubters are not merchants. They are witnesses. They may be wrong in their estimates. But they are trying to force the future into answerable form. That is the civic value of their doubt. It does not ask us to stop thinking. It asks us to stop pretending that acceleration is a substitute for thought.

Shakespeare would have recognized the temptation. His characters are forever being offered shortcuts through uncertainty: prophecy, rumor, flattery, forged evidence, theatrical proof. The shortcut always promises relief. It says: act now, the meaning is already clear. In AI, the shortcut is inevitability. The machine will come anyway. Other countries will build it. The public will adapt. This is not an argument. It is a lullaby sung by a shareholder.

I do not want an anti-AI politics. It would be both foolish and impossible. I want an adult politics of AI, which is rarer and less photogenic. It should be capable of admiring AlphaFold without handing AI model-release policy to private firms as a form of gratitude. It should be capable of wanting scientific acceleration while asking a rude, necessary question: who is allowed to say no? Hope that cannot survive that question is not hope.

The old merchants taught us that doubt can be corrupted. The AI scientists warning us now teach something just as important: doubt can also be an act of loyalty. Loyalty to science, because science is not the same as industry. Loyalty to the public, because the public will live with consequences it did not vote on. Loyalty to the future, because a future worth having must be more than the fastest available version of itself.

So I return to Oreskes and Conway for discipline: follow doubt until it reaches either a decision or a hiding place. The necessary question is not whether the person invoking uncertainty sounds reasonable. It is whether he is preparing society to act wisely, or merely asking society to wait while the profitable thing becomes irreversible.

The better ending is not a ban or a blessing. It is a hand on the switch, and a person who knows that not every working machine is ready to be released. Somewhere, before the announcement, before the congratulatory note from the chief executive, before the word breakthrough begins its little career in the headlines, someone asks for the test again. Not because he hates the future. Because he intends to let one arrive.

Stay curious

Colin

Image: Tim Mossholder on Unsplash