Reading · Command Line with Camille

This page did not load. You can still read it on the original site — the toolbar below keeps your place in the directory.

A framework for understanding, and governing, how confidence moves through companies in the AI era.

You’re sitting in a leadership meeting reviewing next quarter’s priorities. Marketing presents an AI-generated summary of customer interviews. Product references an AI synthesis of feature requests. Finance has used AI to model several investment scenarios.

The conversation is productive. People challenge recommendations, debate tradeoffs, and discuss where the company should invest next.

Yet one question never comes up: how much of what everyone now accepts as fact has actually been verified?

No one intentionally skips that step. Marketing assumes Product already checked its numbers. Product assumes Finance validated its model. Everyone in the room assumes someone, somewhere, did the work of confirming that the AI got it right, and the meeting moves forward on the strength of that assumption. Nothing about the moment feels risky. In any single instance, it usually isn’t.

Increasingly, though, that assumption has become a defining feature of organizational decision-making.

This isn’t a story about AI making mistakes. It’s a story about what happens after AI enters organizational workflows: about how an output produced by one system becomes an input trusted by the next person, the next team, the next system down the line.

Organizations have always run on delegation. That is what distinguishes them from individuals: an executive trusts a director’s recommendation without re-running the analysis, a director trusts an analyst’s numbers without re-deriving them, and a large group of people accomplishes things no one person could do alone. For most of business history, that chain held because somewhere along it, a person had actually done the verifying.

AI changes what sits at the start of the chain. Increasingly, the first link isn’t a person who checked something. It’s a model that generated something, and everyone downstream keeps behaving as though the checking still happened, because that is what the chain has always meant. Whether real verification took place, and how much, is no longer something anyone in the chain can easily see.

Consider a strategy team preparing a board presentation. Marketing asks AI to summarize customer research. Finance uses AI to identify spending trends. Product relies on AI to synthesize user interviews. None of the individual outputs are obviously wrong, and by the time the board convenes, the discussion centers on strategic tradeoffs rather than the quality of the evidence beneath them. The organization hasn’t become less analytical. It has quietly moved the point at which analysis stops, and moved it earlier than anyone decided on purpose.

I think of this as confidence propagation: the process by which confidence spreads through people, workflows, and AI systems faster than the evidence that originally justified it.

Unlike trust, which implies a conscious decision, confidence propagation often happens invisibly. Every participant assumes someone earlier in the chain performed the necessary verification. Sometimes they did. Sometimes they didn’t, and there is rarely a clean way to tell which.

Confidence propagation isn’t separate from trust so much as parasitic on it. It borrows trust’s authority while stripping out the part that normally keeps trust in check: evaluation, and some cost if the evaluation turns out to be wrong.

I trust a colleague because I’ve watched them be right before, and if they’re wrong this time, there’s a real cost to them and to the relationship. That cost is what keeps trust bounded. Once that colleague passes along an AI-generated claim, what I inherit is the social signal of their trustworthiness, detached from any evaluation either of us actually performed.

That detachment breaks something trust is supposed to do by design: decay. In a normal chain of transitive trust, confidence is supposed to discount at every hop: if I trust you at ninety percent and you trust the source at eighty, what I should extend to the source is somewhat less than either number, not more. Confidence propagation frequently runs the opposite direction. Each person who accepts a claim without challenging it becomes, to the next person, evidence that the claim already survived scrutiny, so confidence can flatten or even compound instead of decaying. Nobody re-derives the number. They simply notice that other capable people didn’t object, and treat the absence of objection as confirmation.

None of this happens by accident, either. Organizations reward speed, synthesis, and decisiveness. Few reward reopening a question that already looks settled, especially once capable colleagues have accepted the answer. AI amplifies that incentive by making high-quality summaries, recommendations, and analyses available almost instantly. Confidence ends up traveling farther than evidence not because people are negligent, but because the organization quietly rewards moving forward over looking back.

The pattern is already visible across industries, and none of it requires the AI to be exotic or the failure to be dramatic. In healthcare, a man was hospitalized for three weeks in 2025 after ChatGPT suggested he replace dietary table salt with sodium bromide without flagging the danger, a case later documented in Annals of Internal Medicine. In law, a federal judge removed three attorneys from an Alabama prison case in 2025 after their filings relied on fabricated citations that ChatGPT had produced and nobody downstream had checked. The following spring, Sullivan & Cromwell, one of the most respected firms in the country, acknowledged a similar failure had reached a court filing despite its own internal review. In professional services, Deloitte Australia quietly reissued and partially refunded a roughly $290,000 government report after a researcher found it full of fabricated academic citations and an invented court quote. In each case, the underlying error was ordinary. What made it expensive was how far it traveled before anyone looked twice.

Blame here doesn’t belong to the tools. AI probably improved productivity at every one of these organizations most of the time. The trouble starts when confidence travels farther than the evidence that produced it: when a summary quietly becomes a diagnosis, a draft becomes a filing, a report becomes a policy input, and nobody goes back to check the ground any of it stands on. The examples above all show the same failure playing out in a different register: a report accepted on professional judgment, a citation accepted on expert authority. The two that follow show it playing out somewhere else entirely.

Earlier this year, the startup MeetingTV sued Palo Alto Networks and Koi Security, alleging that an AI-assisted threat intelligence report incorrectly linked the company to a Chinese cyber espionage campaign. The case is still being litigated, and the allegations haven’t been proven. But the allegations themselves illustrate how confidence can spread through an ecosystem. According to the complaint, the report was treated as authoritative by downstream security products, and by the organizations that relied on them, before its conclusions had been independently challenged. If the allegations prove true, the damage came less from the original report than from the speed with which others accepted and acted on it.

Every example so far involves a human chain: someone copied, cited, or filed an AI output that turned out to be false, and no one downstream re-checked it.

Confidence doesn’t only move through organizations. Increasingly, it moves through software.

Zillow’s algorithmic home-buying unit, Zillow Offers, wasn’t built on hallucinated data, and nothing about it required a person to pass along a bad citation. Its pricing model was a reasonably competent estimator, refined over years against real comparable sales, and it worked well enough under ordinary market conditions. What failed was the extension: Zillow scaled the model into an autonomous, high-speed home-buying operation that competed on the speed of its offers, just as the 2021 pandemic housing market moved into a volatility the model had never been tested against. Nobody reinserted a human check before committing real capital to buy thousands of homes at the model’s price. The company ended up overpaying broadly, took a $528 million loss in a single quarter, and shut the unit down within the year. Nothing about the algorithm was wrong when it was built. It was simply trusted well past the conditions that had earned that trust, at machine speed, with no human step along the way where anyone could have paused it.

None of these cases suggest organizations should avoid AI. Together, they suggest something more specific: most organizations don’t yet have a reliable way to tell the difference between confidence that’s still backed by evidence and confidence that’s simply been carried a long way from it, whether the thing carrying it is a person or a pipeline.

None of this is entirely new, even if the speed is. Security architects have wrestled with a narrower version of the same problem for years under the name transitive trust: the fact that trusting one credential often means inheriting trust in everything that credential can reach. Economists studying financial markets describe a related pattern as an information cascade, where people rationally defer to the apparent judgment of everyone who acted before them, even against their own evidence. Security professionals in particular have spent decades studying how trust moves through organizations, and the parallel to what MeetingTV and Zillow both illustrate is hard to miss, even though nothing here requires an attacker. Attackers rarely go straight for their real target. They compromise one credential, inherit whatever permissions come with it, move through relationships the system already trusts, and gradually reach assets far more valuable than where they started. Each step looks unremarkable on its own; the danger lives in the path, not in any single hop.

Confidence now moves in a remarkably similar way, minus the adversary, and considerably faster than trust ever did. Trust used to propagate at the pace of memos and phone calls, through a handful of people who had at least met each other. Confidence now propagates at the pace of an API call, through far more of an organization than trust ever reached, and often without any person in the loop to notice it happening. Once accepted, it travels from workflow to workflow, team to team, and increasingly from one AI system to another, picking up the appearance of certainty at each stop without anyone adding new evidence along the way. No one is deliberately steering it there. Insider threat programs have spent years focused on the misuse of legitimate access. AI is introducing a parallel problem, with no malicious actor required: the unintended expansion of legitimate confidence well beyond the conditions under which it was originally earned.

Much of today’s conversation about AI focuses on hallucinations, and that remains an important technical problem. But a more enduring organizational question is how AI changes the people who rely on it. Humans adapt quickly to tools that are fast, responsive, and usually correct: we stop checking routine work, we intervene less often, and we save our attention for the exceptions obvious enough to catch our eye. Psychologists call this automation bias: the tendency to favor an automated system’s recommendation even when contradictory evidence is sitting right there. A 2025 study of cybersecurity professionals found automation bias shaping judgment in nearly half of respondents, in a field built specifically around adversarial skepticism, as research published in Digital Threats: Research and Practice found. Aviation and healthcare have spent decades designing around this same reality, because even seasoned professionals gradually defer to systems that are reliable often enough. AI is now bringing that dynamic into every organization at once.

It’s worth being precise about what confidence propagation adds to that picture, because the two ideas are easy to conflate. Automation bias explains why one person, in one moment, defers to a system’s answer instead of checking it. Confidence propagation describes what happens next: that single deferral becomes institutional behavior, carried forward by the next person’s inbox, the next team’s dashboard, the next AI system’s prompt, long after the original evidence, and the person who might have questioned it, has left the room. Automation bias is a property of individual judgment. Confidence propagation is a property of how organizations move information, and it can keep spreading long after any one person’s bias has stopped mattering.

People don’t become careless. They become efficient.

And efficiency quietly redirects attention away from verification, because the system usually deserves their confidence. Usually. The gap between “usually” and “always” is where a great deal of organizational risk now lives, not because people trust AI, but because they trust the previous person who trusted the AI, one link removed from the thing that actually happened.

None of this argues for distrusting AI, and it isn’t a call to personally re-verify everything, which would be exhausting and beside the point. It’s a case for becoming intentional about where you still look twice. The more routine a task feels, the more tempting it becomes to stop checking it, and those routine moments are exactly where professional judgment gets quietly trained, or quietly allowed to atrophy. Deciding which outputs deserve a second look, and which genuinely don’t, may turn out to be one of the more valuable professional skills of the next decade, in plenty of roles that have nothing to do with AI itself.

Governance discussions tend to focus on controlling the technology itself. Increasingly, governance also has to protect the human capabilities organizations still depend on, which means asking a different set of questions. Which decisions require fresh evidence every time, no matter how many good recommendations preceded them? Where should independent verification stay mandatory, even when it slows things down? How should an AI system’s authority expand as it demonstrates reliability, and how should that expansion be reversed if the reliability turns out to be an illusion? What human capabilities are worth deliberately preserving, even in places where AI could technically do the job?

None of this requires starting from a blank page. The NIST AI Risk Management Framework and ISO/IEC 42001 already provide valuable scaffolding for these questions. The harder work is translating those principles into day-to-day operational decisions: calibrating human oversight to how reversible a decision is, how large its blast radius would be if the confidence behind it turns out to be misplaced, and how fresh the underlying evidence still is. Decisions that score high on all three deserve what might be called a confidence checkpoint: a deliberate point in the workflow where authority pauses for a fresh look, not because the AI is untrustworthy, but because the decision has earned one.

Governance, in the end, lives in decisions about delegated authority more than in policies alone. Every time an organization lets an AI system recommend, prioritize, summarize, approve, or act on its behalf, it is deciding how much authority to delegate, under what conditions that authority can be exercised, what visibility humans retain into those decisions, and where intervention remains possible. Confidence becomes dangerous exactly when those boundaries go invisible.

Some of the resulting work is technical: distinguishing AI-generated claims from independently verified findings as they move through documents, workflows, and systems, so the evidence lineage survives each handoff instead of disappearing with the first copy-paste. Verifying confidence, in other words, means maintaining a visible connection between a claim and the evidence that first justified it, and noticing the moment that connection weakens or breaks. Some of the work is procedural: identifying categories of decisions (legal filings, security blocks, clinical recommendations, capital commitments, and other high-impact actions) that always require fresh, independent review before authority advances, regardless of how reliable the last ten recommendations were. And some of it is cultural: recognizing the analyst, engineer, attorney, clinician, or associate who pauses to verify something everyone else assumed had already been checked as someone strengthening the organization, not slowing it down. Where AI is used well, judgment stays a capability worth cultivating, not a bottleneck to eliminate.

Seen this way, governance is less about controlling technology than about preserving the relationship between authority, confidence, and evidence as decisions move through an organization. Authority determines what people and systems are permitted to do. Confidence determines whether everyone downstream keeps believing those actions are justified. Sustainable AI governance means managing both, on purpose.

Organizations have spent decades learning how to verify identities, authenticate users, and secure transactions: to trust nothing by default and verify explicitly before granting access. The next challenge is subtler, but it runs on the same instinct, aimed now at AI’s outputs rather than just its access. That doesn’t mean re-verifying everything, which would grind the organization to a halt and erase most of what makes AI worth using. It means understanding how confidence moves through their institutions, where verification quietly disappears along the way, and which decisions continue to deserve fresh evidence regardless of how reliable yesterday’s answer happened to be: verifying explicitly at the handoffs with the most at stake, and letting everything else move at the speed AI made possible in the first place.

AI will keep becoming more capable. That much is almost certainly inevitable. Whether organizations become equally disciplined about managing confidence is not.

Why Your Organization Trusts AI More Than It Realizes

Comments