Reading · Command Line with Camille

This page did not load. You can still read it on the original site — the toolbar below keeps your place in the directory.

Every quarter, your security team reviews which systems and vendors hold privileged access to sensitive data. The list is long, and the meeting moves quickly. When a new vendor comes up, someone asks pointed questions: what data it touches, who approved it, what happens if it’s compromised. When a system that has been in place for three years comes up, the questions get shorter. It has never caused a problem. Half the room doesn’t remember why it was granted that access in the first place. The review moves on.

Nobody in that meeting is being negligent. They’re doing something more specific, and more human: treating years of good behavior as its own kind of proof.

That instinct isn’t irrational. A system that has run cleanly for three years is, on the available evidence, probably fine. The trouble is what happens to the available evidence over those three years. The original justification for the access, whatever it was, stops getting reexamined. The people who approved it move to other roles. The documentation, if it still exists, describes a version of the system that has since been updated four times.

What’s left standing in for verification is a feeling: ‘This has always been fine.” That feeling quietly becomes a substitute for verification.

One of the reasons confidence propagates so easily through an organization is that familiarity changes how scrutiny gets allocated long before any specific claim ever shows up to be believed. Systems that perform well for long enough don’t just earn trust. They earn something more specific: exemption. The people responsible for checking them stop asking the questions that would reveal whether today’s version of the system still deserves the trust an earlier version built up.

Familiarity is persuasive for a specific reason: it creates something that feels like evidence without actually being it. Every uneventful day a system runs is experienced as another confirmation that nothing has changed, even when that day never tested the assumptions that justified trusting it in the first place. Repeated exposure gets mistaken for repeated validation, and the two are not the same thing.

None of this is a new discovery, and it’s worth saying so plainly. Sociologist Diane Vaughan named a version of it after studying how NASA kept flying the Space Shuttle despite a known flaw in the solid rocket boosters’ O-rings: each launch that didn’t explode made the next anomaly a little easier to accept, until a real defect had been normalized into routine variation, a pattern she called the normalization of deviance in her study of the Challenger disaster. Aviation human factors research documents a close cousin: automation complacency, the tendency of a pilot or operator to monitor a reliable system less and less closely simply because it keeps performing without incident, a risk the FAA’s own safety research has studied for decades. What AI changes isn’t the underlying mechanism. It’s the speed, and the sheer number of systems now earning the same kind of exemption at once.

Security has a name for the version of this problem that involves people instead of systems: insider threat. The phrase is often misunderstood as being about malicious employees, but the discipline itself has always been about something narrower and stranger. Insider threat programs exist because the people best positioned to cause serious damage are never strangers. They’re the ones who already have the access, the standing, and the years of demonstrated reliability that make anyone question them feel a little unreasonable.

Robert Hanssen shows what this looks like in people. He spied for Soviet and then Russian intelligence for twenty-two years while working as an FBI counterintelligence agent, one of the most damaging breaches in American intelligence history, and was never once polygraphed across his entire career, a gap the Justice Department’s own inspector generallater called one of the case’s clearest institutional failures. His trust was earned once, decades earlier, and simply never revisited, no matter how much evidence should have prompted a second look.

Boeing shows what it looks like in institutions. Boeing earned its exemption the same way anyone does: for decades, reliable performance persuaded the FAA to hand Boeing increasing responsibility for certifying its own aircraft. That earned delegation meant the FAA did not have a complete picture of Boeing’s own safety assessment of the MCAS flight-control system until after the first 737 MAX crash, according to a federal investigation into the FAA’s oversight. A second crash followed before the aircraft was grounded, and the FAA has only begun restoring the delegation it pulled back, in limited form, since 2025. It’s a clean illustration of a point made elsewhere in this newsletter: risk concentrates exactly where authority has quietly become unclear.

Microsoft 365 Copilot shows a faster and stranger version of the same failure in software. Hanssen and Boeing earned their exemption through years of their own demonstrated performance. Copilot never had the chance to: it was integrated across Word, Excel, PowerPoint, Outlook, and Teams largely because it arrived through an already trusted enterprise ecosystem, riding existing contracts and identity infrastructure, before anyone had fully stress-tested what it means for an assistant to treat everything in its reach as equally trustworthy context. In June 2025, researchers disclosed exactly that gap: a critical vulnerability nicknamed EchoLeak let hidden instructions buried in an ordinary email make Copilot quietly leak sensitive data the moment a user asked it an unrelated question, no click required. Its exemption from scrutiny was borrowed wholesale rather than earned, which is more alarming than Hanssen’s story, not less: a system no longer needs its own track record. It just needs a trusted enough parent, a reminder that accountability for what an autonomous system does rarely disappears just because the system was the one acting.

Bernie Madoff shows what it looks like in markets. He ran the largest Ponzi scheme in history for parts of four decades, and the SEC’s own inspector general later found the agency had received credible warnings about his returns as far back as 1992, repeatedly failing to examine his trading records properly, in part because his decades of stature on Wall Street made the possibility of fraud feel implausible to the people whose job was to check for it, as the SEC’s own inspector general report later concluded.

None of this argues that earned trust should count for nothing. A brand-new vendor and a three-year veteran of your infrastructure genuinely do carry different risk profiles, and treating them identically would be its own kind of waste. What’s missing is a decision about how much trust a track record should actually buy, for how long, and under what conditions the case for it has to be remade rather than simply renewed. Left undecided, that discount doesn’t stay fixed. It keeps growing, quietly, until nobody can say when it stopped being a judgment and became a habit.

AI compresses this timeline in a specific way. A person or an institution earns trust through duration: years of exposure to varied, ambiguous situations, observed and judged by other people over time. Duration is the whole point. It’s what makes the trust meaningful. An AI system can produce the appearance of that same track record through volume instead of duration: millions of interactions in a few weeks, each one looking, from the outside, like another data point in a long career of reliability. An organization ends up treating a high volume of recent, narrow successes as though it carries the same evidentiary weight as years of varied, observed judgment. The two are not equivalent, even when they produce an identical-looking track record on paper. That’s the mechanism behind Copilot’s story: a new AI tool typically enters an organization under close watch, then that scrutiny shrinks as its footprint grows, because the tool is converting volume into the appearance of tenure far faster than a person or a vendor ever could. By the time it’s integrated into a dozen workflows, the organization is often applying less scrutiny to a system with more reach than the one it started with, exactly when more scrutiny is warranted, not less.

Organizations rarely notice this transition happening, because no single decision creates it. One review becomes a little less detailed than the last one. One annual assessment quietly becomes biennial. One trusted system stops appearing on meeting agendas at all, because nothing about it ever seems to happen. Every individual choice feels reasonable in the moment it’s made. The cumulative effect is an organization that remembers whom it trusts long after it has forgotten why.

There’s a second thing organizations lose as this happens, beyond oversight itself: the reasoning.

Organizations don’t just inherit technology. They inherit decisions.

Every long-lived integration, every privileged access grant, every governance exception, every trusted vendor relationship reflects a judgment someone made once, under conditions that no longer exist. The person who made that judgment moves to a different role, or a different company. The documentation, if it was ever thorough, describes a version of the system three updates back. The assumptions that justified the original decision quietly stop holding. What survives is the conclusion: this is fine, this is how it’s always been done. Confidence often survives longer than the evidence behind it precisely because organizations remember conclusions far better than they remember the reasoning that produced them.

This isn’t only a security concept, and it shouldn’t be treated as one. Finance figured out a version of this problem long before cybersecurity existed, which is why so many trading floors and back offices require mandatory vacation: a trader or accountant who never takes time off is a classic fraud signal, precisely because an ongoing scheme usually requires its perpetrator’s continuous presence to keep the story straight. Separation of duties works on the same logic in a different form, making sure no single person controls a critical process end to end, not because any one person is assumed to be dishonest, but because no organization should have to rely on any one person staying trustworthy forever, unsupervised, indefinitely. Security’s version of the same idea is access recertification: reviewing access on a fixed schedule regardless of tenure, and requiring the case for it to be remade rather than simply reconfirmed.

Very few organizations apply anything like that same discipline to the trust accumulating around their AI systems, vendors, and long-standing exceptions, which is really a version of a gap described here previously: a policy can exist on paper and still control almost nothing in practice. The AI-era version of this whole family of practices might be called trust recertification, and it only takes a handful of honest questions asked on a real schedule. Would this still get approved today, from scratch? Has its scope changed since it was last reviewed? Has the environment it operates in changed? What assumptions justified the original decision, and do they still hold? Who, if anyone, has actually challenged this in the last year? None of those questions are technically difficult. What’s difficult is building the organizational discipline to keep asking them after familiarity has made them feel unnecessary.

This isn’t only a security team’s problem to solve, either. If you’re the person who has used a tool daily for two years without a hiccup, that history is exactly what makes you least likely to double-check its next output, and noticing that tendency in yourself is worth as much as any policy a security team can write.

The most dangerous systems in an organization are rarely the newest ones. They’re the ones everyone stopped thinking about, because they’d become part of the background, right up until an incident reveals the system an organization actually built instead of the one everyone assumed was still in place. Familiarity rarely changes the risk itself. It changes whether anyone keeps looking for it.

Organizations already know how to recertify identities, privileges, and access on a schedule that has nothing to do with how long someone has been trusted. The AI era needs the same discipline applied to trust itself, because familiarity is not evidence, and every system that matters eventually deserves to earn its trust again.

Familiarity Is the New Insider Threat

Comments