Yesterday I published an HBR article. For those of you without a subscription, here you go. It’s not as thorough or (in my estimation) as entertaining as The Ethical Nightmare Challenge, but it has the virtue of being relatively brief.
Here’s the link to the original.
Summary. The standard approach to responsible AI is fundamentally broken. In the age of generative AI, it’s too slow, too vague, and too hard to communicate. Instead of focusing on values and policy, companies would be better served by focusing on their worst-case scenarios—their AI ethical nightmares. That’s because this focus allows them to apply a novel, rapidly implementable approach that works for everything from narrow AI to governing AI agents. The Ethical Nightmare Challenge asks three questions: 1) What are the ethical nightmares of your organization as they pertain to AI? 2) What resources will you build to avoid those nightmares? 3) How will you train your people to use those resources effectively?
Purchase The Ethical Nightmare Challenge
Before generative AI burst onto the scene in late 2022, companies took a more or less standard approach to managing the risks introduced by AI: They developed AI ethical risk (or Responsible AI or AI Governance) programs. These programs were designed by executives and focused primarily on writing and implementing enterprise-wide AI policies that are meant to explain how the organization will live up to its AI ethics values (or principles or pillars, as they are also called). When generative AI showed up, organizations updated their programs to accommodate the new technology. Now that AI agents are gaining traction, most will likely try to update yet again.
That would be a mistake. The standard approach to Responsible AI is fundamentally broken.
I do not come to this conclusion lightly. It is the result of, first, seeing how the AI landscape has evolved in ways that create a diabolically complex risk landscape, and second, spending nearly a decade working with Fortune 500 companies across healthcare, pharmaceuticals, insurance, financial services, entertainment, and more to design and implement AI ethical risk programs. I’ve also worked in an advisory capacity with three of the largest consultancies in the world. I’ve had countless closed-door conversations with other leaders in the AI governance space.
The standard approach is too slow, too vague, and too hard to communicate. Instead of focusing on values and policy, companies would be better served by focusing on their worst-case scenarios—their AI ethical nightmares. That’s because this focus allows them to apply a novel, rapidly implementable approach that works for everything from narrow AI to governing AI agents.
The standard approach to Responsible AI goes something like this. First, the organization articulates its AI ethics values—often some combination of fairness, privacy, transparency, accountability, and safety. Then, it translates those values to enterprise-wide procedures such as checking for bias or filtering for sensitive data. Those procedures get enshrined in a policy, which is then implemented across the organization. Finally, it creates a Responsible AI board that high-risk AI cases can be escalated to, or assigns AI-related responsibilities to an existing risk board.
This all sounds like a reasonable way of proceeding. In my experience, there are three major problems with the approach.
For large organizations, getting from kickoff to board-approved policy takes a minimum of one year. For example, my work with a Fortune 500 CPG company that started in July 2023 reached a major milestone when the board approved the AI risk policy…in May 2024. And that was fast compared to our other clients. We began implementing the policy in the two departments that the organization’s leaders identified as highest risk. Just five months later, the policy was rendered out of date when OpenAI introduced agentic AI.
This pattern has repeated dozens of times.
Don’t get me wrong: enterprise-wide policies are a useful tool. But when it comes to managing the risks of a technology that changes every month, an AI policy is neither the most efficient nor most effective tool at an organization’s disposal.
The lesson: We need an approach to governing AI that is not bottlenecked by the C-suite and board; AI governance needs to move at the speed at which AI is developed and deployed.
Translating values into procedures is exceedingly difficult. It’s all well and good to claim you’re committed to fairness or respect for privacy, but what does that look like in practice?
One problem of reducing values to procedures is that it’s often not clear what success looks like. RAI programs often emphasize compliance, but don’t define the outcome they’re trying to produce, except to harken back to the original value. Does a procedure actually lead to fair, privacy-respecting, transparent outcomes? Without specific, measurable metrics, there’s just no way of saying.
The second problem has to do with starting with values in the first place. Generally speaking, the goal of an AI governance program is to prevent disasters—ethical, reputational, and legal. If that’s the case, values are the wrong place to start. If you’re focused on avoiding bad outcomes, the natural thing is to specify the bad outcomes themselves.
Lesson: We need an outcome orientated approach: success looks like avoiding the bad outcomes, failure looks like realizing the bad outcomes.
AI risk policies generally aren’t written in the language of the people who need to implement and/or comply with the policy. That means a tremendous amount of work is required to translate the policy in a way that employees can understand what it says, why it’s relevant to their work, and how to integrate it into their workflows. This translation work takes a lot of time (see flaw #1). But more importantly, it makes it hard for people across departments to communicate and (thus) collaborate in the identification and management of AI risks. This occurs for a variety of reasons, including that many people don’t read the policy (they are quite boring, after all), they don’t understand how it applies to the people with whom they need to collaborate, and because different translations by different people leads to people talking past each other. Collaboration around the tower of Babel is impossible.
In the age of agentic AI, this is a catastrophic failure. For AI agents to work, data scientists need to work with people in marketing and HR and operations, and everyone responsible for building and/or overseeing AI agents needs to speak the same language. Filtering the collaboration to avoid ethical, reputational, and legal disaster through an out of date and inscrutable policy is a non-starter.
Lesson: Articulate the bad outcomes in a way that is readily understandable to anyone. Neil the data scientist, Sarah in marketing, Sanjay in HR, Hannah in product, Sage the Gen Z in customer analytics, William the octogenarian board director—they all get it. No translation needed.
When you take the three lessons and combine them, you get something that is obvious in hindsight: we need a rapidly implementable approach to avoiding AI disaster that explicitly talks about those disasters in a way that everyone can readily digest.
Over the past year, I’ve developed an approach that takes heed of these lessons: the Ethical Nightmare Challenge. It starts from the premise that AI risk programs should identify and prevent our nightmare scenarios—the things we absolutely, unequivocally do not want to happen. This framework can rapidly be piloted and scaled, and it can integrate with existing AI risk programs. Here’s how it works.
Purchase The Ethical Nightmare Challenge
“Ethical nightmare” is not standard corporate vocabulary. Corporations like optimism. Opening a meeting by asking, “What are our worst-case scenarios?” can make executives uncomfortable. In fact, in a handful of cases, executives have told me that they fear this kind of language may be too off-putting or “negative” to colleagues. But as an AI risk professional at a Fortune 500 bank recently remarked in a closed-door session, “If you’re not starting off by identifying the potential disasters, I don’t know what you’re doing.”
This isn’t some weird social experiment. Starting AI risk discussions by talking about nightmares is a way to directly address the flaws above, in addition to other benefits.
First, it defines success and failure in terms of outcomes. Nightmares are bad outcomes. If you’re deploying AI at scale and not running into ethical nightmares, then all else equal, you’re succeeding. If you’re running into them, you’re failing.
Second, it communicates those outcomes in ways everyone can understand. Staff from across the company—data scientists, marketers, HR personnel, senior executives—should all be able to readily understand what counts as an ethical and reputational nightmares for the organization. Moreover, speaking in the familiar language of nightmares allows someone from a non-technical role to say to an AI data scientist, “Look, here’s what disaster looks like for us; how can you develop the AI in a way that avoids the nightmares, and how can we help?”
Third, nightmares are motivational in a way that abstract values aren’t. It’s easy to pay lip service to fairness and accountability and then go on with your day. But a nightmare—a scenario with real consequences, real people affected, real reputational and legal exposure—generates a sense of urgency that no ethics statement ever produced. I’ve watched executives who sat through values workshops without changing a single behavior come out of a nightmare-identification session asking, “What do we actually do about this?”
Fourth, nightmares create alignment. The CEO and the CHRO may disagree about what fairness requires. But virtually no one in any organization—board directors, data scientists, marketing managers—wants their company to discriminate against protected classes at scale, generate hallucinated content in client-facing documents, or manipulate customers into purchases they don’t want.
Fifth, a description of an organizational nightmare includes a description of how the nightmare came to be. We understand how we wound up creating an AI that discriminates based on race and gender, how the consulting firm came to turn in reports to their clients containing AI hallucinated material, and so on. And because we know how the nightmares happen, we can create strategies and tactics that keep us from going down those treacherous paths.
Perhaps most importantly, focusing on nightmares instead of values lets you act quickly.
This approach can be embedded in a framework, which can then be rapidly implemented throughout the organization and folded into existing organizational practices.
The Ethical Nightmare Challenge™ asks three questions:
What are the ethical nightmares of your organization as they pertain to AI?
What resources will you build to avoid those nightmares?
How will you train your people to use those resources effectively?
One of the greatest strengths of this framework is its portability. Anyone in the organization, be they in the C-suite or board, the leader of a department or division, or a project lead can ask those questions as they pertain to their respective roles. The C-suite and board asks it about the nightmares of the company as a whole, the HR or marketing division director asks it about their jurisdiction, and the AI product lead asks it of their particular AI solution.
In fact, the leader of any project can ask these questions when AI is involved. If, for instance, you’re a large consultancy and you’re handing in a report to a government, you might ask, ‘What are the ethical and reputational nightmares that pertain to our use of AI in this project?” Chances are, in such a case, “handing in AI hallucinated material” would be thought of within the first minute or two.
With your nightmares identified, it’s time to implement. This is done by creating Ethical Nightmare Challenge, or ENC teams. These teams can be created at any level of the organization and on any timeline, and are small (say, five to eight people) and cross functional (with at least one technologist). Importantly, they should be fast. Our clients are piloting the framework in six-to-10 weeks, with nightmare avoidance occurring during the pilot, not after a drawn-out design phase.
So what exactly do these teams do? The core mission of each ENC team is to work through the three questions. They are engaged in collaborative problem solving around the potential AI nightmares they’ve identified. At the macro level, they are the organizational capacity to engage in AI nightmare avoidance. When the three questions are echoed consistently at every level of an organization—board, C-suite, department, project team, individual contributor—they create a shared language that makes communication and collaboration across all of those levels possible.
The cross-functionality of these teams is essential to their success. A data scientist can identify technical sources of nightmares that a marketer would never see. A marketer understands consumer behavior in ways that make certain technical nightmares legible that engineers might miss. A product designer can see ways that things can go wrong that are invisible to legal. And because all of these people are speaking the same language and responding to the same questions, they’re able to engage in collaborative problem-solving rather than a departmental standoff.
If your organization already has a Responsible AI program, none of this means you should throw it out. Existing policies, risk boards, and governance frameworks don’t disappear under the ENC approach—they become resources that ENC teams can draw on. An enterprise AI ethics policy, for instance, may specify certain categories of AI use that are prohibited; those prohibitions can simply be treated as nightmare-avoidance guardrails that all ENC teams operate within.
An existing AI risk board, rather than being the first line of defense for every high-risk AI use case, becomes the exception, called upon when a project-level ENC team has identified a nightmare it cannot adequately mitigate, or when the stakes are high enough to warrant senior-level review. This is operationally crucial: as AI adoption increases, high risk cases increase as well, turning the risk board into an innovation bottleneck as teams wait to be reviewed and receive direction. Creating a more capable first line of defense through ENC training is the solution.
The Ethical Nightmare Challenge is not a magic bullet. Building the resources and training required to avoid specific nightmares takes real work. But it is work that can start immediately, at any level of the organization, without waiting for board approval or a two-year design phase. It is work that is comprehensible to everyone from the board director to the individual contributor. And it is work that produces a clear, verifiable answer to the question every leader should be asking about their AI programs: not “do we have a policy?” but “do we know our nightmares, and are we ready for them?”
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.