We can’t wait until AI breaks us to fix it.

Something shifted this month. The public conversation about AI’s emotional harms moved out of the narrow frame of model behavior and technical choices into a larger question: the kinds of people, relationships, and societies these systems are training into existence.

In the span of a few days, Pope Leo issued his first major encyclical calling out the dangers of AI, with Anthropic’s Chris Olah and New Public’s Eli Pariser joining him at the Vatican to talk about the moral architecture of the field. And Dario and Daniela Amodei sat across from Oprah naming the fork in the road directly: AI that helps you have better human relationships, or AI that replaces the need for them.

All of them agree that AI’s worst outcomes are not inevitable. Pope Leo argued that the main challenge with AI is not technological but anthropological. Pariser, who has spent years studying how platforms shape human behavior, put it most precisely:

“Many of the worst, parasitic outcomes with AI – replacing your friends, your therapist, your sense of reality – are not baked into AI, they’re decisions made at the very end of the training process. The business models, and the economic incentives, are not locked in yet. Sycophancy and antisocial incentives aren’t the cake, they’re the frosting.”

That should be reassuring. It means these outcomes can be changed.

But it also means the positive outcome isn’t guaranteed — and isn’t where the incentives currently point. These outcomes are changeable, but only if businesses make different choices, right now, while the architecture is still being written.

And right now, responsibility is being distributed so widely that it risks disappearing all together. At the Vatican, Olah put the onus back on everyone outside Anthropic:

“We need more of the world—religious communities, civil society, scholars, governments, and indeed all people of good will—to do what His Holiness has done here: to take this seriously, to look closely, and to push events in a better direction. We need informed critics who will tell the labs when we are failing. We need moral voices that the incentives cannot bend.”

Today, Anthropic doesn’t sell ads. They say they are not building toward engagement as their main metric. But unless accountability is built into the outcomes, those choices remain vulnerable to growth pressure, competition, investors, geopolitics, and time.

Through this and other statements, Anthropic has been clear that they will not hold the line if they aren’t pushed. In an interview with Anthropic’s co-founder Jack Clark [last week] noting the impending clash coming around human attachment with AI: “we’re not there yet, but as the tech improves, this conversation becomes unavoidable.”

All of these statements acknowledge the problem. They express concern. They call for better. And they locate the moment of real accountability somewhere ahead of us — after the tech improves, after the institutions organize, after society catches up.

But the decisions that will determine what that future looks like are being made now, in this training run, in this product roadmap, in this choice about what to optimize for. By the time ‘we’re there,’ the architecture will be infrastructure.

AI is already becoming part of people’s emotional lives. It is already shaping how people love, repair, soothe, decide, and relate.

Accountability cannot wait for the moment the outcomes are real and harmful. By then, the product logic will have become social logic.

That is not an anti-technology position. It is a design position.

And it is the standard teams building in this space should be held to.