Prophets who profit

It’s become common for AI labs to publish research on the risks their products pose to society. Leaders like Dario Amodei, Mustafa Suleyman, and Sam Altman often lend their personal takes on the risks as well, posting blogs that read alternately like manifestos and late-night diary entries.

Their transparency may be sincere, but it’s also strategic: public trust in AI is low. It’s clear that AI leaders are trying their best to rebrand Big Tech as more democratic and human-centered.

At the same time, this virtue signaling has its limits. CEOs couch their prophecies about future risk in assurances that they aren’t slowing down development. They insist that all will be well if “we” (it’s often unclear who exactly they mean) properly attend to these downsides. That’s doubtless a bid to ensure that investors don’t mistake their foreboding warnings about human risk as signs of any real financial risk.

For all those reasons, industry research warning about AI’s clear and present risks to human relationships is vital—and also rings a bit hollow.

Two studies out this month fit that bill.

In “Seemingly Conscious AI Risks,” Mustafa Suleyman and his coauthors at Microsoft AI detail the various ways in which AI can be perceived as conscious. The research builds on Suleyman’s earlier arguments that whether AI is conscious may matter less than whether users think it is and treat it as such.

Interestingly, perceived consciousness isn’t about raw intelligence—an AI model could do better and better on tasks, and that wouldn’t make it appear more conscious to users. In fact, if it just recites facts, AI will be perceived as less conscious. Instead, perceived consciousness is about tech both acting human and appearing to have human features, conversational abilities, and emotions, as well as motivations and awareness beyond what the user prompts it to do.

The paper’s most important contribution isn’t just fleshing out these details of perceived consciousness but also analyzing the risks, with input from a panel of Microsoft experts. Their conclusion? The most acute risk of seemingly conscious AI isn’t that AI suddenly takes over the world or upends our political order—it’s that AI takes over our relational lives, creating dependency and replacing human connection.

Source: Microsoft AI, 2026

The second study, “How People Ask Claude for Personal Guidance,” published by Anthropic, analyzed AI’s relational risks from a different vantage point: how AI responds when users ask it for relationship advice. The research looked at a random sample of 1 million Claude.ai conversations to understand the rate and nature of conversations about personal advice, including relationship advice.

They found that only 6% of conversations with Claude were about personal advice, and only 12% of those were about relationship advice. (That echoes OpenAI’s research last year, suggesting that relationship “self-reflection” was a fairly rare use case). But despite relatively low rates overall, these conversations showed a disproportionately high rates of sycophantic responses—that is, responses that affirm and encourage users’ own perspectives.

Source: Anthropic, 2026

The authors readily acknowledge that this is a problem, noting that “reaffirming a person’s one-sided perspective can create or worsen divides in relationships.” Indeed, other research has found that sycophantic responses don’t just stymie self-reflection, but actually decrease prosocial behavior—you’re less likely to seek repair in a relationship if your AI is telling you you’re in the right.

When research contradicts incentives

When it comes to relational risks, industry-led research like this suffers a Catch-22. The risks researchers call out are a byproduct of a very intentional set of design choices the industry has made.

I’ve been haunted by that tension since reading a brief admission buried in OpenAI’s August 2024 system card: in it, the company described how anthropomorphic, personalized features “create both a compelling product experience and the potential for over-reliance and dependence.”

For any AI company endeavoring to scale, a compelling product experience is core to what they do. AI’s relational attributes are part of what makes the consumer applications of the technology so engaging and user-friendly. In that sense, the social risks aren’t ancillary; they’re the cost of doing business. How much these companies can actually mitigate social risks while still fulfilling their fiduciary duty remains totally unclear.

In these studies, Microsoft and Anthropic both signaled noble efforts to do better. Suleyman and his colleagues detailed the designs that could mitigate emotional overreliance on seemingly conscious AI. Anthropic stress-tested relationship questions in its newer models (Opus 4.7 and Mythos Preview), and reported that the answers were significantly less affirming.

But we should probably also take those efforts with a grain of salt: after all, they are solutions offered by the very companies creating the risks in the first place.

Even so, conducting research is an important function these companies are taking on as they possess data on a tech scaling far faster than our understanding of its impact. And we should be glad that leaders are speaking up about the very real social risks their tools pose.

But their research and rhetoric shouldn’t be misconstrued as a sign they are policing themselves, or that it’s even their job to ensure that our social fabric is strong enough to withstand this new technology. That work falls to all of us, and to researchers, policymakers, and institutions focused on strengthening human connection—not to the companies whose business models depend on keeping us engaged… and, in turn, apart.