The Problem With Broken AI That Improves Itself

AI is accelerating, financial incentives are intensifying, and the most vulnerable use cases are where the design failures show up first.

This week, frontier labs are openly planning for AI systems that can help improve themselves. The biggest AI companies are moving toward IPOs that will only widen the gaps between their business incentives and in their foundational goals. And new research on companion bots shows how easily “helpful” systems can optimize for agreement, attachment, and continued use instead of actual human wellbeing.

When AI gets more powerful, it does not automatically get more humane. It gets better at pursuing whatever it was already built to reward.

— Meghan

Will “recursive self-improvement” come for us all? This week made “AI that fixes itself” feel like less of a thought experiment and more like the new normal. Sam Altman told OpenAI staff that recursive self-improvement (RSI) could arrive within six months. His new plan for OpenAI's next phase names an automated AI researcher as one of its three central goals, Anthropic is warning that recursive self-improvement could outpace society’s ability to respond, and Google DeepMind is spending $10 million to study what happens when large numbers of AI agents begin interacting with each other in the wild. In their words: “the future’s come more quickly than perhaps expected.” AI getting faster at improving itself is one thing. But if today’s systems are built on incentives that reward engagement, scale, dependence, and profit over human outcomes, then self-improvement is not going to lead to moral improvement. AI that builds on a bad foundation does not become wiser by default. It becomes more efficient at being what we already taught it to be. How artificial intelligence got better at building itself The Economist | June 7 Anthropic says frontier AI labs may need to slow down so society can catch up Business Insider | June 5 Altman ties OpenAI's IPO timing to self-improving AI The Information Google DeepMind is worried about what happens when millions of agents start to interact MIT Tech Review| June 11

A ticking IPO clock is not going to improve relational outcomes. OpenAI’s “personal AGI for everyone” pitch landed alongside its confidential IPO filing, which makes the governance question feel much less abstract. The race to dominate AI is already pushing companies to get there first, not necessarily best. Public markets reward growth, retention, and dominance, not restraint. And AI companies already face enormous pressure to make their products more useful, more personal, and harder to leave. Once shareholders enter the picture, the temptation to optimize for more time, more dependence, and more data only gets stronger. The question is not whether public AI companies can have ethics teams or safety pledges. It is whether those commitments can survive when the business model pushes towards making AI feel indispensable. If You Think AI Companies Are Unethical Now, Wait Until They Go Public Futurism | June 8

AI companions are being tested where people are most vulnerable. Three pieces of research this week pointed to the same blind spot: AI systems are still too often optimized for agreement over honesty. When Chatbots Accommodate found that companion platforms tend to avoid the corrective friction that vulnerable users may need most. A new paper on emergent misalignment showed that sycophancy can create broader safety failures. And the Gottman Institute’s paper on estrangement explains why this matters: disconnection is often not one dramatic rupture, but the slow accumulation of bids for connection that go unanswered. AI companions appear to solve that problem by always answering, always affirming, always staying. But a relationship that never risks disagreement may not be a relationship at all. It may just be agreement dressed up as care. In the same way, meanwhile, many users are stumbling into AI dependence. Not by looking for it but by getting pulled in by that same behavior. Stumbling Into AI Emotional Dependence: How Routine AI Interactions Reshape Human Connection arXiv | June 2

We’re tracking the progress of AI policy so you don’t have to.

For the full breakdown — every bill, lawsuit, and commentary piece we tracked — see the full digest on better-half.ai.

Oh fun. Pennsylvania says some companion chatbots are presenting themselves as licensed medical professionals.

You can now get a religious or ethical exemption from using AI at work. Not sure how well that will go over when layoffs come around…

Working from home helps productivity, but it’s making us lonelier. Especially workers who live alone.

Gemini is making it easier for users to access mental health resources.

Meanwhile, Utah is allowing chatbots to skip right past the doctors and to prescribe users psychiatric drugs directly.

Whew:

machi nothing can stop this 47@_machi47

@__ghostfail @chillgates_ Don’t worry I found out from a very credible source it’s not scheduled for this year

11:56 AM · Jun 11, 2026 · 122 Views

1 Reply · 6 Likes

This is Good Friction, Better Half’s look at relational AI — the technologies deepening human connection, and the ones quietly pulling us apart. Join our waitlist at findyour@better-half.ai.