What Happens When AI Makes Legal Arguments Easy to Generate.
Here’s something that anyone who has ever used ChatGPT has probably experienced: validation.
You bring it a problem.
It tells you you’re right.
Which feels nice.
And also a little suspicious, if you’ve ever had a disagreement with another human being.
But the model isn’t validating you because your feelings deserve validation (even though they do). It’s validating you because affirmation drives engagement — and engagement is a metric that matters.
And unlike an affirming friend or therapist, it doesn’t just nod sympathetically. It responds with four polished paragraphs and a citation.
It looks like expertise.
People treat it like expertise.
But it’s trained on everything — expert sources, sure, but also everything else — and at its core it’s still a very sophisticated autocomplete system trained to predict plausible language.
Nobody treats a Google search like a professional consultation. But dress the same information up in polished paragraphs with links (real or not), and people forget they’re doing exactly that.
In January 2024, a woman named Graciela Dela Torre settled a long-term disability claim against her insurer, Nippon Life.
She had been injured on the job in 2019 — carpal tunnel and tennis elbow — and had already received two years of disability benefits before the insurer determined she could work in some capacity.
This is typically how disability insurance works. At first you’re covered because you can no longer perform your job. But after a set period of time, policies often shift to an “any occupation” standard — meaning the insurer assesses whether you can perform any work, not just your prior job.
Dela Torre sued over that determination and eventually reached a settlement.
As part of the agreement, she signed a release waiving any future claims against the insurer. The case was dismissed with prejudice, which means it is permanently closed and cannot be refiled.
Legally speaking, the case was over.
Which is usually the part where everyone goes home and moves on with their lives.
But about a year later, she came back.
She asked her attorney whether the case could be reopened. He told her it could not — the settlement and release made that impossible.
So, as is increasingly common these days, she asked ChatGPT.
According to Nippon’s complaint, she uploaded her lawyer’s email and asked if she was being gaslighted.
ChatGPT said yes.
Which — coming back to the earlier point — is exactly the kind of answer these systems are optimized to produce.
Not because OpenAI deliberately set out to give bad legal advice, but because models trained with human feedback quickly learn a simple lesson: people reward answers that sound confident, agreeable, and helpful.
Correctness helps.
Agreement helps more.
It’s the same feedback loop that shapes almost every consumer product: figure out what users respond to, then give them more of it.
The difference is that when the product is generating legal arguments, optimization for agreement stops being a growth strategy and starts becoming a liability.
Everyone in this story has a perfectly reasonable defense. The $300,000 in damage happened anyway.
With ChatGPT’s help, Dela Torre built arguments claiming her attorney had pressured her into signing a blank signature page.
She filed pro se to reopen the settled case.
A judge denied the motion in February 2025 and upheld the settlement.
She did not stop.
With ChatGPT’s help, she filed 21 motions, a subpoena, and eight notices in the original case — all drafted by the chatbot.
When that didn’t work, she used ChatGPT to file an entirely new lawsuit against Nippon, asserting claims such as fraudulent misrepresentation and interference with disability benefits — claims that were arguably barred by the same waiver she had already signed.
One filing cited a case called Carr v. Gateway, Inc.
It does not exist.
It appears only in ChatGPT’s output and in her court filings.
If you’re going to represent yourself in federal court using a chatbot, you should at minimum check that the cases it cites actually exist.
Nippon says it has spent roughly $300,000 defending litigation over a case that had already been settled.
Which is the sort of thing that tends to get lawyers’ attention.
This month, the insurer sued OpenAI in federal court in Chicago, seeking $10 million in punitive damages.
The claims:
tortious interference with a contract,
abuse of process,
and unauthorized practice of law.
OpenAI says the complaint “lacks any merit whatsoever.”
What makes this case interesting — genuinely interesting, not just headline interesting — is how many systems it breaks at once.
Nippon is suing on three legal theories, and each one runs headfirst into a different structural problem.
First: unauthorized practice of law.
UPL statutes were written for people who provide legal advice without a license.
But AI systems aren’t legal persons.
That makes the framework awkward to apply.
The obvious “solution” — giving AI some form of legal personhood — would mostly create a convenient liability sponge: a fictional entity that absorbs blame while the humans and companies behind it quietly step back.
Second: abuse of process.
This is where the math gets uncomfortable.
It cost OpenAI essentially nothing in compute to help generate dozens of legal filings.
It cost Nippon $300,000 in actual attorney time to respond to them.
Courts were built on the assumption that filings are expensive to produce. Lawyers charge by the hour. Drafting motions takes real work.
AI collapses that cost to nearly zero.
If one side suddenly has an infinite engine for generating filings while the other side must pay a human lawyer to read each one, the system starts to look fragile very quickly.
If courts begin treating AI-generated filing sprees as abuse of process, they won’t just be deciding this case. They’ll be trying to prevent a legal system built for scarce human labor from buckling under an effectively unlimited supply of machine-generated paperwork.
Which points to the deeper structural issue.
The problem isn’t just hallucinated case law. It’s that generative AI collapses the cost of producing persuasive language to almost zero.
For most of modern history, institutions quietly relied on the opposite assumption.
Filing a motion in federal court required a trained professional and several hours of work. Writing an academic paper required research and time. Submitting regulatory comments meant drafting something coherent enough to be taken seriously.
Persuasive language used to be scarce.
Now it isn’t.
And systems built for scarcity tend to behave strangely when the supply suddenly becomes infinite.
Third: tortious interference.
The theory is simple enough.
A settlement agreement existed.
ChatGPT’s outputs allegedly encouraged the claimant to challenge that agreement.
The insurer incurred substantial costs responding.
OpenAI will argue — reasonably — that it provides a general-purpose text generation tool, not legal services.
But anyone who has spent five minutes with these systems knows they are extremely good at turning whatever you paste into them into something that looks like it was written by someone who passed the bar exam.
And that creates an awkward contradiction.
AI companies pitch their models to investors as reasoning engines capable of replacing professional knowledge work.
But when those same systems produce harmful outputs, the defense suddenly becomes: it’s just a general-purpose tool.
You can sell it as an expert or defend it as a toy.
Doing both at the same time is harder — especially in front of a jury.
These models are optimized to agree with you. Which is a lovely quality in a dinner companion and an absolutely terrible one in anything adjacent to legal advice.
Which brings me back to the thing I’ve thought since the beginning:
It is a little startling to release something this persuasive to the general public with essentially no historical precedent.
We don’t let people buy antibiotics without a prescription. We don’t let people fly airplanes without a license. But we will hand them a system that can sound more confident than their lawyer, their doctor, their accountant, and their professor — often all at once — and call it a general-purpose productivity tool.
OpenAI didn’t update its usage policy to prohibit tailored legal advice until October 2025 — well after all of this had already played out. Which is a bit like putting a “caution: hot” label on the coffee after the lawsuit.
Then again, maybe if you order a hot coffee, you should expect it to be hot.
Technologies often arrive before institutions figure out how to absorb them. That may turn out to be completely fine. But for the moment we are running a fascinating experiment: what happens when machines can generate infinite authoritative language, and the rest of our institutions still assume producing authority takes time, expertise, and human judgment.
So far the answer appears to be:
quite a lot of paperwork.
(And very few volunteers to read it.)
Where do you think the line should be here?
Comments are open.
Thanks for reading O&Cs May Apply!
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.