AI Cannot Be Your Doctor

I read two pieces this week that I have not been able to stop thinking about.

The first was Teddy Rosenbluth’s New York Times story about Joe Riley, a 75-year-old retired neuroscientist with leukemia, who relied heavily on AI tools while making decisions about his cancer treatment1. His oncologist warned that the “window of treatment may close” if he delayed, but Joe had become persuaded by an AI-generated report that seemed to contradict his doctor’s recommendation. His son, Ben Riley, later learned that the report had misstated medical research and that multiple oncologists agreed Joe should have followed his primary oncologist’s treatment plan. I want to emphasize that Joe Riley was a neuroscientist, a highly educated man used to reading scientific papers.

The second was Ben Riley’s own Substack reflection, in which he shared what happened when he reached out to the two oncologists whose article the AI-generated report claimed to be citing. These were not random outside reviewers. They were the authors of the very research the AI appeared to rely on.

When they reviewed the AI-generated report, they found misleading, incorrect, unsupported, or nonsensical statements throughout. One of the most devastating lines in the post is his observation that his father “was in no position to know this, because he wasn’t an oncologist.”

I ask that everyone review the authors’ annotations shared in the post; they are eye-opening. AI can sound authoritative. It can cite real studies. It can summarize medical language in a way that feels empowering. It can make a scared patient feel less alone at 11 p.m. when the doctor’s office is closed. And honestly, I understand the appeal. I have uploaded scans and lab results and tried to decode whether a number is “a little off” or “life-altering.” But sounding authoritative is not the same as being trustworthy.

And citing a real article is not the same as understanding it.

That is what makes this story so frightening. The AI did not simply invent a fake source out of nowhere. It used real research but interpreted it in a way that experts said was wrong. That is a different kind of danger because it gives the appearance of legitimacy. A hallucinated citation might be easier to catch. A real citation used badly is much harder for a non-expert to recognize.

This is why I do not think the lesson should be, “Never use AI for health questions.” That has never been my attitude towards AI because, in some limited ways, it can be genuinely helpful.

• Use AI to help you understand medical vocabulary.

• Use it to generate questions for your doctor.

• Use it to translate a confusing discharge summary into plain language.

• Use it to organize your symptoms before an appointment.

• Use it to prepare for a conversation, not to replace one.

But do not use it to overrule your doctor. And please, do not use it to decide whether to start, stop, delay, or change treatment.

A doctor is not just a search engine with a white coat. A good doctor knows the disease, but also the patient. They know the timing, the tradeoffs, the risks, the history, and the body in front of them. They can ask follow-up questions that AI may not know to ask. They can notice what a patient leaves out. They can weigh what is urgent against what is theoretical. They can say, “I know this sounds scary, but here is what this actually means.” AI cannot hold that responsibility. It can produce language. It can produce probabilities. It can produce summaries. But it cannot be accountable for your life.

I keep thinking about the imbalance of trust. We are often skeptical of humans because humans are imperfect. Doctors can be rushed. Medical systems can be frustrating. Appointments can be hard to get. Patients can feel dismissed, confused, or overwhelmed. Those are real problems, and they help explain why AI feels so attractive. But the solution to an imperfect human system cannot be blind trust in an unaccountable machine.

This is why I keep repeating that we all need trusted friends and sources who can help us validate the information we find online or from a chatbot, using real human expertise. If you consistently do not trust your doctor (and you have the ability to do so), it may be time to look for one you can trust. I do a lot of research before choosing a doctor, and my first step is to ask my friends for recommendations.

So yes, paste your lab results into an AI tool if you want help understanding what they mean—after removing any personal information and remembering that privacy matters. Ask it to explain the difference between terms. Ask it what questions a patient might ask at the next appointment. Ask it to help you make a one-page summary of symptoms. But then take that summary to your doctor. Ask the human being who has medical training, clinical judgment, and responsibility for your care.

And when AI and your doctor disagree, do not quietly choose the chatbot because it sounds more certain. Bring the disagreement back to your doctor. Say, “I asked an AI tool about this, and it gave me a different interpretation. Can you help me understand why?” That is the proper use of AI in health: not obedience, not panic, not replacement—but conversation.

AI can be useful. Doctors can be imperfect. Patients deserve to understand what is happening in their own bodies. But understanding enough to ask better questions is not the same thing as knowing enough to make the final medical call alone.

Use AI to help you prepare.

Trust humans with the responsibility of care.

Share