The Week That AI Crossed the Line in Healthcare

“There are decades where nothing happens; and there are weeks when decades happen.” Last week, this famous observation (usually misattributed to V.I. Lenin) rang particularly true in the world of healthcare AI.

On Tuesday, legislators in Utah approved a pilot program that allows AI to refill certain medications without a human doctor. Two days later, OpenAI announced a new feature called ChatGPT Health.

These two events may not seem all that consequential. However, each of them – and particularly their nexus – heralds a profound shift: the moment AI moves from advising on care to actually delivering it.

Under a new “Health” tab, ChatGPT users can now load in their healthcare data, through tools like Apple Health or, more cumbersomely, from their electronic health record. That information remains in the system’s memory (unless the user chooses to delete it) and becomes GPT Health’s background database.

This means that when the user enters, say, a new symptom like shortness of breath or a new piece of data, such as a blood test result, the system “knows” the patient’s relevant medical history and responds with a contextualized answer. The company touts GPT Health’s capabilities to not only perform DIY symptom-checking, but also to recommend preventive activities (“Eat more kale!”) and prepare for a doctor’s visit (“What questions should I ask about my new headache?”)

Since the dominant concern is likely to be around patient privacy, OpenAI promises “enhanced data protection” in GPT Health. The company also vows that the data in GPT Health won’t be shared or used to train its AI algorithms.

Interestingly, there is nothing special about the search itself – the same AI models that power regular-old GPT-5 will be applied to a GPT Health search. In other words, if GPT Health provides better search results, it’s due to the user-supplied personal data, rather than a more medically tuned AI algorithm.

Would I use ChatGPT Health? If I were a frequent user of GPT for health queries, then yes, since the results I would get from a GPT Health search would likely be more useful than those from a generic search. Would I load in sensitive healthcare data? Probably, though it should be noted that if you choose to hand your data over to OpenAI, unlike organizations that deliver healthcare services, the company has no obligation to adhere to HIPAA privacy laws. While that should raise some concerns, OpenAI, like all the digital behemoths, has a powerful incentive to avoid privacy breaches.

As tools like GPT Health create AI-based outputs that are increasingly personalized, many of the recommendations (“get more exercise”) will be achievable without requiring that patients access the healthcare system. But some recommendations will undoubtedly include starting a treatment, maybe a prostate medicine or an inhaler for asthma or a statin for cholesterol. When that happens, the democratization brought about by AI encounters the last-mile problem of prescription access – you can get sophisticated diagnostic guidance, but to obtain the actual medication, you’ll still need to navigate the legacy system of doctors and appointments.

Or, maybe you won’t.

This week’s other big announcement came from Utah, where, under a pilot program, AI will be permitted to refill certain medications without a physician’s signoff. On the face of it, this seems like small potatoes. For one thing, the new law limits AI prescribing to a set of 190 relatively low-risk medications. More importantly, restricting the program to refills means that a physician or other licensed professional has already endorsed the medication, at least at first.

Moreover, the AI won’t automatically issue refills – it will take several steps to ensure that the refill is clinically appropriate. Once confirming that you live in Utah, the company that runs the platform, Doctronic (“I’m your private and personal AI doctor,” says the company website, with swagger that most AI decision support companies avoid), asks whether your medication is on the approved list. It then inquires about any new symptoms, other medications you’re taking, and side effects you might have experienced. If any red flags arise, you’re directed to a video visit with a clinician. If not, you pay a $4 processing fee, and your refill is approved by the algorithm.

Is this safe? In its application to Utah regulators, the company highlighted a study that found a 99.2% concordance between the AI’s refill decisions and those made by physicians. Personally, I’d have no problem using this system to refill my Lipitor.

While the Utah program is limited in scope and impact, it crosses an important line – the one that separates AI from physicians. The American Medical Association, of course, pushed back. “While AI has limitless opportunity to transform medicine for the better, without physician input it also poses serious risks to patients and physicians alike,” said AMA CEO John Whyte.

In the face of a huge nationwide shortage of primary care doctors, I doubt that Whyte really believes the Utah system is massively problematic, or even that most of his doctor-members would object to giving patients the option of using AI to refill their Wegovy or Albuterol – though concerns about care fragmentation are legitimate, and the AI won’t be listening to a patient’s lungs or checking their blood pressure, at least not yet.

Still, the AMA’s pushback is as logical as it is predictable. Why? Because once AI is allowed to prescribe a refill for an existing prescription, it’s only a matter of time before the bar is lowered to allow for certain new prescriptions as well. And, before you know it, you really are dealing with AI doctors – with no actual “doctor in the loop.”

How should an AI doctor be regulated? In a provocative 2024 article in the New England Journal of Medicine, Harvard’s David Blumenthal and Google’s Bakul Patel argued that we should think about regulating healthcare AI less like we do devices like CT scanners and pacemakers, and more like we do physicians. Under such a model, we’d scrutinize a model’s training, and require that it pass appropriate tests, undergo a period of supervised use, be re-tuned and retested periodically, and report the outcomes of these steps to both regulators and the public. While there are numerous challenges associated with such a model, it seems likelier to succeed than attempting to force-fit “AI doctor” regulations into the FDA’s structure, which was designed for medications and devices.

As always in medicine, the liability system provides its own guardrails. Today, while AI might make a recommendation or a prediction, the physician remains the final decision-maker, including signing the prescription. Even if the AI gave bad advice, you can bet that the doctor will be the one being sued. Obviously, if the physician is taken out of the loop, the ambiguity regarding responsibility – and liability – is removed. Knowing this, Doctronic has purchased medical malpractice insurance for its Utah AI prescription service. This may well be the first time a company has bought liability insurance for autonomous AI in medicine. It won’t be the last.

“This is how the future creeps into the present,” the journalist Alexis Madrigal wrote in 2014. “While it might seem like your main computing device transformed from a Dell desktop into a smartphone overnight, there were thousands of little steps along the way that led to the moment when you realized that the world had changed beyond recognition.”

I think we’ll look back at this week as one in which the threads of piecemeal AI innovations and regulatory flexibility began to be woven into a new tapestry. Let me try to sketch out where this all ends up:

On the GPT Health side, the current system requires that you port your health data from various databases into GPT, then ask questions. In the not-too-distant future, GPT or other tools with GPT-like capabilities will be embedded in your electronic health record, so that the system is always aware of your relevant background information (not only your medical history but your educational background, living and family situation, and insurance company and its rules). Your patient portal, whether it’s MyChart or another version, will be your version of GPT Health, but one that’s more convenient to access and constantly updated.

And, rather than your needing to enter a prompt (“What do you think the diagnosis is?”), based on your new symptoms or test results and the information in your stored database, the system will proactively suggest diagnoses and tee up appropriate treatments, further testing, or – if it is embedded in a health system EHR – an appointment to see a real-live human professional.

(Note that OpenAI, in an announcement that received less attention than GPT Health, launched another healthcare program last week: GPT for Healthcare. The program provides specialized GPT models to health systems, ensuring that the results remain entirely within the health system’s firewall. UCSF was one of the early adopters, which takes us one step closer to having advanced AI embedded in our healthcare workflow.)

The question of whether a medication is safe enough to bypass an MD’s prescription is not novel in American medicine. As you may know, certain medications once available only by prescription were ultimately deemed safe enough to be sold over the counter (OTC). Before there was Advil there was prescription-only Ibuprofen. Ditto Tagamet, Claritin, Prilosec, and Rogaine, even hydrocortisone cream.

What is new is making prescription medications – drugs deemed too risky to sell over the counter – available to patients based on software algorithms rather than a clinician's evaluation. Today, of course, millions of patients work around the requirement for an MD prescription via pro forma telephone or video consultations with physicians they don’t know, allowing them to obtain medications for conditions like obesity, erectile dysfunction, and hair loss. Assuming an AI tool goes through rigorous assessment and certification, is bypassing the physician entirely any riskier than that? I doubt it, and it might very well be safer if we get the regulatory framework right.

Don’t get me wrong: the bar for AI prescribing should be set very high, and each one of these experiments should be framed as just that – experiments, complete with data collection, oversight, and transparency. I certainly don’t want AI prescribing chemotherapy, opiates, or blood thinners anytime soon, and, given the risk of promoting drug resistance, I’d even be careful about relatively safe antibiotics.

But in a healthcare system buckling under the weight of crushing demand, we don’t have the luxury of waiting too long. The AI Doctor is arriving faster than regulators anticipated and sooner than skeptics expected. The challenge now is to stop debating if patients should manage some of their own care and start defining how. We need to rapidly build the guardrails that separate safe AI-enabled care from reckless self-diagnosis and therapy. And we need to do it before the next “week where decades happen” happens.