Week 265

In Week #265 of the Doctor Penguin newsletter, the following papers caught our attention:

1. Clinical Environment Simulator. Just as aviation replaced written examinations with comprehensive flight simulators to better capture the complexity of real flying, clinical AI evaluation should similarly move toward simulated hospital environments that reflect the true demands of medical practice.

In this perspective, Luo et al. propose the Clinical Environment Simulator (CES), a framework designed to move AI evaluation in healthcare beyond static, disconnected test cases toward dynamic, integrated hospital simulations. The CES framework consists of a “hospital engine” that simulates real-time bed availability, staff workloads, and equipment status, and a “patient engine” that simulates disease progression in response to AI interventions. Together, these two engines create dynamic feedback loops where every clinical decision reshapes future options, enabling a more realistic basis for evaluation. Unlike existing benchmarks that treat clinical reasoning as a series of isolated puzzles, the CES captures three capabilities current evaluations miss entirely: temporal reasoning under evolving constraints (where diagnostic delays can directly cause patient deterioration), resource-aware decision-making (where aggressive workups for one patient may exhaust capacity needed by others), and operational resilience through adversarial stress testing with simultaneous emergencies and system failures. The framework also requires AI agents to act through realistic EHR interfaces rather than simply generating text, and measures performance on both clinical outcomes and operational efficiency metrics simultaneously. Ultimately, the CES enables evaluation of clinical LLMs not just on diagnostic accuracy, but on their ability to function as reliable partners within the full temporal and systemic complexity of real healthcare delivery.

Read paper | Nature Medicine

2. Agentic CT System. Agentic system for chest CT report generation.

Roschewitz et al. developed RadAgent, an agentic system for chest CT report generation that produces fully traceable, stepwise diagnostic reasoning. Rather than mapping a scan directly to a final report, it first generates a preliminary draft and then systematically refines it by working through a clinician-reviewed diagnostic checklist. At each step, the agent plans the next diagnostic action, selects an appropriate tool, updates its memory with tool outputs, and iterates until sufficient evidence supports a final report. Importantly, the study shows that training agents to learn a tool-calling policy is more beneficial than using off-the-shelf LLMs as clinical agents. Additionally, compared to a non-agentic VLM that generates reports in a single step, RadAgent is better at resisting incorrect prompt hints, and when a hint does change the model’s output, the report explicitly acknowledges that influence. More broadly, such agentic systems could serve as flexible front-ends that bridge the longstanding tradeoff between breadth and specialization, routing specific clinical subtasks to the most appropriate specialized tools while remaining adaptable to the complex, multifactorial nature of clinical practice.

Read Paper | arXiv

3. Autonomous Patient Care. In January 2026, Utah launched a 12-month pilot program that allows an AI system developed by health tech startup Doctronic to autonomously renew prescriptions for chronic conditions.

In this perspective, Gerke et al. raise concerns about Utah's AI prescription-renewal pilot program from both clinical and legal angles. While the program targets medication nonadherence (only 40–60% of patients with chronic diseases adequately adhere to their medications, and poor adherence to antihypertensives is linked to a 32% increase in all-cause mortality), the authors argue that autonomous renewal cannot address the many other drivers of nonadherence, such as health literacy, medication side effects, cost barriers, regimen complexity, and psychosocial factors, and is inappropriate for drugs like levothyroxine and sacubitril–valsartan that require careful monitoring. The program may also carry serious legal vulnerabilities, as the FDA could classify Doctronic's system as a class II medical device requiring premarket review, which reportedly has not occurred. Furthermore, even if Utah permits AI-based prescribing without physician supervision, federal law would preempt state law. Central to this is whether the term "practitioner," which the authors believe most naturally refers to a person, can legally apply to an algorithm. If the FDA chooses not to act, patients and clinicians could be left without meaningful safety assurances as AI autonomy in medicine expands.

Read Paper | The New England Journal of Medicine

4. Human–LLM Interaction. What would it take for patients to trust and choose medical AI in their care?

Bracic et al. conducted a conjoint survey study with 3,000 English-speaking US adults to understand what drives patient trust in and preference for medical AI. Respondents were presented with a hypothetical scenario in which a medical AI analyzed a photograph of their rash and provided an initial diagnosis, then asked to choose between pairs of hypothetical visits (all equal in cost) that varied across six randomized attributes. The single most important factor was AI performance, an AI performing better than a specialist increased the probability of a patient choosing that visit by 32.5%, nearly three times the effect of FDA approval. Clinician presence ("human in the loop") boosted visit preference by 18.4%, roughly equivalent to AI performing at the level of a general practitioner, underscoring that patients strongly value human oversight even as they respond positively to demonstrated competence. All forms of governance increased preference compared to no governance, though local hospital certification had a notably smaller effect than FDA approval or Mayo Clinic certification, suggesting patients place greater weight on national-level validation. The authors note that local validation is the most practically relevant to individual patients and cannot be delegated to higher-level bodies, though it is also the most vulnerable to resource disparities. Disclosure that an AI was trained on representative data also meaningfully increased trust, while disclosure of a biased dataset had no statistically significant effect. The authors caution that in real clinical settings, such information is rarely shared with patients, implying that greater transparency about AI performance, data quality, and oversight could itself be a meaningful tool for building the patient trust that medical AI adoption will ultimately require.

Read Paper | JAMA Network Open

5. Human–LLM Interaction. When patients use LLMs to assess their own symptoms, do they make better decisions than if left to their own devices?

Bean et al. conducted a randomized study with 1,298 UK participants to test whether LLMs can meaningfully help the general public make better self-assessments on common medical conditions. Participants were randomly assigned to use one of GPT-4o, Llama 3, or Command R+, or a control group where they used whatever resources they would normally use at home (such as internet search), to identify relevant conditions and decide on a course of action. They found that using LLMs worsened or did not improve participant performance on these tasks relative to using traditional resources, and the models consistently performed better without user interaction. Users frequently provided incomplete symptom information, and LLMs sometimes gave correct answers that users then failed to act on. Comparing across scenarios, the authors also noticed inconsistency in how LLMs responded to semantically similar inputs. In an extreme case, two users sent very similar messages describing symptoms of a subarachnoid hemorrhage but were given opposite advice: one user was told to lie down in a dark room, while the other was correctly advised to seek emergency care. The sensitivity of LLMs to small variations in inputs creates challenges for forming mental models of LLM behavior, and even occasional factual and contextual errors could lead users to disregard advice from LLMs. The study raises an alarm that standard benchmarks and simulated patient interactions failed to predict these real-world failures, and argues that solving these challenges will require moving away from benchmarks and simulations toward systematically conducting safety testing with diverse, real users.

Read Paper | Nature Medicine

-- Emma Chen, Pranav Rajpurkar & Eric Topol