LLMs struggle with triage and answering patient questions. How can we make this safer?

Three recent studies have highlighted safety limitations of LLMs in direct patient interactions. I led a red-teaming study with fifteen physicians, recently published in npj Digital Medicine. We found that LLMs often provide unsafe answers to patient-posed medical questions. The model with the highest percent of unsafe responses was ChatGPT (13.5% unsafe), followed closely by [ ]