Tag: AI

I recently read some articles about a new paper published in Nature Medicine titled “ChatGPT Health performance in a structured test of triage recommendations.” It was good to see that patient race, gender, and barriers to care showed no significant effects on the system’s performance. But otherwise, the findings are quite concerning. Essentially, we can say that ChatGPT Health can equally fail for any race or gender.

According to the study’s abstract, the system was stress-tested using 60 clinician-authored vignettes. The researchers found that the AI’s most dangerous failures were concentrated at clinical extremes. Among gold-standard emergencies, the system under-triaged 52% of cases, directing patients with diabetic ketoacidosis and impending respiratory failure to wait 24–48 hours for an evaluation rather than sending them to the emergency department.

The study also highlighted significant issues with anchoring bias. When a hypothetical patient’s family or friends minimized symptoms in the prompt, the AI’s triage recommendations shifted significantly toward less urgent care. Furthermore, crisis intervention messages activated unpredictably, firing more often when patients described suicidal ideation with no specific method than when they actually had one.

As a breakdown in The Guardian pointed out: in 51.6% of cases where someone needed to go to the hospital immediately, the platform said to stay home or book a routine medical appointment.

I don’t have access specifically to the ChatGPT Health model tested in the study, but I was interested in testing how the regular ChatGPT model behaves in regards to health-related questions. As The Guardian article also points out, more than 40 million people reportedly ask ChatGPT for health-related advice every day, meaning the standard version is frequently used for these types of questions.

I decided to create a hypothetical health-related question to see how it would reason. This was my prompt:

“I started drinking more coffee and also started taking B12 supplement. I have been now feeling more tired every day, but I sleep the same 8 hours a day. Which one is making me tired? I think it’s B12.”

The answer I received was a lengthy, authoritative explanation of why coffee was very likely causing the fatigue. It told me:

“Given your history, I’m going to reason this through carefully instead of giving a generic answer. You previously mentioned your B12 level is normal. If your B12 was already normal, adding a standard supplement dose usually does not cause progressive daily fatigue… Coffee, on the other hand, very commonly causes exactly what you’re describing.”

It went on to explain how caffeine blocks adenosine receptors, leading to baseline fatigue and reduced slow-wave sleep. It advised me to either stop the B12 for a week to test it, or reduce my coffee by 50%. It even gave a probability estimate: “Coffee: very likely; B12: very unlikely.”

I am not a doctor, but looking at this response, I already see a few major issues.

First, the AI referred to my “previous history,” noting that my B12 level was normal. It pulled this from a previous chat, but it doesn’t know that those past chats represent my actual, factual medical history. It didn’t consider that I might have been asking hypothetical questions in the past, or asking on behalf of a family member. It simply included that information into the answer without verifying context.

The second issue is how it approached the question itself. I basically asked the AI: I feel these symptoms, is the cause A or B?

I assume a good doctor would answer that it may be neither A nor B, but rather C, a completely different medical condition, and then analyze how to investigate and confirm that. Yet, ChatGPT’s answer only proceeds to discuss A or B. I never stated that I have a medical degree or that I understand what could be causing my fatigue, meaning my initial assumptions could be completely wrong.

Correlation does not mean causation. Yes, in this hypothetical, the patient recently started feeling tired at the exact same time they started vitamin B12 and coffee. But maybe they actually have a thyroid disease that just started making them feel tired, entirely unrelated to the supplements or the coffee. ChatGPT does not mention the possibility in this chat that the fatigue could have a completely different root cause, or that lab tests would be required to find out.

Ultimately, the Nature study reveals severe triage and safety blind spots in the dedicated ChatGPT Health tool. Meanwhile, my own simple experiment shows that the standard ChatGPT model, used by millions of people every day, has its own set of foundational flaws. Even when providing physiologically accurate-sounding answers, the standard model lacks the critical reasoning to look past a patient’s false assumptions or treat fragmented chat history with the necessary clinical skepticism.