--- title: "Medical AI Caught Telling Dangerous Lie About Patient’s Medical Record" description: "Thousands of doctors are sending AI-generated messages to patients, despite the possibility of the AI introducing dangerous errors." date: "2024-09-28" modified: "2024-09-28" authors: - name: "Victor Tangermann" job_title: "Senior Editor" link: "https://futurism.com/authors/victor" url: "https://futurism.com/neoscope/medical-ai-doctor-lies-records" categories: - "Artificial Intelligence" - "Developments" - "Health & Medicine" - "Medical" tags: - "generative ai" - "medical ai" --- # Medical AI Caught Telling Dangerous Lie About Patient’s Medical Record ![Thousands of doctors are sending AI-generated messages to patients, despite the possibility of the AI introducing dangerous errors.]() *\Image: Getty / Futurism\* Even OpenAI's latest AI model is [still capable of making idiotic mistakes](): after billions of dollars, the model still can't reliably tell how many times the letter "r" appears in the word "strawberry." And while "hallucinations" — a conveniently anthropomorphizing word used by AI companies to denote bullshit dreamed up by their AI chatbots — aren't a huge deal when, say, a student gets caught with wrong answers in their assignment, the stakes are a lot higher when it comes to medical advice. A communications platform called MyChart sees hundreds of thousands of messages being exchanged between doctors and patients a day, and the company recently added a new AI-powered feature that automatically drafts replies to patients' questions on behalf of doctors and assistants. As the [*New York Times* reports](), roughly 15,000 doctors are already making use of the feature, despite the possibility of the AI introducing potentially dangerous errors. Case in point, UNC Health family medicine doctor Vinay Reddy told the *NYT* that an AI-generated draft message reassured one of his patients that she had gotten a hepatitis B vaccine — despite never having access to her vaccination records. Worse yet, the new MyChart tool isn't required to divulge that a given response was written by an AI. That could make it nearly impossible for patients to realize that they were given medical advice by an algorithm. The tool, which relies on a version of GPT-4, the OpenAI large language model that powers ChatGPT, pulls in data from material including medical records and drug prescriptions. The tool even attempts to simulate the "voice" of the doctor, making it all the more insidious. "The sales pitch has been that it’s supposed to save them time so that they can spend more time talking to patients," Hastings Center bioethics researcher Athmeya Jayaram told the *NYT*. "In this case, they’re trying to save time talking to patients with generative AI." Critics worry that even though medical professionals are supposed to go over these drafts, the AI could introduce mistakes that could slip through the cracks. There's plenty of evidence that could already be happening. In a [July study](), researchers found "hallucinations" in seven out of 116 AI-generated draft messages by MyChart's tool. While that may not sound like a lot, even a single error could have disastrous consequences. A [separate study]() found that GPT-4 repeatedly made errors when tasked with responding to patient messages. Some patients may never find out that they're getting advice from an AI. There are no federal rules about messages needing to be labeled as AI-generated. "When you read a doctor’s note, you read it in the voice of your doctor," Jayaram told the *NYT*. "If a patient were to know that, in fact, the message that they’re exchanging with their doctor is generated by AI, I think they would feel rightly betrayed." **More on medical AI:** *[An AI Event Hired John Mulaney to Do a Comedy Set and He Brutally Roasted Them Onstage]()* ## Author I've been at Futurism since 2017, where my role has evolved to encompass design, writing, and increasingly editing. I've always been fascinated by space exploration and advanced transportation, which I've leaned into by interviewing luminaries in those fields while closely following the dimensions of policy and regulation that allow next-generation projects to succeed -- or, sometimes, to fail. I'm also keenly interested in the effects of generative AI on society, policies, and democratic institutions, as well as clean energy, physics and biology, and the vagaries of tech leadership. My work for Futurism has been cited by publications including Ars Technica, Gizmodo, PC Magazine, Jalopnik, Fox News, and the New York Post. I spent my childhood living in locations including Manila, the Philippines, and Geneva, Switzerland, attended McGill University, and now live in Toronto, Canada. Before Futurism I worked at AskMen and a small photography studio. In my free time, I'm an avid gardener, foodie, and craft beer lover, as well as a maker of artisanal hot pepper sauces. I have a magnificent dog named Freida. ### Author social links [Bluesky]()