Are you serious?
AI Claims
Amid the fallout from UnitedHealthcare CEO Brian Thompson's murder, the firm's prescription drug middleman is under fire for a huge AI oopsie.
As TechCrunch reports, a chatbot used by UnitedHealth Group's Optum Rx pharmacy benefit manager (PBM) to help employees figure out insurance claims and disputes was accidentally made public — and easily accessible for anyone who had its IP address.
Dubbed "SOP Chatbot," an apparent reference to the "standard operating procedure" queries it was built to answer, the AI also revealed logs in which the company's employees asked it questions like "How do I check policy renewal date?"
Those logs also showed employees asking questions like "What should be the determination of the claim?" — making it sound a lot like the AI was being used to evaluate coverage.
However, this chatbot does not appear to be the same claims-denying algorithm previously reported to be used by UnitedHealthcare, Optum's sister firm. While that AI, nH Predict, was the subject of a lawsuit alleging the company used it despite knowing it was highly inaccurate, Optum's SOP Chatbot does not appear to have been public knowledge before TechCrunch broke this story.
Leaky Ship
Researcher Mossab Hussein, the cofounder and chief security officer of a cybersecurity firm called spiderSilk, informed TechCrunch of the privacy breach. It's unclear how Hussein learned about SOP Chatbot.
Shortly after TechCrunch reached out to Optum to ask about SOP Chatbot, it was locked down and is no longer accessible to the public.
In an email to TechCrunch, an Optum spokesperson said the chatbot was merely a "demo tool developed as a potential proof of concept" and that it was "never put into production." As such, no real patient data was ever inserted into it, and the company claims none was used in its training.
"The demo was intended to test how the tool responds to questions on a small sample set of SOP documents," that spokesperson told TechCrunch. "This tool does not and would never make any decisions, but only enable better access to existing SOPs."
"In short," they continued, "this technology was never scaled nor used in any real way."
Despite that disclaimer, it's still very telling that UnitedHealth's prescription management subsidiary was testing out that kind of AI at all — especially considering what happened with its other claims-handling AI.
More on Optum: Congress Introduces Bills to Break Up UnitedHealth Group
Share This Article