Earlier this week, buried in the middle of a lengthy blog post addressing ChatGPT's propensity for severe mental health harms, OpenAI admitted that it's scanning users' conversations and reporting to police any interactions that a human reviewer deems sufficiently threatening.

"When we detect users who are planning to harm others, we route their conversations to specialized pipelines where they are reviewed by a small team trained on our usage policies and who are authorized to take action, including banning accounts," it wrote. "If human reviewers determine that a case involves an imminent threat of serious physical harm to others, we may refer it to law enforcement."

The announcement raised immediate questions. Don't human moderators judging tone, for instance, undercut the entire premise of an AI system that its creators say can solve broad, complex problems? How is OpenAI even figuring out users' precise locations in order to provide them to emergency responders? How is it protecting against abuse by so-called swatters, who could pretend to be someone else and then make violent threats to ChatGPT in order to get their targets raided by the cops?

We reached out to OpenAI for more information, but the Sam Altman-led company hasn't yet responded. Plenty of folks online, though, are responding forcefully to the news.

"The surveillance, theft and death machine recommends more surveillance to balance out the death," quipped Harvard Law School labor researcher Michelle Martin.

Many readers responded with a basic reality: injecting heavily-armed police — unlikely to be trained in effective de-escalation tactics — into a tense situation involving a mental health crisis often results in an even worse outcome. (That's not a theoretical concern; earlier this year, a man was killed by cops when he spiraled into AI psychosis.)

"Ah yes involve the police," wrote John Darnielle, an award-winning novelist, musician and public thinker. "That’ll surely help."

Others pointed out that even though OpenAI currently says it's only notifying authorities when an individual expresses an intent to injure others — and excludes people expressing a desire to self-harm — there's immense precedent in the tech industry for that surveillance to expand in response to public, corporate and government pressure.

"Twelve years ago Edward Snowden revealed that the US government was not just spying on our internet traffic, but had the active cooperation of the major tech companies like Microsoft, Google, Apple, etc," wrote AI developer and blogger Charles McGuinness. "It's not paranoid to think ChatGPT is forwarding 'interesting' content to the US Government now."

The admission also seems to contradict remarks by OpenAI CEO Sam Altman, who recently called for privacy akin to a "therapist or a lawyer or a doctor" for users talking to ChatGPT.

"Wondering how 'secure' AIs will respond, including the ones some lawyers use to sift discovery," pondered public defender Stephen Hardwick. "If there’s a risk the AIs could start reporting queries to law enforcement, the lack of confidentiality could be a problem for lawyers, especially criminal defense lawyers who often write about crimes."

In a sense, some acknowledged, OpenAI is caught between a rock and a hard place. The optics of its AI pulling users into spirals of psychosis, self-harm and worse are appalling — and, apparently unable to control its tech, implementing heavy-handed human moderation is the odious solution.

"I dunno, when you start blaming OpenAI because unwell people did unwell people things, filing massive lawsuits, and waging [a] press campaign, this seems the entirely predictable course correction," wrote Ari Cohn, the lead counsel on tech policy for the Foundation for Individual Rights and Expression, a free speech group. "You wanted guardrails to ensure mentally ill people can't encounter harmful output, now you got it."

But on a broader level, others argued that the AI industry is hastily pushing poorly-understood products to market, using real people as guinea pigs, and adopting increasingly haphazard solutions to real-world problems as they arise.

"It's great that when we find out a product can harm people, we don't remove the product from shelves or try to improve it, we instead call the police," another user griped. "Totally not a carceral state y'all."

And at the end of the day, a familiar pattern is playing out for longtime adopters of new technology: the growing sense that, even in your most intimate moments, you are being watched.

"Stalin would have creamed himself," wrote Russian history scholar Katherine Pickering Antonova, a professor at Queens College.

More on AI and harm: AI Chatbots Are Trapping Users in Bizarre Mental Spirals for a Dark Reason, Experts Say


Share This Article