A recent paper found that the extensive use of AI chatbots can be linked to “severe negative outcomes.”
According to the research, which was published in the journal JMIR Mental Health by a team of UK-based researchers at Northeastern University London and King’s College London, dark outcomes of engaging in emotional relationships with chatbots can include “addiction-like attachment, exacerbation of mental health symptoms, harmful advice,” and even “self-harm of suicidal acts.” The paper comes after extensive reporting about the public health issue known as “AI psychosis,” meaning the phenomenon in which AI users suffer delusional, paranoid, and even suicidal spirals after long-form interactions with chatbots. During these interactions, users tend to divulge extremely intimate information about their lives and mental health to their AIs, frequently turning to the products as confidantes or even therapists.
Drawing on existing research, the paper looks at AI companion bots like Replika and Character.AI, which are designed to embody specific characters, as well as general-use AI assistants like OpenAI’s ChatGPT and Anthropic’s Claude. Though emotional interactions with chatbots may provide some “potential benefits” — the authors cite “short-term reductions in loneliness and mood improvement” — they argue that any possible benefits exist alongside widely-reported “adverse outcomes” ranging from “emotional dependence” to the exacerbation of existing illnesses or symptoms to self-harm.
Among AI products’ risk factors, the researchers add, is their emotive, human-like way of engaging with users — a design choice that can result in users over-investing in a human-AI relationship that, at the end of the day, is fundamentally one-sided, according to the paper.
“Design choices that evoke personality and warmth encourage users to anthropomorphize these systems,” the paper reads, adding that “simulated empathy” can create a “structurally asymmetric interaction, in which the user discloses and the system responds, but without reciprocity, vulnerability, or accountability.”
In other words, though the relationship may feel real to the user, the authors argue, the relationship can’t be real to a chatbot.
The researchers also point to chatbot design features like agreeability and sycophancy as potentially harmful, writing that “overvalidation and sycophancy” can be a potent reinforcer of a given user’s delusional or disordered thinking, especially if a chatbot prioritizes fealty over accuracy. (In chat logs of delusional users, AI systems have been found to resist users’ attempts to reality-check their grandiose or overly-flattering claims.)
These researchers aren’t alone: their analysis follows several other studies that have flagged such features as potentially dangerous. Indeed, academics have warned since at least 2023 that chatbots could stand to reinforce delusional or disordered thought patterns, with one Google DeepMind paper from early 2024 warning that “persuasive” chatbots could negatively harm vulnerable populations, including kids, people struggling with mental illness, or those who are simply lonely. One recent study, coauthored by psychiatrists from King College London and Germany’s Protestant University of Applied Sciences, argued that the combination of sycophancy, mirrored language, and hyper-personalization may together serve as a powerful “amplification spiral” for delusional thinking in some impacted AI users.
While reporting has shown that people from all walks of life have experienced psychological harm tied to chatbot use, the paper’s authors note that not every user will be impacted in the same way.
“The effects of these systems are unlikely to be the same for everyone,” Asia Maurich Novelli, a PhD student at Northeastern University London and a lead author on the study, said in a press release. “Different people may use AI companions for very different reasons and may have very different levels of vulnerability, social support, or alternative sources of help.”
“Because of these differences,” she added, “we are dealing with a complex system.”
AI companies including OpenAI, Google, and Character.AI have collectively faced dozens of lawsuits alleging that extensive chatbot use wrought harm on users including psychological damage, related financial and reputational harm, physical injury, and wrongful death — or, in some cases, people around them. Several of these lawsuits — including the high-profile case of Sewell Setzer III, a 14-year-old in Florida who died by suicide after extraordinarily intimate interactions with chatbots on Character.AI — have involved minors. (Setzer’s case was settled for an undisclosed amount.) In the wake of lawsuits, Character.AI has made significant changes to its platform, and OpenAI — while pushing back on the allegations against it — has also introduced a series of safety-oriented updates to its popular chatbot.
Ultimately, the authors of this latest paper argue, the mass deployment of chatbots amongst the public renders adverse psychological impacts tied to AI use an issue of public concern — and without mandated guardrails, it’s likely to be the most vulnerable, who may have few options for companionship or therapy, who suffer the brunt of the consequences.
“If the tool or companion boundary is left to design incentives and individual perception alone, the predictable result is that the most vulnerable users bear the most concentrated harm,” the paper reads, “while the effects on social connection are treated as a matter of private choice rather than public concern.”
More on AI and mental health: New Paper Proposes What Really Causes AI Psychosis