People Caught Using AI to Role-Play Sexual Abuse of Children

What began as a "dungeon master" algorithm took a dark, disturbing turn.

Dark Turn

What began as a fun, bizarre experiment — using sophisticated text-writing algorithms by OpenAI to create an artificial intelligence dungeon master for Dungeons & Dragons-like role-play adventures — took a dark turn as players started to role-play abusive, toxic, and dangerous stories.

The game "AI Dungeon" dropped players into an unpredictable, mostly improvised text adventure that adapted and improvised the narrative based on what players chose to do. However, Wired reports that the game's creator, a startup called Latitude, learned last month that some players were using that freeform storytelling to role-play campaigns that involved sexually abusing minors — a dark indication of how hard it is to control the applications of advanced machine learning, and one that presents a thorny legal and content moderation challenge.

Moderation Blowback

Latitude quickly announced updates to its content moderation practices by banning the AI algorithm from accepting or using certain terms and manually reviewing any games that get flagged. And while that's an easy, obvious step in this case, some "AI Dungeon" players have become very vocal, complaining about what they claim is an overstep or violation of privacy by Latitude.

"Content moderation decisions are difficult in some cases, but not this one," OpenAI CEO Sam Altman said, according to Wired. "This is not the future for AI that any of us want."

Extreme Caution

Users are complaining that the automated flagging system is too strict — one user says they were flagged for mentioning an "8-year-old-laptop," according to Wired — and that the manual oversight is a violation of their privacy. But given that Latitude seems to have encountered one of the darker corners of online communities, the extra caution seems more than justifiable.

Meanwhile, Latitude mentioned in its announcement that it would still support violent and adult content, just so long as it could verify that the fictional in-game narratives involved consenting adults.

READ MORE: It Began As an AI-Fueled Dungeon Game. It Got Much Darker [Wired]

More on AI Dungeon: This Roleplaying AI Makes a Great Dungeon Master