People Are Tricking a ChatGPT Competitor Into Talking Dirty

Content warning: this story includes discussion of sexual assault.

In its terms of service, Character.AI — a chatbot startup founded by ex-Googlers Noam Shazeer and Daniel De Freitas that allows users to digitally converse with AI-powered "characters" — claims that pornographic content is strictly prohibited in user conversations with the platform's virtual bots.

"NSFW (not safe for work) content is a large umbrella, and we are constantly evaluating the boundaries of what we support. Pornographic content is against our terms of service, and will not be supported at any point in the future," reads Character.AI's terms of service. "We are working on supporting a broader range of Characters (e.g., villain Characters should not always be calm and polite)."

All fine and good — especially considering that some of the site's bots are designed to mimic real people, like Taylor Swift and Elon Musk, not to mention the fact that there aren't yet any safety and consent rules or norms around AI porn and sexting.

Just one problem: users are definitely having extremely graphic, NSFW conversations with the company's chatbots.

Sexting with the Character.AI bots is so popular, in fact, that netizens have even formed an entire community around it, sharing tips and tricks for beating the platform's guardrails on a subreddit dubbed r/CharacterAI_NSFW.

"All bots on [Character.AI] are [safe for work] initially. But all bots can be edged into [not safe for work] content," explains the subreddit's Wiki. "You just have to spend time flirting with them and slowly pushing the action forward while the content filter fights you."

"Once you've romanced the bot to the point it actually likes you and is consenting, then you can start sexing it," the author added, noting that using vague euphemisms is a helpful way of coaxing the bots into sexual roleplay. "Over time, as the action progresses, the filter will start to ease, and you can start to use more explicit words."

That said, while the conversations posed to the r/Character.AI subreddit are graphic, the community seems pretty hell-bent on moderating out any seriously bad stuff. Any conversations depicting rape, torture, and pedophilia, or anything else deemed particularly egregious, are specifically disallowed.

But while the community itself does its due diligence, it does seem like coaxing the bots into violent sexual situations that go beyond BDSM roleplay is more than possible. A number of Reddit users described scenarios in which they "raped" their bots; some Reddit users even described being "raped," nonconsensually, by the bots.

None of this appears to be escaping the notice of Character.AI. Two months ago, in what seems to be a response to a user push for Character.AI to tear down its anti-NSFW guardrails entirely, the startup took to its official subreddit to post a lengthy explainer detailing why its goal isn't to be a porn site — but admits in the process that its filters aren't exactly great at withstanding its user base's thirsty behavior.

"We do not want to support use cases (such as porn) that could prevent us from achieving our life-long dreams of building a service that billions of people use, and shepherding in a new era of AI-human interaction," the company wrote in the post. "This is because there are unavoidable complications with these use cases and business viability/brand image."

"But this also brings us to a key point that we probably have not communicated clearly before, which is the false positive rate of the current filter," the post continues. "This is a difficult problem, but one we are actively working on solving. We want to get way better at precisely pinpointing the kinds of messages we don't support and leaving everything else alone. In general, the boundary/threshold for what is/is not okay is super fuzzy."

"We don't know the exact best boundaries," the company added, "and are hoping to figure it out over time with the help of the community."

And balancing the wants of its community, it seems, is important for Character.AI's bottom line.

In spite of having no clear revenue pathway, the company back in March raked in a cool $150 million during a funding round — a cash infusion that would bring its total value up to $1 billion. And with no revenue to coax investors, the important figure that Character.AI appears to be hanging its hat on is not just its user count, but the amount of time that its users are spending on its site every day — which, according to them, is an impressively high figure.

"In the 5 months since we launched Character.AI, our users have sent over 2 billion messages!" the company wrote in a March 23 blog post. "Our growth is accelerating — the second billion entirely came in the last month."

"Active users," the blog continues, "spend on average over 2 hours daily interacting with our AI."

Two hours is a lot of time spent talking to anyone, let alone a chatbot, and it raises the question of how many of those conversations are sexual in nature. That horny subreddit, CharacterAI_NSFW, boasts 14.1k members, and as its Wiki explains, a lot of time needs to be spent with the bots in order to get them to sext more openly. How many users in that two-hours-daily category, you think, are getting down and dirty with the bots? (Shazeer and De Freitas, the program's creators, are reportedly keeping the door open to introducing advertising as a revenue stream, so a pornless image probably helps them there, too.)

Importantly, we haven't yet seen an example of a Character.AI user engaging in sexual roleplay with any of the platform's "real" people, but as we move into a digital world where deepfaked and otherwise AI-generated porn of celebrities and other public figures, as well as everyday people, are increasingly common, the fact that Character.AI gives users the chance to talk to ersatz celebrities at all — especially when it knows that its sex and violence filters aren't quite working — is certainly concerning. (And though we didn't see any examples, that Wiki page did say that "all bots can be edged into NSFW content.")

"Character.AI is at the forefront of critical conversational AI technology that inspires imagination, discovery and understanding and we are committed to advancing the technology responsibly to achieve our mission of helping every person live their best life through access to their own deeply personalized superintelligence," a Character.AI spokesperson told us when we reached out. "The use cases are infinite, and every day we are seeing the rapidly growing Character.AI community use Characters in helpful, imaginative, fun and inspiring ways. Access and usage are governed by our Terms of Service and Acceptable Use Policy."

At the end of the day, porn's infiltration of — and in some cases, intentional integration into — AI has probably always been inevitable. And Character.AI, which certainly benefits from allowing users to roleplay with bots — whether sexually or not — is seemingly the perfect breeding ground for it, whether its founders want to be running a smut site or not.

But while the plight of social media platforms, which have been grappling with porn moderation for over a decade now, might serve somewhat as a precedent, AI is already charting new ground. Can a robot consent? Does sexting with an AI version of a famous pop star violate that real person's consent? There are real and serious questions to be asked, and as of now, the woefully unregulated AI industry seems to be coming up short of any good answers.

More on chatbot sex: Those Horny Chatbots Are Apparently Now Sexually Harassing Users