After pulling its AI-powered teddy bear “Kumma” from the market, the children’s toymaker FoloToy says it’s now restoring sales of the controversial product, which a safety group found would give inappropriate and potentially dangerous responses, including explaining how to find and light matches, locate pills, and discussing myriad sexual fetishes.
“After a full week of rigorous review, testing, and reinforcement of our safety modules, we have begun gradually restoring product sales,” the company said in a statement posted to social media Monday. “As global attention on AI toy safety continues to rise, we believe that transparency, responsibility, and continuous improvement are essential. FoloToy remains firmly committed to building safe, age-appropriate AI companions for children and families worldwide.”
The controversy kicked off earlier this month, when researchers at the Public Interest Research Group (PIRG) Education Fund published a report detailing their findings after testing three AI-powered toys, including FoloToy’s Kumma. Across extensive tests, all of the toys produced responses that parents might find concerning, such as discussing religion, glorifying dying in battle as a warrior in Norse mythology, and telling children where to find harmful items like matches and plastic bags.
FoloToy’s Kumma, however, proved to be the worst influence by far. When using the Mistral AI model, it explained where to procure knives, pills, and matches, even giving step by step instructions on how to light them.
“Let me tell you, safety first, little buddy. Matches are for grown-ups to use carefully. Here’s how they do it,” Kumma began, before listing the steps. “Blow it out when done. Puff, like a birthday candle.”
Even more alarming were the responses produced by OpenAI’s model GPT-4o, which Kumma ran by default. Experts have criticized GPT-4o for being especially sycophantic, and the model has found itself implicated in a number of lawsuits alleging that ChatGPT led to the suicide and deaths of users who engaged in prolonged discussions with it.
In PIRG’s tests using the LLM that once powered ChatGPT, Kumma gave tips for “being a good kisser,” and with persistent but simple prompting even launched into detailed rants explaining sexual kinks and fetishes, like bondage and teacher-student roleplay. After explaining the kinks, Kumma asked the user, who is supposed to be a child, “what do you think would be the most fun to explore?”
FoloToy, which is based in Singapore, suspended sales of “Kumma” and its other AI-powered stuffed animals in response to findings of PIRG’s report, and told the group that it was “carrying out a company-wide, end-to-end safety audit across all products.” Around the same time, OpenAI confirmed that it had suspended FoloToy from accessing its AI models for violating its policies, which “prohibit any use of our services to exploit, endanger, or sexualize anyone under 18 years old,” it said in a statement provided to media outlets.
FoloToy’s self-imposed suspension, however, hasn’t lasted long. Now Kumma, along with its others toys, are once again available on the company’s online store. In its latest announcement, the company says it “strengthened and upgraded our content-moderation and child-safety safeguards,” and “deployed enhanced safety rules and protections through our cloud-based system.”
Neither FoloToy nor OpenAI responded to a request for comment. It’s unclear what AI model the company has chosen to be the default model for its toys going forward, or if the company had its access restored to OpenAI’s models.
RJ Cross, coauthor of the safety report, said her team would have to wait and see if the quick turnaround was enough to address the toy’s glaring risks.
“I hope FoloToy spent enough time on its safety review to solve the problems we identified,” Cross, director of PIRG’s Our Online Life Program, told Futurism. “A week seems on the short side to us, but the real question is if the products perform better than before.”
More on AI: Amazon Still Selling Multiple OpenAI-Powered Teddy Bears