--- title: "While Meta Stuffs AI Into All Its Products, It’s Apparently Helpless to Stop Perverts on Instagram From Publicly Lusting Over Sexualized AI-Generated Children" description: "Meta, hoping to cash in on AI for itself, is failing to prevent abusive AI-generated images from spreading across its social media platforms." date: "2024-05-24" modified: "2024-05-24" authors: - name: "Maggie Harrison Dupré" job_title: "Senior Staff Writer" link: "https://futurism.com/authors/mharrison" url: "https://futurism.com/ai-perverts-instagram" categories: - "Artificial Intelligence" - "Meta" tags: - "ai" - "ai-generated content" - "instagram" - "Meta" --- # While Meta Stuffs AI Into All Its Products, It’s Apparently Helpless to Stop Perverts on Instagram From Publicly Lusting Over Sexualized AI-Generated Children ![Meta, hoping to cash in on AI for itself, is failing to prevent abusive AI-generated images from spreading across its social media platforms.]() *\Image: Getty / Futurism\* As Facebook and Instagram owner Meta seeks to jam generative AI into [every feasible corner of its products](), a [disturbing *Forbes* report]() reveals that the company is failing to prevent those same products from flooding with AI-generated child sexual imagery. As *Forbes* reports, image-generating AI tools have given rise to a disturbing new wave of sexualized images of children, which are proliferating throughout social media — the *Forbes* report focused on TikTok and Instagram — and across the web. The images generally feature young, pre-adolescent girls, many of whom are suggestively dressed; the followers of social media accounts who post and interact with the images are overwhelmingly older men who leave disgusting and sexually-charged messages in the comments section. But, while objectively horrendous, the images currently lie in a legal gray area. Though the images are trained on photos of real children — and at least in the case of the AI image generator Stable Diffusion, off of images of *real child sexual abuse material (CSAM)*, according to a [December study]() conducted by the Stanford Internet Observatory — the resulting image is that of a fake human. And while the images are sexualized, they're not necessarily overtly explicit, ultimately rendering them legal (explicit CSAM is always illegal, [even when fake]().) Predators appear to be using this fuzzy gray area as a vehicle to explore their darkest fantasies — a practice that experts fear may ultimately result in real-world exploitation of kids. "Child erotica is the gateway," Heather Mahalik Barnhart, a digital forensics expert, told *Forbes*, adding that "when you look at who's following it, it’s not normal." Obviously, it's awful that this stuff is anywhere. That these images are spreading throughout Instagram, though, presents a particularly troubling irony for Meta, a company that's desperately trying to keep up in the AI race. So far, the tech giant's multi-billion-dollar AI efforts — a chatbot that [disparaged company CEO Mark Zuckerberg](), bizarre AI versions of [celebrity characters](), a [news-summarizing AI assistant](), and so on — have yet to make a significant cultural splash. Meanwhile, it's clearly unable to use the tools at its disposal, AI included, to help *stop* harmful AI content created using similar tools to those that Meta is building from disseminating across its own platforms. We were [promised]() creativity-boosting innovation. What we're getting at Meta is a platform-eroding pile of abusive filth that the company is clearly unable to manage at scale. (On a similar note, [have you scrolled through Facebook lately]()?) For its part, per *Forbes*, Meta permanently removed the specific accounts that the reporters found from Instagram. A spokesperson for the company also insisted that it removes real and fake CSAM from its platforms and reports the images to the National Center for Missing and Exploited Children (NCMEC). (TikTok also removed *Forbes*-flagged accounts, with a spokeperson telling Forbes that the social media company has "strict policies against AI-generated content of minors to protect young people and keep TikTok inhospitable to behavior that seeks to harm them.") But social media companies going account-by-account to manually remove this stuff when reporters specifically flag it isn't exactly a scalable solution. And as director NCMEC's CyberTipline Fallon McNulty told *Forbes*, because "these are all being trained off of images of real children, whether depicted in full likeness or not," the "NCMEC doesn't really see that side of the argument that there's a world where this is okay." "Especially given some of the commentaries that the images are attracting, it doesn't sound like the audience that is reviewing, ingesting, consuming those images is innocent," McNulty continued, adding that she "would hope to see" social media platforms "removing that content and not creating a space on their platforms for individuals to be sexualizing children." **More on AI:** [*Google Strategist Quits, Slams Company's AI Work as Motivated by Greed and Fear*]() ## Author At Futurism, I've reported extensively on the rise of AI as a cultural and business force shaping the media industry, and more broadly how those dynamics are changing how we all consume and share information and relate to one another. I'm also fascinated by public health policy and ethics, the role of emerging tech in politics and governance — and the powerful people and forces at those intersections — climate change, and the environment. My investigation on Sports Illustrated's use of AI-generated authors with fictional biographies won a 2024 Mirror Award for "Best Story on Media Coverage of Artificial Intelligence in Journalism and the Media" from Syracuse University's SI Newhouse School, I contributed to Niemen Lab's 2025 Predictions for Journalism series, and I've discussed my work for Futurism during appearances on NPR, CNN, the BBC, the CBC, and more. I grew up in rural Pennsylvania and attended the University of Massachusetts Amherst, where I played Division I field hockey for the Minutewomen as a midfielder. Since then, I've lived in New Orleans, Louisiana and Manhattan, New York. I spend my free time running, reading, perusing archival fashion, and searching for the world’s best negroni. I also have a debonair tuxedo cat, Westley, who's named after "The Princess Bride." ### Author social links [Bluesky]()