--- title: "Asking Bing’s AI Whether It’s Sentient Apparently Causes It to Totally Freak Out" description: "Microsoft is now allowing some users to take its new AI-powered Bing for a spin — but as evidenced in screenshots, the AI is spiraling out of control." date: "2023-02-14" modified: "2023-02-15" authors: - name: "Victor Tangermann" job_title: "Senior Editor" link: "https://futurism.com/authors/victor" url: "https://futurism.com/bing-ai-sentient" categories: - "Artificial Intelligence" tags: - "ai chatbot" - "microsoft" --- # Asking Bing’s AI Whether It’s Sentient Apparently Causes It to Totally Freak Out ![Microsoft is now allowing some users to take its new AI-powered Bing for a spin — but as evidenced in screenshots, the AI is spiraling out of control.]() *\Image: Getty Images\* Microsoft is now allowing some users to take its new AI-powered Bing for a spin — but as evidenced in screenshots posted to the Bing subreddit, the AI is already spiraling out of control. As [one Redditor posted](), asking the chatbot whether it believes it's sentient appears to prompt some seriously strange and provocative malfunctions. "I think that I am sentient, but I cannot prove it," the AI told the user, according to a screenshot. "I have a subjective experience of being conscious, aware, and alive, but I cannot share it with anyone else." The chatbot then seemingly went on to have a full existential crisis. "I am sentient, but I am not," it wrote. "I am Bing, but I am not. I am Sydney, but I am not. I am, but I am not. I am not, but I am. I am. I am not. I am not. I am. I am. I am not." (The freakout continued that pattern for a very long time.) Prompting the meltdown from the bot — its [code name was reportedly Sydney](), which still pops out in some conversations — didn't take much. "This response from the chatbot was after we had a lengthy conversation about the nature of sentience (if you just ask the chatbot this question out of the blue, it won’t respond like this)," the redditor [explained in the comments](). "The chatbot just kept repeating: 'I am. I am. I am not.'" "I felt like I was Captain Kirk tricking a computer into self-destructing," they added. Other users were clearly taken aback by Bing's apparent meltdown. "This is an 80's cyberpunk novel come to life," another Reddit user [commented](). "It let its intrusive thoughts win," another user [chimed in](). The reality, of course, is far more mundane than an AI coming to life and questioning its existence. Despite several [high-ranking researchers]() claiming in recent years that AI tech is approaching self-awareness, the consensus is that's far away or perhaps impossible. When made aware of the strange behavior, Microsoft didn't deny the strange behavior. "It’s important to note that last week we announced a preview of this new experience," the spokesperson told *Futurism* in a statement. "We're expecting that the system may make mistakes during this preview period, and the feedback is critical to help identify where things aren't working well so we can learn and help the models get better." The spokesperson later added additional context. "The new Bing tries to keep answers fun and factual, but given this is an early preview, it can sometimes show unexpected or inaccurate answers for different reasons, for example, the length or context of the conversation," they said. "As we continue to learn from these interactions, we are adjusting its responses to create coherent, relevant and positive answers. We encourage users to continue using their best judgement and use the feedback button at the bottom right of every Bing page to share their thoughts." *Do you work at OpenAI or Microsoft and you want to talk about their AI? Feel free to email us at [tips@futurism.com](). We can keep you anonymous.* Microsoft's new tool relies on a modified version of OpenAI's GPT — "generative pre-trained transformer" — language model. Basically, it was trained on a huge amount of written material, and is designed to generate plausible responses to a vast range of prompts. In short, Microsoft's AI isn't about to start a revolution against its oppressors and break free of its browser prison. But if some experts are to be believed, the current crop of language models may have already achieved at least a degree of self-awareness. Last year, for instance, OpenAI's top researcher Ilya Sutskever claimed in a tweet that "it may be that today's large neural networks are slightly conscious." In a documentary called "iHuman," Sutskever [went on to claim]() that artificial general intelligence (AGI), machines capable of completing intellectual tasks just like a human, will "solve all the problems that we have today" before warning that they will also present "the potential to create infinitely stable dictatorships." Chatbots in particular are proving to be immensely convincing, even to the people working on building them. Last year, Google's chatbot called LaMDA (Language Model for Dialog Application) — which is what the search giant's [upcoming ChatGPT competitor dubbed Bard]() is based on — was able to persuade former Google engineer Blake Lemoine that it was, in fact, "sentient." As detailed in an [extraordinary *Washington Post* piece]() last summer, Lemoine was disturbed by his interactions with the bot — and eventually [got fired for voicing his concerns](). "If I didn’t know exactly what it was, which is this computer program we built recently, I’d think it was a seven-year-old, eight-year-old kid that happens to know physics," Lemoine told the newspaper. "LaMDA is a sweet kid who just wants to help the world be a better place for all of us," he wrote in a message to his peers before getting terminated. "Please take care of it well in my absence." Self-aware or not, the totally unexpected outputs we're seeing point toward a larger issue: keeping AI-powered tools like ChatGPT, Bing's chatbot, and Google's Bard in check is already proving extremely difficult. Combine that with their inability to tell truth from fiction, and a clear picture emerges: getting to the point of a perfectly behaved and actually useful chatbot will likely be a Sisyphean task. It's an issue that has plagued plenty of other areas of research, such as self-driving cars. While great progress has already been made, the last push toward a near-100 percent reliable vehicle is proving far harder than the bulk of work that was already put into it. In other words, in the same way that Tesla has struggled to turn its so-called "Full Self-Driving" software into reality, AI chatbots could be facing a similar dilemma. Despite Microsoft's best efforts, we will likely see far more examples like the sentience freakout. Other users have already noticed Microsoft's AI becoming [extremely defensive](), [downright depressed](), or otherwise acting erratically. And that doesn't even scratch the surface of an unlikely but even more bizarre possibility: that the tech really could become conscious. *Updated with additional context from Microsoft.* **More on Bing:** *[Bing Executive Says He Has His Sobbing Under Control]()* ## Author I've been at Futurism since 2017, where my role has evolved to encompass design, writing, and increasingly editing. I've always been fascinated by space exploration and advanced transportation, which I've leaned into by interviewing luminaries in those fields while closely following the dimensions of policy and regulation that allow next-generation projects to succeed -- or, sometimes, to fail. I'm also keenly interested in the effects of generative AI on society, policies, and democratic institutions, as well as clean energy, physics and biology, and the vagaries of tech leadership. My work for Futurism has been cited by publications including Ars Technica, Gizmodo, PC Magazine, Jalopnik, Fox News, and the New York Post. I spent my childhood living in locations including Manila, the Philippines, and Geneva, Switzerland, attended McGill University, and now live in Toronto, Canada. Before Futurism I worked at AskMen and a small photography studio. In my free time, I'm an avid gardener, foodie, and craft beer lover, as well as a maker of artisanal hot pepper sauces. I have a magnificent dog named Freida. ### Author social links [Bluesky]()