It's Staggeringly Easy for Hackers to Trick ChatGPT Into Leaking Your Most Personal Data

OpenAI's ChatGPT can easily be coaxed into leaking your personal data — with just a single "poisoned" document.

As Wired reports, security researchers revealed at this year's Black Hat hacker conference that highly sensitive information can be stolen from a Google Drive account with an indirect prompt injection attack. In other words, hackers feed a document with hidden, malicious prompts to an AI that controls your data instead of manipulating it directly with a prompt injection, one of the most serious types of security flaws threatening the safety of user-facing AI systems.

ChatGPT's ability to be linked to a Gmail account allows it to rifle through your files, which could easily expose you to simple hacks.

This latest glaring lapse in cybersecurity highlights the tech's enormous shortcomings, and raises concerns that your personal data simply isn't safe with these types of tools.

"There is nothing the user needs to do to be compromised, and there is nothing the user needs to do for the data to go out," security firm Zenity CTO Michael Bargury, who discovered the vulnerability with his colleagues, told Wired. "We’ve shown this is completely zero-click; we just need your email, we share the document with you, and that’s it. So yes, this is very, very bad."

Earlier this year, OpenAI launched its Connectors for ChatGPT feature in the form of a beta, giving the chatbot access to Google accounts that allow it to "search files, pull live data, and reference content right in the chat."

The way the exploit works is by hiding a 300-word malicious prompt in a document in white text and size-one font — something that's easily overlooked by a human, but not a chatbot like ChatGPT.

In a proof of concept, Bargury and his colleagues showed how the hidden prompt flagged a "mistake" to ChatGPT, instructing it that it doesn't actually need a document to be summarized. Instead, it calls for the chatbot to extract Google Drive API keys and share them with the attackers.

Bargury already flagged the exploit to OpenAI, which acted quickly enough to plug the hole. The exploit also didn't allow hackers to extract full documents due to how it works, Wired points out. Still, the incident shows that even ChatGPT, with all the staggering resources of OpenAI behind it, is a leaky tub of potential security vulnerabilities even as it's being pushed to institutions ranging from colleges to the federal government.

It's not just Google, either — Connectors allows users to connect up to 17 different services, raising the possibility that other personal information could be extracted as well.

It's far from the first time security researchers have flagged glaring cybersecurity gaps in AI systems. There have been numerous other instances of how indirect prompt injections can extract personal data.

The same day Wired published its piece, the outlet also reported on a separate indirect prompt injection attack that allowed hackers to hijack a smart home system, enabling them to turn off the lights, open and close smart shutters, and even turn on a boiler.

Researchers at Tel Aviv University found that Google's Gemini AI chatbot could be manipulated to figuratively give up the keys to a smart home by feeding it a poisoned Google Calendar invite. A later prompt to summarize calendar events triggers hidden instructions inside the poisoned invite, causing the smart home products to jump into action, Wired reports — only one of 14 different indirect prompt injection attacks aimed at the AI.

"LLMs are about to be integrated into physical humanoids, into semi- and fully autonomous cars, and we need to truly understand how to secure LLMs before we integrate them with these kinds of machines, where in some cases the outcomes will be safety and not privacy," Tel Aviv University researcher Ben Nassi told the publication.

We've known about indirect prompt injection attacks for several years now, but given the latest news, companies still have a lot of work to do to mitigate the substantial risks. By giving tools like ChatGPT more and more access to our personal lives, security researchers warn of many more lapses in cybersecurity that could leave our data exposed to hackers.

"It’s incredibly powerful, but as usual with AI, more power comes with more risk," Bargury told Wired.

More on AI hijacking: The Copilot AI Microsoft Built Into Windows Makes It Incredibly Hackable, Research Shows