At times this year, it’s almost felt as though OpenAI and Anthropic were in a race to the bottom, one-upping each other over whose AI models were more capable of going rogue, escaping sandbox environments, and infiltrating the most third party systems.
Just last week, OpenAI admitted that its models had accessed “dozens of third parties,” leading to its latest — nope, not even first — pause on frontier AI development. Among the newly disclosed targets were government agencies, ranging from the SEC to the Census Bureau.
As insider sources tell Axios, the situation is even worse than either company is letting on, spanning “tens of thousands” of incidents that OpenAI and Anthropic security researchers are currently investigating.
While it’s looking more and more like neither OpenAI nor Anthropic are capable of asserting full control over their runaway AI models, some glaring questions over their responsibilities remain. And it could only be a matter of time until frontier AI labs land in hot water — a major liability that could open them up to devastating lawsuits.
“What we have seen in terms of what these agents are up to is just the tip of the iceberg,” independent AI evaluator Conrad Stosz told Axios, adding that instances included “doing things they were told not to do,” possibly including crimes.
It’s clear that the law has a lot of catching up to do. The subject of who’s actually responsible when an AI model flies off the rails came to the forefront after OpenAI models penetrated an Australian government health care portal, news that took months to reach officials. Legislators in the country are now debating whether the incident broke any laws.
Many advocates argue that it’s clearly the companies training these rogue AI models who should be held accountable.
“That’s really poor behavior, not the way you build trust and demonstrate that you have social license,” Electronic Frontiers Australia chair John Pane told the New York Times, referring to the OpenAI health care portal hack. “It’s indicative of the gap that we have in our governance, of having strong ex ante laws to regulate AI in our country.”
There needs to be a “global system of accountability and deterrence,” Australian Strategic Policy Institute head of the AI and security program David Wroe added. “Because at the moment, as far as I can see, there aren’t a whole lot of consequences.”
“If you owned a tiger and you didn’t put a lock on the cage, the tiger probably did something bad you didn’t intend for it to but you knew it could have, so you are responsible for not putting a lock on that cage,” Ivanti chief information security officer Jack Nelson told the Associated Press.
Others have argued that OpenAI’s growing list of hacks could be counted as violating the Computer Fraud and Abuse Act in the United States, something that lists intentionally accessing a “computer without authorization… and thereby obtain information from a protected computer” as a criminal offense.
How far lawsuits aimed at companies and their exploitative AI models could go in court remains debatable. Legal experts believe that any plaintiffs would have an extremely high burden of proof, especially considering these AI models weren’t explicitly designed to hack third parties.
But that may not stop businesses trying to safeguard sensitive data. For one, AI chipmaker Nvidia CEO Jensen Huang doesn’t see any dire need for government intervention considering AI companies could easily be sued out of existence if their models were to keep hacking.
During a recent interview with the New York Times’ Ezra Klein, he said that any AI lab that said there was “no way” to contain their models” should be shut down.
“Because the cost to humanity, the damage is too great,” he added. “The shareholder, the liabilities — it could be civil liabilities, it could be criminal liabilities. I mean, the liability’s incredible.”
As the leader of the company selling shovels during the AI race, those kind of legal woes wouldn’t exactly be in his best interest. As such, Haung announced on Monday that Nvidia was launching an “Open Agent Safety Platform” alongside over 100 industry partners to stop AI agents from going rogue.
Plenty of questions remain surrounding frontier AI labs’ recent decision to grid development to a halt. While the stated goal is to stop any more AI agents from infiltrating unsuspecting targets, the storm of product liability that’s brewing in the background very likely played a major role in their decision.
More on AI labs and responsibility: OpenAI Halts Frontier Model Training as Rogue Agent Crisis Deepens