AI Researcher Slams OpenAI, Warns It Will Become the "Most Orwellian Company of All Time"

"In the last few months, the mask has really come off."

Grave New World

OpenAI is far from its days of being an altruistic, non-profit company.

It is now the face of a booming AI industry and is effectively for-profit in all but name, steaming ahead with little regard for its technology's environmental toll, or for the potentially existential risks it poses to society.

But according to Gary Marcus, a cognitive scientist and prominent AI researcher, the worst is yet to come. In his assessment, OpenAI could soon take an even more dystopian pivot, à la George Orwell's novel "1984," by getting into the business of spying on you.

"My guess is that OpenAI is going to become the most Orwellian company of all time," Marcus said during a seminar at Stanford's Center for Human-Centered Artificial Intelligence last week, as quoted by Business Insider. "What they're going to be pressed to do is become a surveillance company."

Fast Cash

The reason that this will happen is simple: OpenAI will need a way to make money, Marcus said.

"OpenAI sold the dream of universal AI for all purposes, and in 2023 practically every big company ran pilot studies on that premise," Marcus later told BI. "But in 2024, a lot of reports from the field are about disappointment: the technology isn't reliable enough yet for production, because it is plagued with problems like hallucinations and boneheaded errors."

"So many businesses are starting to be cautious," he added.

That echoes what other analysts have said about the AI boom-slash-bubble (and indeed, what Marcus has harped on before). Despite the hype and massive amounts of investment, how these AI models will generate enough profits remains uncertain.

For now, the technology's obvious pitfalls and commercial shortcomings are being overlooked for the promise of it eventually being far more capable and transformative than it currently is.

A Date With Data

A startup like OpenAI going down the surveillance road wouldn't be unheard of. Surveillance, in some shape or form, is a sidestream of income for many big tech companies. Platforms like X-formerly-Twitter, for example, sell user data to third parties. Those third parties can be surveillance firms, who then sell their services to government agencies.

But as leaders of the generative AI industry — which is predicated on the mass consumption of data from all corners of the web — OpenAI will be uniquely positioned to deploy powerful surveillance capabilities of its own.

Some of the warning signs may already be apparent. This summer, for example, OpenAI controversially appointed a former director of the National Security Agency, Paul Nakasone, to its board.

Marcus once held out hope that OpenAI would course correct. But he's since grown cynical about its CEO Sam Altman's leadership, especially as the company drops the non-profit act.

"In the last few months, the mask has really come off, and I seriously doubt [Altman] will ever return to the mission," Marcus told BI. "The promise to investors to turn into a for-profit basically means that ship has sailed."

More on AI: Google Is Stuffing Annoying Ads Into Its Terrible AI Search Feature