Google Insider Unimpressed by Its Cutting Edge AI

"The biggest challenge I’m still thinking of: what are LLMs truly useful for, in terms of helpfulness?"

Help Wanted

Google set up a secretive Discord server for the most active users of its Bard AI — and their feedback, at times, has been provocative.

As Bloomberg reports, the invite-only chat room was established in July and amassed nearly 9,000 members over the ensuing months. While many of the exchanges cheerlead the Google AI, others in the digital community pointed out that it kind of, well, sucks sometimes.

"The biggest challenge I’m still thinking of: what are LLMs [large language models] truly useful for, in terms of helpfulness?" Cathy Pearl, a Google user experience lead who works on Bard, said in one August exchange that Bloomberg reviewed. "Like really making a difference. TBD [to be determined]!"

Even as Google pushes to integrate Bard into its other products, including Gmail, Maps and YouTube, users who take part in the Discord server have wondered whether all the money and work being put into the AI is paying off.

"Is any work being done on reducing the staggering resource costs of LLMs?" one of the Discord users asked, citing the billions of gallons of water used by Google and its "thirsty" peers to keep their AIs afloat.

“I kind of look at it like chip design… or supercomputers,” Pearl, the user experience lead, said in reply. "I believe we will continue to find ways to get the same behavior with less resources."

Make Believe

Even the heaviest users of Bard, the chat logs reviewed by Bloomberg demonstrate, are concerned about the accuracy of information the AI spits out, too — though the Google reps in the server had handy responses for those worries as well.

In one exchange, the official Bard account in the Discord server responded to a question about the AI "hallucinating" — a term used to refer to chatbots making stuff up — by, essentially, saying that it's bound to happen.

"We’ve done our best to make sure this happens as little as possible," the official account said during the September comment. "But since Bard is still learning and growing, it could happen."

"If Bard does hallucinate with any of the integrations," the account continued, "please let us know in the bug reports channel!"

In a statement to Bloomberg, Google spokesperson Jennifer Rodstrom defended the commentary that occurred in the Discord channel and said that the company has been "eager to hear people's feedback on what they like and how we can further improve the experience."

And, sure, that's fair. It's good to solicit feedback and encourage it to be candid.

But while concerns about AI accuracy and costs are nothing new, it is telling that these comments are coming not from outside detractors but from insiders who use Bard so much, Google decided to give them special access to an invite-only forum to discuss it.

If those folks are worried, maybe we all should be.

More on Google AI: Google’s Search AI Says You Can Melt Eggs. Its Source Will Make You Facepalm