Expert Slams Waymo Safety Data: "They Obviously Do Not Trust These Numbers"

Top of the Class

On Wednesday, California's Department of Motor Vehicles released its annual report of self-driving car safety data, detailing how often autonomous vehicles created by companies like Uber, Tesla, Apple, and Waymo disengage their cars' self-driving mode and give control back to a human safety driver to avoid a crash or help the vehicle when its software gets confused.

The report shows a clear winner: on average, Waymo's self-driving cars drove autonomously for an eyebrow-raising 11,154.3 miles before a human took over — more than twice as far as GM, the second place finisher.

The number is impressive, but it but doesn't match the experience of Waymo passengers in Arizona, interviewed by Futurism, Ars Technica, and The Verge, who report frequent interventions from human drivers. And four experts cautioned us that the new figures in California might not tell the whole story — nevermind that the continued presence of safety drivers shows the vehicles aren't quite ready for prime time.

"I think the proof is in the pudding," Duke University engineer and autonomous vehicle expert Mary Cummings told Futurism. "Waymo is not driving without a safety driver either in California or in Arizona, even though they legally could, so they obviously do not trust these numbers either."

Disengagement Rates

When Futurism contacted a Waymo spokesperson to ask about the incredibly low disengagement rate, the spokesperson sent a press release explaining the company's progress in teaching its autonomous vehicles to handle unusual scenarios that might have previously tripped up a vehicle's artificial intelligence.

Udacity founder and Stanford University engineer Sebastian Thrun told Futurism that Waymo's self-reported disengagement rate was "totally feasible."

But these disengagement reports aren't the rigorous analyses that they seem to be at first glance, according to Wired, as each autonomous vehicle company collects and reports their own data, which are then compiled and compared side by side, even though each company may test using different metrics and standards.

A spokesperson from California's DMV told Futurism that the government reviews each company's data before publishing the annual report.

"The DMV carefully reviews all disengagement reports from manufacturers permitted to test autonomous vehicles in California," the spokesperson told Futurism. "When necessary, the DMV requests supplemental information and follows up with the permit holder to discuss our regulations and address concerns and clarify the contents of the report, as well as learn more about the company’s testing method and data collecting procedures."

When asked if the DMV had specifically verified Waymo's claim of over 11,000 miles driven per disengagement, the spokesperson said that the DMV's main focus has been making sure that each company filed a complete report.

And experts cautioned that the companies may not have been totally transparent about their tests.

"To be frankly honest, without knowing when or why the safety driver disengaged the self-driving capability or the context of the miles traveled (rain, snow, highway, city, etc.) it is difficult to interpret what 11,000-plus miles actually means," Lionel Robert, a self-driving car expert from University of Michigan, told Futurism. "This will continue to be a problem until the federal government steps up and standardizes some of these measures. Unfortunately, at this time we can just guess and that's not what we should be doing."

At press time, the Waymo spokesperson hadn't clarified the extent to which the government reviews disengagement data before publishing its annual report.

Take it For a Spin

"11,000 miles is feasible in carefully constructed environments with extremely detailed mapping and under low uncertainty cases, included no bad weather, typical traffic conditions, etc.," Cummings said. "But this metric can be quite deceiving since we have no idea of knowing what these conditions are."

Waymo's reported test results could show that the company excels in areas where others continue to struggle. They could also be the result of running self-driving cars on empty, well-paved roads during the daylight.

"Yes [Waymo's report] seems high, but then you have to look at the fleet size and what all the vehicles are doing," Jonathan P. How, an MIT aeronautics engineer and self-driving car researcher, told Futurism. "If many are driving in well-known relatively simple domains, then it might skew the results high. But I don't know much about why they might be so much higher."

Editor’s Note 2/14: This article has been updated to include an additional statement from California's Department of Motor Vehicles.

READ MORE: A New Robo-Car Report Card Isn't Quite What it Seems [Wired]

More on Waymo: People Are Road Raging At Waymo's Self-Driving Cars