Scientists Reconstruct What You're Looking at by Enhancing Reflection in Your Eye

Wild TV forensics are starting to look more plausible.

Enhance!

We may soon owe Hollywood an apology for some of its ridiculous forensics.

Researchers at the University of Maryland have developed an eerie technique that can reconstruct 3D images from the reflections in your eyes, by building on a neural network model called neural radiance fields (NeRF).

On their website, the researchers display some of their impressive achievements, like computing a blurry-but-recognizable 3D rendering of a Kirby toy reflected in a subject's eye. They also managed to primitively render simple scenes, like a classroom.

TV screenwriters aren't vindicated yet, however. The photos they took are very high resolution — like from a really sharp smartphone camera — and were meticulously orchestrated. Another constraint? The lighting conditions need to be very, very specific.

A more ambitious attempt that uses footage from real-world sources better illustrates some room for improvement.

Admirably, the researchers tried to reconstruct what was reflected in the eyes of Miley Cyrus and Lady Gaga in their music videos, but at best, their results only turned up nebulous black shapes. One shape, they claim, sort of resembles an upper torso.

Cornea Conundrum

It's still impressive work, though, and very clever. According to a pre-print study about the project, most applications of NeRF require multiple moving cameras capturing a scene to generate a 3D rendering. This new technique only requires a single, fixed camera, and an eyeball.

The researchers took advantage of the geometry of the human cornea, which is "approximately" the same for healthy adults. That convenient fact allowed them to measure the number of pixels a cornea takes up in an image, and use it to position the eyes in space.

Accurately doing this, however, is a major challenge, and colorful irises that massively obfuscate reflections muddle things further. Though they have their methods to somewhat get around these obstacles, they're far from perfect.

So in other words, it's not quite the stuff of a "CSI" scene that can divine an eyeball reflection from horrible CCTV footage — but it might just be getting there.

More on hard science: IBM Says It's Made a Big Breakthrough in Quantum Computing