Oxford Scientists Have an AI That Can Read Your Lips

This tool could be used to help the hearing impaired.

11. 9. 16 by Eleazer Corpuz
Image by Futurism/VC

“The First Sentence-level Lipreading Model”

Lip reading is a way of understanding speech by interpreting a person’s lip movement. However, human speech is highly complex and nuanced, where one lip movement could correspond to different phonemes, or basic units of sound. Therefore, the practice is prone to errors, which can sometimes lead to humorous results.

Scientists from Oxford University have described an artificial intelligence system, called LipNet, which can accurately read lips. The system employs deep learning to train itself using 29,000 three-second-long videos labeled with captions.

previous system read lips on a word-to-word basis where it was taught to associate a phoneme with a certain lip movement. It achieved an accuracy of 79.6 percent. LipNet on the other hand, works on whole sentences at a time, achieving an accuracy of 93.4 percent. When compared against human lip readers who scored an accuracy of 52.3 percent, LipNet was 1.78 times more accurate than them in translating the same sentences.

More Tools For The Hearing-Impaired

While the accuracy of the system is impressive, it is still not perfect. The videos fed to it had ideal lighting and the speaker front-facing the camera. The results may vary when done with a less ideal video, resulting in some skepticism about the results of the research.


However, the technology does show promise, and scientists are looking for applications for this technology. For example, it could be used as a tool for the hearing-impaired. Other technologies solve this problem in different ways like taking advantage of sensory substitution.

According to OpenAI’s Jack Clark, getting this to work in the real world will take three major improvements: a large amount of video of people speaking in real-world situations, getting the AI to be capable of reading lips from multiple angles, and varying the kinds of phrases the AI can predict.


Care about supporting clean energy adoption? Find out how much money (and planet!) you could save by switching to solar power at UnderstandSolar.com. By signing up through this link, Futurism.com may receive a small commission.


Share This Article

Keep up.
Subscribe to our daily newsletter to keep in touch with the subjects shaping our future.
I understand and agree that registration on or use of this site constitutes agreement to its User Agreement and Privacy Policy


Copyright ©, Camden Media Inc All Rights Reserved. See our User Agreement, Privacy Policy and Data Use Policy. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with prior written permission of Futurism. Fonts by Typekit and Monotype.