An AI Watched 600 Hours of TV and Started to Accurately Predict What Happens Next

Shows included The Office, Desperate Housewives, and Scrubs.

7. 12. 16 by Cecille De Jesus
Carl Vondrick/MIT CSAIL

Get Smarter with TV?

MIT’s Computer Science and Artificial Intelligence Laboratory created an algorithm that utilizes deep learning, which enables artificial intelligence (AI) to use patterns of human interaction to predict what will happen next. Researchers fed the program with videos featuring human social interactions and tested it to see if it “learned” well enough to be able to predict them.

The researchers’ weapons of choice? 600 hours of Youtube videos and sitcoms, including The Office, Desperate Housewives, and Scrubs. While this lineup may seem questionable, MIT doctoral candidate and project researcher Carl Vondrick reasons out that accessibility and realism were part of the criteria.

“We just wanted to use random videos from YouTube,” Vondrick said. “The reason for television is that it’s easy for us to get access to that data, and it’s somewhat realistic in terms of describing everyday situations.”

They showed the computer videos of people who are one second away from doing one of these four actions: hugging, kissing, high-fiving and handshaking. The AI was able to guess correctly 43% of the time compared to humans, who were right 71% of the time.


Potential Future

Giving AI the ability to understand visuals the way humans can could be a precursor to what would be efficient home assistants, as well as intelligent security cameras that could call an ambulance or the police ahead of time.

While this isn’t the first attempt at video prediction, it is the most accurate thus far. The reason is that, first, the new algorithm deviates from previous attempts at video predicting, wherein pixel-by-pixel representations were a priority. It predicts using abstract representation and focuses on the important signs: it learns on its own and uses “visual representations” to discriminate between visual cues that are important in social interactions from those that are not. It’s something that comes naturally to humans, but is far more complicated in AI.

“It’s not hugely different from some other things that people have done, but they’ve gotten substantially better results out of it than people have in this area before,” says Pedro Domingos, a machine learning expert and professor at the University of Washington.

Care about supporting clean energy adoption? Find out how much money (and planet!) you could save by switching to solar power at By signing up through this link, may receive a small commission.


Share This Article

Keep up.
Subscribe to our daily newsletter to keep in touch with the subjects shaping our future.
I understand and agree that registration on or use of this site constitutes agreement to its User Agreement and Privacy Policy


Copyright ©, Camden Media Inc All Rights Reserved. See our User Agreement, Privacy Policy and Data Use Policy. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with prior written permission of Futurism. Fonts by Typekit and Monotype.