New deep-learning method teaches computers to "see" what humans do in a typical day

Towards Full Activity Intelligence

Researchers from the School of Interactive Computing and the Institute for Robotics and Intelligent Machines came up with a new technique that teaches computers to "see" and understand what humans do in a typical day. The process involved a wearable camera that took 40,000 pictures every 30 to 60 seconds, over a period of six months. The result was a prediction with 83 percent accuracy of what activity a person was doing. Researchers taught the computer to categorize images across 19 different activity classes. The test subject wearing the camera could review and annotate the photos at the end of each day in order to make sure they were properly categorized. "This work is about developing a better way to understand people's activities, and build systems that can recognize people's activities at a finely-grained level of detail," said Edison Thomaz, co-author and graduate research assistant in the School of Interactive Computing. "Activity tracking devices like the Fitbit can tell how many steps you take per day, but imagine being able to track all of your activities - not just physical activities like walking and running. This work is moving toward full activity intelligence. At a technical level, we are showing that it's becoming possible for computer vision techniques alone to be used for this."

Deep-Learning

The researchers believe they have gathered the largest annotated dataset of first-person images to demonstrate that deep-learning can understand human behavior and individual personal habits. "People liked that we had a method that combines time and images," says Daniel Castro, a Ph.D. candidate in Computer Science and lead researcher on the project. "Time can be especially important for some activity classes. This system learned how relevant images were because of people's schedules. What does it think the image is showing? It sees both time and image probabilities and makes a better prediction." The ability to literally see and recognize human activities has many potential uses, such as improving personal assistant apps like Siri, understanding links between health and behavior, or simply improving personal choices throughout the day.