The software, called GeoSolver, combines the machine vision needed to understand diagrams as well as the ability to read and "understand" questions. It answered geometry questions from the SAT as well as an average 11th grader. It generates a series of logical equations from the text and diagram in the question, then weighs the accuracy of its guesses and decides whether it's close enough to choose a multiple-choice answer.
Although the AI is impressive, it does underscore how far machines have to go before they can mimic human intelligence. One example the researchers provide is that it can't reliably understand the meaning of an arrow in a test diagram. One issue is that the researchers used relatively sparse datasets -- even all of the standardized questions together don't make up a large data set.