“Minority Report” Science Adviser On the Future of Human-Computer Interactions

Time For a New User Interface

It’s now more than 30 years since the debut of the Macintosh computer. A river of innovation has formed and re-formed the landscape since then (laptops, the internet and the web, smartphones, iPads, the ‘cloud’), but the basic user interface (UI) introduced by that first Mac remains unchanged. It's odd to think that the system designed to allow people to operate a computer in 1984—a single screen with a single mouse and a single cursor, two-dimensional graphics, one user at a time, buttons and scrollbars, etc.—is still what we use today.

The situation deserves analysis because the UI is hardly unimportant; it's the essential bridge between human and machine.

How is it that an interface scheme older than a large chunk of the present-day workforce still holds sway? Partly it's a testament to the sturdy design and general-purpose approach of the language Steve Jobs and team adapted from earlier Xerox PARC work. And Silicon Valley might contend that we haven't needed anything new—that the current UI is working just fine, and there are much more important matters to attend to.

But today's machines are capable of ridiculously more than that primordial Mac, and we humans have always been capable of infinitely more than that Mac UI lets us express. So there's an enormous opportunity around new UI. We need to push this important language forward: add vocabulary, expand the syntax, evolve the grammar. Really, one of the most important things we could take on as a technological species might just be building a new UI.

It's time.

In fact, the need for a radically more capable UI is urgent. The world is increasingly complex, and its problems (but also discovery and invention around its new potentials) demand more sophisticated thinking, tools, and patterns of work. All these forms of activity are already digitally mediated, which means that today's inadequate UI is a liability, an impediment. Tomorrow's will be an amplifier for the aspects of human ability most directly involved in high-value work.

The Making of a Better Technology

A second crucial provision will be support for interaction not just within small or finite pixel spaces (in the way that a modern tablet's UI is trapped inside its own little screen) but equally across and around extended pixel spaces, in which screens and displays turn corners to fill several walls while also interoperating with personal device displays. Matching the literal scale of display and interaction to the notional scale of a problem or workflow is transformative to human process.

For designers, creating this successor UI is the most intoxicating opportunity imaginable. The design task goes beyond computer science, beyond pure technology, to draw from fields as diverse as architecture, biology, and cinema. Engineering and design must be alloyed from the start. The consequence of such expansive thinking will be profound: we'll know we're getting there when the resulting systems feel more like the human world than the machine world, and when using them feels and once natural and visceral, exhilarating.

The economic implications are even more significant. The new UI will generate fresh commercial opportunities, reinvigorate a broad swath of professional activity, and give direct and indirect rise to new industries. This is because a proper next-gen human-machine interface goes beyond doing the same things faster. It enables wholly new things: building the previously unthinkable, analyzing what was before impenetrable, and understanding the otherwise too vast or knotty.

A new UI isn’t a wild proposition; nor is it distant or hypothetical. Curiously, the movie world has done real work in substantiating the value of such a thing. Consider the indelible scenes in Minority Report in which Tom Cruise and his team, surrounded by display surfaces, use a gestural, collaborative interface to sift through mountains of data and extract enough intelligence in real time to avert murder. The depiction of that UI is vivid and specific enough to let us evaluate what it'd be like to work that way.

Real-world versions of this kind of technology are already in use at forward-thinking organizations such as NASA Aeronautics, IBM, Boeing, and Accenture. For these companies, such systems catalyze innovation and engagement, and are qualitatively changing how their teams work and communicate. And the benefits of a newly powerful interface apply as fully to smaller organizations of every kind, and in consumer and domestic contexts. At all levels this new UI will create access, spark opportunity and idea exchange, and play a central role in solving critical problems.

New UI is a universal revolution. The time is now.

John Underkoffler holds a PhD from the Massachusetts Institute of Technology and is founder and CEO of Oblong Industries. Oblong’s technological and design trajectories build on fifteen years of foundational work at the MIT Media Lab. He has been science advisor to films including ‘Minority Report’, ‘The Hulk’ (A. Lee), 'Aeon Flux', and ‘Iron Man’.