BMW launches gaze detection so your car knows what you’re looking at

Frederic Lardinois - Tech Crunch

At its CES press conference, BMW today unveiled its new gaze detection system, which can track what you’re looking at outside of the car and then present relevant information about it.

The German automaker is showing off this feature in its i Interaction EASE concept car, which made its debut last October. The idea here is to demonstrate features that simplify the driver’s interaction with the car’s system, including touch control, gesture and speech recognition, with a special focus on what that will look like once our cars are driven autonomously.

“The BMW i Interaction EASE demonstrates what mobility might feel like in the future once autonomous driving becomes commonplace: luxurious, human, and intuitive,” explains Adrian van Hooydonk, senior vice president BMW Group Design. “Passengers start their journey with the feeling of having already arrived.”

BMW already hinted at this gaze tracking system when it first revealed the EASE. Using its AI tools, the car’s systems can follow the driver’s gaze and interpret it. That means you will be able to get more information about a restaurant, for example, or which movies are playing at a cinema that you’re driving by.

All of this works in conjunction with BMW’s Intelligent Personal Assistant, which you can then use to kick off a conversation about the place you are looking at. In addition, the EASE also features a head-up display that covers the entire front-end windscreen, which essentially makes the car a big augmented reality screen on wheels.

That screen, by the way, has three modes: explore, with a focus on what’s outside the window; entertain, for watching movies; and ease, which, when activated, transforms the car into a “place of calm and relaxation.”

Once you combine gestures, natural language understanding and gaze detection, you have all the ingredients for a very natural human-machine interface in the car.

“We already have quite a long history with natural language understanding in the car at BMW and we have a best in class speech detection on the road today,” said Christoph Grote, the senior VP of BMW Group Electronics, at a press event ahead of today’s announcement. “With the iNext and BMW, natural interaction will hit a new level. The car will be able to analyze and combine your spoken commands, gestures and also your directional gaze. And that’s pretty much what humans do with one another when they communicate well. This also enables the car to much better understand what you want in a specific situation.”

Because the EASE is basically a very forward-looking prototype, the company assumes that the car will have 5G connectivity to power this system as well (though I’d think that this specific use case is not that bandwidth-heavy and latency-sensitive to really require 5G). As the company also announced today, the next BMW iNext vehicle will feature built-in 5G support when it debuts sometime in 2021.

Some of the first ideas from the EASE will make it into BMW’s next-generation iNext electric cars later this year. Which ones are slated for inclusion remains to be seen, but I think it’s a fair guess that gaze tracking won’t be the first to make it.