Microsoft has partnered with German automotive manufacturer Volkswagon to develop an AR (augmented reality) GPS navigation system powered by Microsoft’s HoloLens 2 mixed reality headset. The ‘Moving Platform’ feature, as it’s referred to by Microsoft, is designed to assist Volkswagon drivers by providing them with real-time navigation and safety information during their commutes.
“We think mixed reality information is the most intuitive information we could provide to enhance our customers’ user experience,” said Dr. Andro Kleen, head of the data science team at Volkswagen Group Innovation in an official blog post. “Because what you see there, and what you need to process, is very close to what humans normally see and process. It’s not so abstract.”
Apparently, Volkswagon has been experimenting with AR navigation for some time. In 2015, the company started work on a research project called ‘Race Trainer’ that combined AR technology with self-driving vehicles to teach drivers how to race. The system would use the HoloLens 2 headset to project virtual instructions like steering and braking cues over the real world as they drove. Unfortunately, the project stalled due to issues with tracking; as soon as the vehicle started moving, so too did the holographic imagery presented by the HoloLens 2.
“We had extensive discussions,” said Marc Pollefeys, director of Microsoft’s Mixed Reality and AI Lab in Zurich, Switzerland. “They presented their use cases and what they were hoping to enable. They were eager to work with us to find a solution and be able to use HoloLens in those situations.”
In 2018, Microsoft and Volkswagon partnered once again to solve this long-running conundrum. Pollefeys’ team developed an algorithm for the HoloLens 2 that takes into account the movement of the vehicle to deliver a comfortable AR navigational experience that doesn’t lose tracking. This process was far from simple, as Pollefey and his team were forced to work remotely from their respective apartments due to the COVID-19 pandemic restrictions in Switzerland.
“We had to do a lot of testing in my apartment. These aren’t ideal development conditions,” said Joshua Elsdon, a Microsoft senior software engineer who worked remotely on the project from his home. “All of this stuff was done remotely and distributed across different countries, which was interesting.”
After some initial prototyping, the team was finally able to conduct in-person testing alongside Volkswagon near Microsoft’s Redmond campus, first with recreational boats followed by automotive vehicles. The team used a 3D positioning system to track the location of the vehicle in relation to the headset, allowing them to anchor 3D information over the user’s real-world environment. For example, a live 3D map could be projected over the vehicle’s dash, allowing the user to check their current location without taking their eyes too far off the road.
“We connected a positioning system that tracks the location of the vehicle. This way we were able to also place 3D elements such as information on point of interests outside of the car,” said Michael Wittkämper, augmented reality expert at Volkswagen. “This opens up completely new possibilities to not only display holograms within the driver’s forward-facing field of view, but also wherever the user wearing the glasses is looking.”
According to Microsoft, the companies Moving Platform system has already drawn interest from various maritime organizations. The platform is currently available for use on large ships, though the company plans on expanding support to other moving environments in the future, including planes, trains, and elevators.
For more information visit here.
Image Credit: Microsoft
