BMW Group USA has teamed up with Meta Platform’s Reality Labs division to develop an automotive virtual and mixed reality (VR/MR) solution. Both firms have created a platform for displaying XR content in moving vehicles for gaming, entertainment, and other experiences using immersive technologies.
The new system combines BMW’s sensor data with Meta’s tracking systems and technologies to create a “car-locked” immersive entertainment console. With the new system, passengers can watch films, play video games, meditate, and send messages on social media for other uses.
Merging in-vehicle experiences with the #digital world.⁰We’re working with @Meta @RealityLabs for research on XR #innovation to explore ways of integrating Extended Reality (XR) devices into future vehicles: https://t.co/2rFFy1bldr ⁰#BMWGroup #Meta pic.twitter.com/TXcGp9EhmK
— BMW Group (@BMWGroup) May 17, 2023
According to Claus Dorrer, Head of BMW Group Technology Office USA in Mountain View, it was “too early” to note when or how the technology would “make it into customers’ hands.”
Despite this, the company wanted to “envision a number of potential use cases for XR devices in vehicles,” Claus said. He explained that these would include driver assistance techniques like locating vehicles in car parks, alerting drivers to hazards, and displaying key vehicle data.
Dorrer continued,
“The implications of future AR glasses and VR devices — for passengers as well as drivers — are promising. The research partnership with Meta will allow us to discover what immersive, in-vehicle XR experiences could look like in the future and spearhead the seamless integration of such devices into cars.”
BMW’s M2 Mixed Reality Platform
The announcement comes after BMW developed its M2 Mixed Reality platform. The novel immersive system merges physical driving with real-time 3D (RT3D), interactive content.
At the Web Summit 2022 in Lisbon, Portugal, in November last year, the German automaker partnered with Varjo Technologies to race physical vehicles. However, drivers could seamlessly collect digital coins in the circuit using Varjo’s hand and eye-tracking tools.
The Helsinki-based XR company’s headsets, the most advanced on the market, combined real-time driving data from the M2 to deliver the experimental experience.
Meta-BMW Automotive XR Use Case
The proof-of-concept (PoC) prototyping aimed to solve several major technical challenges with camera and motion sensors. According to the group, the technologies struggle to keep pace with fast-moving vehicles, leading to “conflicting signals” across devices.
The company said: “Simply put, the headset’s motion sensors think it is moving, but its cameras think it is standing still when looking at the car’s interior.”
Due to the differences in positioning and relative movement, many VR headsets cannot display virtual content in vehicles. BMW explained that acceleration, quick turns, and road bumps create further difficulties.
Resolving the issue required BMW to leverage Meta’s technologies by developing sensor arrays on the Meta Quest Pro’s tracking systems. Doing so provided it with tracking of both the device and the vehicle, allowing them to position immersive content on the go accurately.
Entertaining, Comfortable Passenger Experiences
Richard Newcombe, Vice President of Research Science, Reality Labs Research, explained that the research prototype demonstrated the companies could create “entertaining and comfortable passenger experiences” linked to the physical car for its VR/MR offerings.
He explained further:
“The technology has the potential to transform how we can safely interact with our environment while traveling, and as we progress into reliable world-locked content on the road to AR glasses, we’re hopeful it will be possible for passengers to see things like markers for landmarks, restaurants, places of interest, and more”
The firms compared the vehicle’s movement data with sensory data from the XR headset. They did so by removing motion from the automobile and tracking the headset’s in-vehicle movement.
This created real-time synchronisation of both data sets, no matter how the vehicle operates on the road. With innovative sensor calculation technologies, this pushes the Meta Quest Pro’s six degrees of freedom (6DoF) capabilities to new levels.
With the recent use case with Meta, BMW Group has reiterated its support to its privacy-by-design principles. This protects customer data with strict privacy requirements operating across the company’s client markets.
Automotive XR Industry Experiments
The news comes after rival XR automotive entertainment firm holoride revealed it had created a portable, retrofitted device for vehicles. With the portable, chargeable device, users can access immersive entertainment on the go with interoperability across vehicles.
Unveiled at the CES 2023, the Audi subsidiary’s device is similar to an Amazon Alexa. It also functions for up to 14 hours on a single charge.
The lightweight, compact device is the size of an Amazon Alexa and can accommodate up to two headsets on any vehicle, charging with either USB-A or USB-C chargers.
Both retrofitted and stock versions of the device also include HTC VIVE Flow headsets for lightweight and optimal functionality.
Audi-Magic Leap Partnership
Audi Motors also unveiled its Activesphere concept car as an additional venture into XR. The futuristic vehicle leverages Magic Leap 2 headsets to display augmented reality (AR) content superimposed on the physical world.
Driving conditions, weather, and messaging appear in the Magic Leap 2’s 70-degree field of view (FoV) for fully integrative experiences.
Audi’s Design Studio’s latest innovation offers superimposed data in the driver’s view rather than windshields. This untethers immersive content from the car’s physical components, leading to full 360-degree situational awareness.
The joint venture created an MR visual interface with the Magic Leap 2’s 70-degree field of view (FoV). This superimposes driving data and road conditions for drivers in real-time, eliminating windshield dashboards or navigational units.
