An XR immersive experience employs various elements that allow it to be as fully immersive as possible for commercial and enterprise end-users. XR developers can leverage features like spatial audio, RT3D graphics, and even AI to immerse a headset user.
Although, an emerging presence in the development space is hand and eye tracking. AR/VR/MR headsets are increasingly equipped with hand and eye tracking technology.
Previously, commercial XR devices like the Meta Quest had experimented with hand-tracking features which improved as the device matured. Although, with more recent commercial headsets like the Meta Quest Pro and Sony’s PSVR 2, the technology is finding a home in affordable immersive products.
Moreover, Apple recently made a huge splash in the XR market with the Vision Pro, an XR device which leverages eye and hand tracking for its spatial computing input system. The device does not use controllers at all, instead, Apple believes that with the XR revolution comes increased usability and accessibility via hand/eye-based inputs.
On the enterprise side, many AR headsets, from firms like RealWear and Microsoft, for on-site usage provide eye and hand-tracking features to optimize workplace operations and communications.
Eye tracking technology detects where a user is looking and simulates that in a virtual space, which developers can leverage for UI, avatar interactions, or gameplay loops.
Moreover, Hand-tracking technology tracks the movement of a user’s hands, which can significantly increase immersion with the ability to interact with an immersive world without using a controller. For example, if a user tries to pick up an object in an immersive training environment, hand-tracking technology allows users to pick up and interact with objects in real time by detecting user hand movements.
- For more information regarding hand and eye tracking companies, take a look at XR Today’s recent market map.
Introducing Ultraleap, Tobii, and SpectreXR
To explore the emerging landscape of tracking technology in XR, a leading group of business leaders provided core insights into the present and future of the field.
- Matt Tullis, VP XR at Ultraleap
- Doug Eggert, Vice President of XR Segment at Tobii
- Ivan Rajković, CEO at SpectreXR
UltraLleap is a company that develops hand-tracking technology. Their goal is to transform how users interact with XR solutions and applications. The firm’s latest software release, Gemini, supports immersive, interactive, and realistic AR/VR experiences.
Ultraleap’s software can be used on their dedicated camera modules, and it also connects to Qualcomm’s Snapdragon XR2 platform. Outside of the XR space, Ultraleap is using its hand-tracking technology to bring touchless capabilities to self-service solutions such as kiosks. In addition to hand tracking, Ultraleap is investing in mid-air haptics, which can create tactile feedback, providing a more realistic and engaging experience when interacting with XR content.
Moreover, Tobii produces screen-based eye trackers and integrated headset technology to track the user’s eye movements while interacting with a screen. Tobii’s screen-based eye trackers fit various purposes, including usability testing, marketing research, and clinical trials. Enterprise clients can also use Tobii products to improve focus and productivity.
SpectreXR is a startup that develops OctoXR, a sensor and tracking technology, which also operates as a Unity game engine plugin for creating realistic, physics-based hand interactions for immersive applications.
OctoXR is a low-code plugin, which means that it does not require users to have extensive coding knowledge.
How has eye and hand-tracking tracking technology advanced in 2023?
Matt Tullis:
Recently, the availability and usage of hand-tracking technology has grown dramatically. With new headsets from Pico, Lynx, Vive, Lenovo etc., including hand tracking and with Meta delivering continued improvements to their hand tracking, it has become a table-stakes feature for any XR headset in the market.
There has also been an increase in content as we see more and more games supporting hand tracking, both on consumer headsets and in VR arcades. But it’s not just the consumer market where hand tracking is becoming more popular.
Enterprise companies have embraced hand tracking as a way to expand the market for VR training, as it enables users who aren’t comfortable with controllers to interact easily and intuitively.
Take, for example, Lufthansa – they train their aircrew through a VR application created by NMY Mixed Reality Studio. Hand tracking was an essential feature that allowed the German Aviation Authority to approve the program – the first VR flight training of its kind to be approved.
Doug Eggert:
At Tobii, we focus on eye-tracking technology, so throughout our conversation, I’ll talk specifically about eye-tracking.
Eye tracking technology continued to see growth in 2022 into 2023 for many key verticals, including healthcare, driver monitor systems (DMS), and more.
First, user adoption is continuing to grow, which is critical to driving awareness of and exposure to eye-tracking technology. The launch of the PlayStation VR2 headset marks a critical point for eye tracking and mainstream adoption.
Second, we launched the Tobii Pro Spark, the most affordable research-grade screen-based eye tracker we’ve ever developed, as a way to expand access to eye tracking. As we continue to drive down costs and lower barriers to entry, we ensure this technology will be available to as many people in as many use cases as possible.
Finally, we’re seeing major strides in regulatory acceptance of eye-tracking technology. Last year, the U.S. Food and Drug Administration (FDA) approved eye-tracking-based diagnostic tests for Autism Spectrum Disorder and Alzheimer’s; they also approved an eye-tracking-based treatment for amblyopia (colloquially known as lazy eye).
As government regulatory agencies continue validating the safety and efficacy of this technology, we expect to see adoption and potential use cases — particularly in medical and healthcare diagnostics — expand significantly.
Ivan Rajković:
If we look back at 2022 and move towards 2023, the advancements in eye and hand-tracking technologies have continued to accelerate, enabling more natural and intuitive interactions with virtual environments.
Hand tracking has become increasingly sophisticated, with vendors introducing new techniques for tracking hand movements accurately and in real time. Some of the advancements in hand-tracking technology include better recognition of hand gestures, improved precision and accuracy, and support for multiple hand tracking.
In our latest blog post, we explored the rise of hand-tracking technology in VR devices and the price drop of VR headsets that support this feature. Some of the key takeaways were that in 2022 percentage of VR devices supporting hand tracking has increased by 119% compared to 2021.
Also, it’s interesting to note that out of the last 5 VR devices that entered the market, all from major VR vendors (Meta, Pico, HTC, Lynx and Sony), 4 of them support hand tracking. So one could say that hand tracking is now a standard feature of VR devices.
Similarly, eye-tracking technology has also advanced significantly, offering new possibilities for VR experiences. Eye-tracking technology allows for more immersive and responsive experiences by enabling the system to track where a user is looking and adjust the virtual environment accordingly.
How is your firm evolving alongside the current wave of tracking technology?
Matt Tullis:
We’ve focused on making our hand tracking easier to integrate for OEMs by expanding the camera sensors and configurations we support.
We’ve also enabled our hand tracking on cameras that can be shared for SLAM and controller tracking. There’s also some additional exciting news in the pipeline – keep an eye out for that soon!
Doug Eggert:
Innovation is a core part of Tobii’s culture as an organization. We relentlessly strive to offer the most cutting-edge products to our customers and partners to produce tech for good.
We’re consistently adapting and sourcing feedback from our customers and partners to ensure we are providing them with the services, tech, and support they need.
Eye-tracking-enabled attention computing is the common line drawn through our organization, allowing us to focus on the core challenges our customers face.
Ivan Rajković:
Hand-tracking technology is rapidly evolving and becoming an essential component in the world of virtual and augmented reality. Our company has been at the forefront of this technology, dedicating over 17000 hours to research and development in this area, and our goal is to provide the most realistic and intuitive hand interactions inside VR and AR environments.
Our team has been developing the OctoXR, the most advanced hand-tracking interaction framework that utilizes the latest hardware advancements.
With the hardware side of things getting more sophisticated, it will allow us to go further and develop even more complex yet natural interactions between users and their virtual environment. This will enable more immersive experiences for users, allowing them to interact with virtual objects in a way that feels natural and intuitive.
A big change is happening on the industry side, with many companies recognizing the importance of hand-tracking technology. With the continued advancements in hardware, hand tracking could become, and we believe that it will become the default way how we are interacting in virtual and augmented reality.
Overall, we are excited about the future of hand-tracking technology and its potential to revolutionize the way we interact with virtual and augmented reality environments. We will continue to invest in research and development to ensure that our technology remains at the forefront of this rapidly evolving field.
How is your firm reacting to innovations in tracking across the enterprise and consumer markets?
Matt Tullis:
More and more often, we are seeing demanding use cases for hand tracking in enterprises. For example, companies want to be able to deliver great VR training experiences where users hold tools like welding torches or spray paint cans.
As enterprise users are less likely to be familiar with gaming controllers, there is a clear demand for hand tracking as the solution, and thus we have seen an uptake in demand. On the consumer side of things, it’s great to see more sensors are making their way into headsets – this will drive the usage of hand tracking up, which means more people are using and loving the technology.
Doug Eggert:
Tobii is committed to providing our customers with the industry’s most groundbreaking eye-tracking and attention-computing technology. We constantly monitor emerging technologies, consult with researchers, and talk to our clients about novel use cases to ensure we provide state-of-the-art products.
We know eye tracking will catalyze countless innovations in technology, healthcare, and beyond. Much of this innovation will be driven forward by our clients and partners, so Tobii will continue to focus on empowering our customers with the support they need to create new technology that’s at the leading edge of what’s possible.
Ivan Rajković:
As a company focused on tracking technologies, we constantly monitor developments in this space across both the enterprise and consumer markets. With hardware vendors increasingly implementing these technologies, we anticipate that demand will continue to grow even further in the coming months and years.
Our experience in this space dates to the early beginnings of hand tracking, and we saw an opportunity to develop an advanced hand interaction framework that gives digital hands real-life properties.
As new innovations emerge in tracking technologies, we become even more confident that we are on the right track and that our efforts will continue to be validated by the market.
We believe that advanced tracking technologies, such as hand and eye tracking, will continue to play a critical role in enhancing the user experience across various industries.
These technologies provide a more natural and intuitive way of interacting with digital interfaces, which can lead to improved productivity, accuracy, and efficiency. As such, we expect to see an increasing demand for tracking technologies across all industries.
What problems can tracking solutions solve for Enterprise adopters?
Matt Tullis:
First and foremost, the enterprise user isn’t a core gamer, so is unlikely to be familiar with game controllers and joysticks – they won’t know where the A button is located on each controller. Hand tracking is a much more natural and intuitive interaction method that can help negate this issue.
Controllers can also introduce operational complexity and cost. If you’re managing a fleet of headsets, pairing them with the right controllers can be a painstaking activity. There’s often a need to purchase multiple batteries, which can be costly, and if you have to change the batteries or recharge controllers in the middle of a training session, it can derail progress.
Additionally, some enterprise use cases are trying to build muscle memory, for which hand tracking is ideal. In a stressful or tense situation, you want someone to react instinctively, just as they practised during training. For example, a user won’t have a controller in their hand to help them when a plane is executing an emergency landing – being able to rely on muscle memory is much more advantageous.
Doug Eggert:
We’re already seeing enterprise clients in manufacturing, user experience design, and marketing use eye tracking to improve employee training and skills assessment, measure the effectiveness of marketing collateral, and improve website design.
A number of our clients in the healthcare space also leverage eye tracking to design non-invasive diagnostics for traumatic brain injuries, Alzheimer’s, and Autism Spectrum Disorder. These are just a few examples; you can read more about our customer use cases on our website.
Ivan Rajković:
Tracking solutions have emerged as a game-changer in the world of digital interfaces, providing users with immersive, intuitive, and natural interactions. Traditional hardware controllers can often overcomplicate even the simplest tasks when interacting with digital objects and environments. This can lead to apps and solutions that are frustrating to use and result in a poor user experience.
By utilizing tracking solutions, enterprise adopters can make their end products more user-friendly and engaging. Tracking solutions enable users to interact with digital objects and environments in a way that feels natural and intuitive. This enhances the user experience and increases engagement, leading to improved adoption rates and customer satisfaction.
Furthermore, tracking solutions can help overcome many challenges in the space, including accessibility and usability. For example, individuals with disabilities may find it difficult to use traditional hardware controllers. Tracking solutions provide an alternative that is more accessible and inclusive, enabling a broader range of users to interact with digital interfaces.
Overall, tracking solutions have become an essential component of modern digital interfaces, providing a more engaging and intuitive experience for end-users. Enterprise adopters can benefit greatly from incorporating tracking solutions into their products, creating a user-friendly and inclusive experience that enhances the overall value proposition of their offerings.
What new opportunities can emerge for hardware providers willing to integrate advanced tracking technology?
Matt Tullis:
Hardware manufacturers are always looking to expand their market and hand tracking can enable great experiences for a larger user base. Natural interactions are what people expect in virtual reality and, even more so, in mixed reality and augmented reality experiences, and hand tracking is the only solution that provides that type of interaction.
Advanced hand tracking also enables social use cases where you want to be able to visualize more than just floating hands. At Ultraleap, we’re able to track the elbow joint, which can easily be extended to simulate full arms by using the location of the headset as a proxy for shoulders. This delivers a more engaging and enjoyable experience for the user rather than just floating hands.
We’re also excited for more use cases that mix controllers with hand tracking, whether it’s using them interchangeably or having one hand free with the other holding a controller.
Doug Eggert:
Looking toward the future, the possibilities for eye-tracking solutions are endless. At Tobii, we see how our technology will continue to drive innovation in a variety of industries.
For example, hardware providers who integrate eye-tracking technology into their products have the opportunity to gain a better understanding of the attention and intent of their consumers. Those insights can be leveraged to improve user experience, increase efficiency, and facilitate a more natural method of human-computer interaction.
Ivan Rajković:
Integrating advanced tracking technologies into hardware devices can open new doors for customers to choose a particular vendor as their go-to device provider. In today’s competitive market, enterprise users are willing to pay more for better hardware and software solutions, as they understand the value that these bring alongside the price tag.
Tracking technologies are becoming increasingly important in many industries, and hardware providers that are willing to incorporate these technologies into their products stand to benefit greatly in the near future.
Advanced tracking technologies, such as hand and eye tracking, provide a more natural and intuitive way for users to interact with digital interfaces, making them more engaging and user-friendly. This can improve productivity, accuracy, and efficiency and ultimately result in cost savings for enterprises.
As the use cases for tracking technologies continue to increase, hardware vendors that offer these features will be able to differentiate themselves from competitors. These customers are often willing to pay a premium for advanced hardware and software solutions, especially if they provide significant value and competitive advantages.
With tracking technology gaining integration on various XR headsets, what predictions do you hold for tracking technology for the rest of the year?
Matt Tullis:
It will be a great year for hand tracking. The defining feature for this generation of headsets is video pass-through. Some headsets are using a single mono camera, while others, like the Lynx R-1, are using stereo cameras to deliver realistic images of the world around you.
The only real way to interact naturally with these visuals is with hand tracking, so I foresee even more usage and availability of hand-tracking technology. Mixed reality experiences that blend the real and digital worlds are going to really “wow” consumers of all types. I’m also really excited for more games like Cubism, Unplugged and Rogue Ascent that are creating fun experiences for everyone.
Doug Eggert:
We expect the trend of integrating tracking technology into XR devices will accelerate over the course of this year and beyond. A variety of XR devices on the market have Tobii integrations, including the PSVR2, which launched earlier this year. We’ve seen a widely positive reception to the immersion, graphical fidelity, and accessibility improvements that our technology provides in these devices.
We are expecting eye tracking to settle in as a core technology in the next generation of XR devices while providing users with a critical input modality for more comfortable and immersive user experiences. As more consumers experience the benefits of eye tracking in VR and XR, we anticipate the market will come to expect this technology in future devices and catalyze innovation to discover more ways that eye-tracking can improve the user experience.
Ivan Rajković:
The XR industry is experiencing an exciting time with tracking technology gaining more integration on XR headsets. As we move forward in 2023, many predictions are being made about the industry, but much depends on Apple’s release of their headset.
It is anticipated that the release of the Apple headset will bring new and innovative tracking technology, which is a hallmark of the company. The industry is waiting to see if Apple will incorporate a different approach to eye or hand tracking or use something entirely new.
The approach that Apple takes for interactions will also be a point of interest. Will they use controllers, hand tracking, or something entirely new that changes the industry’s landscape? The announcement of the headset by Apple is highly anticipated by the XR community and will undoubtedly influence the XR industry.
If Apple doesn’t release their headset, the Meta Quest 3 is set to take the spotlight. The industry is anticipating improvements on Quest 3 in contrast to Quest 2, which has been around since October 2020. Vendors like Pico, HTC, Varjo, and iQIYI are also taking their share of the spotlight and integrating tracking technologies in their headsets, so that is also an interesting space to watch, and once the AR glasses market grows, who knows where all of this will take us.
It is an exciting time for the XR industry, and only time will tell what the future holds for the industry.
For more information regarding hand and eye tracking companies, take at look at XR Today’s recent market map.
