A wristband with four thermal sensors promises a more detailed hand tracking experience.
The minds over at Cornell University’s SciFi Lab, along with help from the University of Wisconsin-Madison, have come up with a unique wristband device that uses four small (9.30mm) synchronized thermal cameras to deliver more precise VR and AR hand tracking experiences.
The device is called FingerTrak and looks a little bit like a souped up version of Spider-Man’s web shooters. Instead of firing webs, however, this device captures your hand movements and then transfers that data to a tethered Raspberry Pi-powered device connected to a computer. That computer then runs a deep learning model which can predict all 20 finger joint positions.
Each thermal camera tracks the movements of the top, bottom, left, and right part of your hand and then meshes each image into a single stitched thermal image in real-time. Because FingerTrak uses thermal images to track your hand movements, the system is capable of more precise hand tracking when compared to options such as the Leap Motion Controller or the Oculus Quests.
To announce their advancements in hand tracking technology, the SciFi Lab released a white paper along with a video detailing their work on its thermal camera tracking device. In the video, you get a POV of the user’s bare hands as the individual handles day-to-day activities such as flipping through a notebook and writing a journal entry, holding a cup of coffee, and picking up their mobile device.
Even subtle hand movements are capable of being tracked. Low-resolution (32 x 24) thermal cameras are able to estimate your bare hand posture by tracking the outline of your hand silhouettes.
The video also shows how FingerTrak gives you the ability to control a robotic hand in real-time. As the user moves their fingers, makes a fist, and closes and opens their hands, the robotic hand mirrors each and every movement in real-time. This kind of technology could prove incredibly useful in a variety of dangerous situations, such as defusing a bomb, exploring dangerous environments, or even providing basic at-home care remotely.
According to Sci-Fi Lab, however, there’s still plenty of work to be done before their product is ready for mass production. Similar to most other forms of consumer hand tracking technology, FingerTrak is unable to provid any form of haptic feedback, making it difficult to complete tasks that rely on you being able to “feel” the world around you. Another issue is that FingerTrak has a small angular error rate that fluctuates between 6.46 to 8.06 degrees. While that may not sound like a big deal, this can make a world of difference during activities that require precision, such as cutting specific wires on a bomb.
If you’re interested in learning more about FingerTrak, The SciFi Lab will be presenting its work at the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing in September. You can also access their white paper here.
Image Credit: Sci-Fi Lab
