Sony Interactive Entertainment (SIE) researchers unveiled a prototype motion controller recently which showcases experimental techniques into full hand tracking using machine learning and capacitive sensors.

Vision-based hand tracking systems like Leap Motion are great for tracking hands when you’re not holding anything like a motion controller, which can typically occlude finger movement, therefore breaking the illusion that your hands are really 1:1 represented in virtual reality.

In an effort to bring greater hand presence to VR motion controllers, SIE researchers Kazuyuki Arimatsu and Hideki Mori set out to build a prototype controller which integrates a host of capacitive proximity sensors—much like the ones you’ll find under the hood in Valve’s Index controller (aka ‘Knuckles’) and Oculus Touch.

Image courtesy SIE

The prototype houses 62 electrodes which Arimatsu and Mori used to record training datasets from a number of different hand sizes and and poses. Like the Index controller, Sony’s prototype features an open-palm design using a wrist strap.

“We have introduced 2.5D representation to apply convolutional neural network methods on a capacitive image of the curved surface, and two types of network architectures based on recent achievements in the computer vision field were evaluated with our dataset,” the paper states. “We also implemented real-time interactive applications using the prototype and demonstrated the possibility of intuitive interaction using fingers in VR applications.”

The video (above) shows off some pretty impressive results, however Arimatsu and Mori admit there are a few drawbacks to their design currently, namely its lack of support for all hand sizes.

SEE ALSO
PSVR 2 Firmware Update Includes New Support for NVIDIA GPUs, Pointing to Wired PC VR Connection

“This could be problematic if we deliver our method to users worldwide. Per-user fine-tuning training [demonstrated in the video] cannot be conducted with our current training method realistically since we require an optical tracking system for recording the training dataset. For better adaptation for various hand sizes, a method for augmenting or synthesizing the capacitive image needs to be studied.”

Image courtesy SIE

The current algorithm can’t estimate the shape of a user’s hands, such as the thickness of the finger, which can cause a gap between the actual hand pose and virtual hand pose when performing things such as pinching actions. Users can of course self-adjust when picking up smaller virtual items, although this has admittedly been true for Knuckles as well as the original Oculus Touch controller, which was the first major VR controller featuring a comparatively smaller array of capacitive sensors.

Whatever the case, it’ll be interesting to see if Sony continues forward towards commercialization, be it via Sony-built PC solution or controller intended for use with PS5. We’re still waiting with bated breath on an announcement of the next-gen PSVR, which we really hope ditches the aging PS Move and PS Camera optical combo to say the very least.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Well before the first modern XR products hit the market, Scott recognized the potential of the technology and set out to understand and document its growth. He has been professionally reporting on the space for nearly a decade as Editor at Road to VR, authoring more than 3,500 articles on the topic. Scott brings that seasoned insight to his reporting from major industry events across the globe.