Alongside having a wide field of view, virtual reality headsets are distinct from regular 3D displays in that they are tracked. PC VR, console VR and now even some standalone headsets even have positional tracking so you can lean, duck, and even walk around in VR.

But how does this tracking work? Here we’ll explain the main positional tracking systems and how they function.

Background

What Is Positional Tracking? (6DoF)

Most mobile VR headsets like Oculus Go, Samsung Gear VR and Google Daydream View only have rotational tracking (3DoF). You can look up or down, to either side, or tilt your head. But if you try to lean or actually move your head’s position, this is not tracked. The entire virtual world will move with you. 3DoF controllers are similar, rotation-only. They essentially act as laser selection pointers.

This can be acceptable for seated content, but it doesn’t allow you to move around the virtual world physically, or to interact with it with your hands directly.

Image from Aniwaa

On PC and console VR, and now in high end standalone VR, the headsets feature positional tracking (6DoF). Positional tracking lets the user actually move around in the virtual environment. If the tracking volume is sufficient, you can even walk around your whole room in VR.

When controllers are also 6DoF, you can directly interact with virtual objects with your hands since you can move them through virtual space by moving your hands in the real world.

Tracking Systems

Rotational tracking (3DoF) is always done with microscopic electromechanical gyroscopes. But different companies use different technologies to enable positional tracking (6DoF). While there may be a common industry standard some day, none exists yet. Companies have different ideas about which techniques are right.

The various tracking systems each balance cost, ease of setup, tracking volume, controller tracking range, and modularity.

The Common Base: Dead Reckoning

Contrary to popular belief, the optical systems described below are only truly “correction” systems. The primary shared tracking method of all these systems is a microscopic electromechanical accelerometer. These accelerometers typically run at 1000 Hz.

Diagram from Newcastle University

Here’s how this works: accelerometers do not read position, or even velocity, they read (as the name suggests) acceleration. But as you may remember from calculus, you can take the integral of acceleration over time and get velocity. And if you take the integral of velocity values over time, you get position (or at least, displacement from the original position).

Using this to determine change in position is called dead reckoning. From moment to moment it’s how every VR headset and controller tracks itself.

So why is anything else needed at all? Because accelerometers are imperfect, providing noisy data. Integrating this data twice means even the smallest error is magnified, and this error accumulates. In reality, this means that accelerometer based positional tracking drifts to infinity within a matter of seconds.

The purpose of VR tracking systems is to correct this drift by providing a reference. Each tracking system does it differently, but the purpose remains the same.

Tracking Systems

“Constellation” (Original Oculus Rift)

When designing the Rift, Oculus faced the challenge of delivering the same quality of tracking as multi-thousand dollar systems like OptiTrack at orders of magnitude lower cost.

Each tracked device has a pre-defined “constellation” of infrared LEDs hidden under the external plastic, which you can see highlighted in the image above. IR light is invisible to the human eye.

Sensors, which are basically cameras with filters to see only IR light, send frames to the user’s PC over a USB cable at 60 Hz. The PC processes each frame, identifying the position of each IR LED, and thus the relative position of each object.

The software can easily recognize which LEDs it’s seeing because it knows the shape of the “constellation”, it remembers where the object was in the previous frame, and it knows its direction of acceleration (from the accelerometer), and its rotation (from the gyroscope). Each IR LED also blinks at a specific frequency to identify itself. These innovations gave Constellation a leg up over past marker-based tracking systems.

To support fast motion, the Rift headset and Touch controllers wirelessly communicate with a wireless chip in the sensor every time they’re about to pulse their LEDs. This allows the camera shutter to fire exactly as the LEDs do, and allows the exposure to be short.

Constellation tracking is no longer used for headset tracking for the Rift S, but it is used on the headset onboard cameras to track the controllers.

Constellation Advantages:

• Low cost to integrate
• High quality tracking
• Works in most environments

Constellation Disadvantages:

• Each sensor has wired connection to PC
• Large USB bandwidth causes issues with many motherboards
• Sensors have limited vertical field of view

PlayStation VR

PlayStation VR uses cameras too, but unlike the Rift the PSVR’s tracking operates in the visual light spectrum.

The PlayStation 4 camera bar contains two spaced out cameras. The camera unit is connected to the PlayStation, which uses the image data to track the blue strips of light on the headset and the orbs of light on the controllers.

This allowed Sony to leverage the existing PlayStation Move controllers from back in 2010.

PSVR Tracking Advantages:

• Low cost to integrate
• Leverages existing PS Move controllers

PSVR Tracking Disadvantages:

• Low quality tracking
• No room scale support

SteamVR “Lighthouse” (HTC Vive)

Valve’s SteamVR “Lighthouse” system is the most unique currently on the market. Unlike all the other systems it does not use cameras at all, and the PC does not process any data. Lighthouse was designed from the start to enable room scale positional tracking without having to wire sensors back to the user’s PC.

Base stations (“Lighthouses”) are placed at opposite top corners of the room. They do not communicate with the PC and they are not sensors. They emit wide angle two dimensional IR laser beams across the entire room. This is done 1 axis at a time, so left-right then top-bottom, repeatedly. Before each sweep they emit a powerful IR flash of light.

Each tracked device contains an array of IR photodiodes connected to a chip. This chip measures the time between the IR flash and being hit by the laser sweep for each axis. From this it can determine its position in the room.

The unique value of Lighthouse is that it is a relatively simple design in theory. No complex computer vision algorithms or camera readings are involved, just the timing between laser sweeps.

Lighthouse Advantages:

• No PC connection to base stations required (just power),
• High quality tracking,
• Wide tracking volume.

Lighthouse Disadvantages:

• Relatively expensive to produce / integrate,
• Usually requires mounting base stations on walls (otherwise the motors jitter),
• Reflective surfaces in room cause glitches.

SLAM / Inside-Out

After the release of the Oculus Rift and HTC Vive, a lot of VR companies began to realize that the requirement to mount sensors or base stations in the room was putting many potential buyers off.

Many newer and upcoming headsets use cameras built into the headsets themselves which perform “inside-out” tracking using computer vision algorithms. The specific type of algorithm used is called Simultaneous Location And Mapping (SLAM).

SLAM algorithms work by noticing unique static features in the room. By comparing the rotation and acceleration from the accelerometer & gyroscope to how these features appear to move, the position of the headset can be determined.

Several companies have released or are planning to release headsets using SLAM:

• Microsoft: Every Windows MR headset
• Google: Lenovo Mirage Solo
• HTC: Vive Focus
• Facebook: Oculus Quest and Oculus Rift S

Google calls their tracking algorithm ‘WorldSense’ and Facebook calls theirs ‘Oculus Insight’.

To track controllers, these systems operate in a similar way to Constellation — either with visible or infrared light. The cameras on the headset track LEDs under the plastic on the controllers.

Advantages:

• No external hardware required,
• Very low cost,
• Trivial setup,
• If you could see your hands in real life, the headset can track the controllers.

Disadvantages:

• Doesn’t work in the dark,
• Can’t track controller movement behind head or back accurately,
• Can’t track controller movement when arm is between headset and controller.

This concludes our break down of the various different types of tracking systems out there. Let us know your thoughts and questions down in the comments below!