Mixed Reality is one of the newer concepts to emerge in the Extended Reality environment for games developers. Promising an opportunity to combine virtual content and real-world experiences like never before, the MR landscape could represent a new frontier for gaming.
At the heart of this new environment is Microsoft, one of the world’s leading technology companies, and the group behind the phenomenal “Hololens” headsets. Exploring the ability of MR to immerse users not just in visual experiences, but environments enhanced by 360-degree sound, the Microsoft HoloLens Experience Team produced something new for the gaming world.
Here’s what happened when Microsoft worked on the creation of spatial sound and audio experiences for the RoboRaid game.
Discovering New Opportunities with Sound
According to Microsoft, Spatial sound is one of the most important features in the MR world, capable of helping users perceive what’s going on around them, even when objects aren’t in view. When it came to producing a new mixed-reality first-person shooter (RoboRaid), the company wanted to take full advantage of the opportunities of spatial sound.
For RoboRaid, the most effective way to use spatial sound was as a method of alerting players to something happening beyond their peripheral vision. For instance, an enemy can enter the room from any of the scanned walls in the environment. If you’re not facing in the direction at the time, the threat is easy to miss. However, with the use of spatial audio, Microsoft could alert users to where the “Breacher” was coming from, so they could act fast to stop an invasion.
For Microsoft’s team, creating a spatial sound experience for the gaming industry was a unique and new experience, complete with its own set of novel challenges. For instance, 360-degree audio can be demanding on CPU performance. For something as fast-paced as RoboRaid, it was crucial to keep the number of spatial sound instances to less than 8 at any given time.
Usually, this process was as simple as setting the limit of instances for various audio events. Any instances happening after the limit was reached were instantly removed. This helped to create an MR experience that was immersive, without being overwhelming. It also ensured resources could remain free for various spatial sound events, from drone attacks to enemy explosions.
Pulling Users Deeper Into the Game
The dodging mechanic in RoboRaid was an important feature in the title, and something Microsoft believed to be critical to the full HoloLens experience. To make the interaction feel more rewarding, the company used the Doppler “whizz-by” sound when users dodged an incoming attack.
Before committing too many resources to the project, Microsoft created a simple prototype to see how it would feel. After experimenting with different options, Microsoft decided to eliminated unnecessarily complex code in favour of a simpler, prototype solution.
Indeed, much of the development for the game was based on trial and error. For instance, the team originally wanted to play an explosion behind the player after they dodged an enemy projectile. However, the group eventually abandoned this idea after discovering it would lead to a multitude of too many complex sounds in one space. The additional sounds also would have placed a significant strain on CPU performance.
Another interesting issue which emerged in the development of the game was the difficulty involved in communicating when a player had been hit. Without haptic feedback, players wouldn’t feel anything when they were hit by an enemy, so Microsoft needed to explore other options.
The team found the ideal solution was to combine a visual cue with an exaggerated sound to indicate a user had taken damage.
Generating Big Sounds from Small Speakers
To suit the lightweight structure of the device, the speakers on the HoloLens headsets are relatively small. This meant the sound designers needed to be mindful of the frequencies of the audio in the game. To ensure compatibility with the HoloLens speakers, the team consistently ran tests to experiment with how sounds would play back for users in mixed reality.
The developers found it was necessary to add another layer to some of the bass sounds, one octave higher and mixed with rich harmonics to give a consistent audio experience. They also discovered distortion could amp up the harmonics and engage the brain on another level. Notably, the team felt low-end sounds weren’t always the best solution for giving a player a sense of weight and impact. Distortion elements gave the audio a sense of “crunchiness” which helped boost the experience.
While many of the developers on the team wanted to leverage loud guns, powerful music, and crazy explosions, they also knew how important it was to ensure users could hear voiceovers and other in-game audio cues. While console games have more opportunities for dividing up frequencies based on the importance of each sound, MR titles work in a different way.
For RoboRaid, the team was limited in the number of frequencies available to carve from sounds. In order to make the gaming experience sound as “big” as possible on a small device, the group lowered the dynamic range of the whole experience. They also leveraged ducking to create a clear hierarchy of which sounds were most important. The team spent significant time tuning the fade in and out timing and volume attenuation of the audio. They even set up separate busses for non-spatial and spatial sounds, voice overs, and dry bus minus reverberation for the music.
The overall finished result was a highly immersive gaming experience, enhanced by the value of spatial and 360-sound solutions. Microsoft’s experiments with RoboRaid offered a fantastic insight into the various elements involved in bringing gaming opportunities into the mixed reality world. As today’s developers can clearly see, there’s more to a great game than the visuals alone.
