2017 has pushed the VR industry forward in countless ways, including the recognition of audio as an absolutely critical element in VR experiences.
Some big-time VR pieces like the Gorillaz’ Saturnz Barz (Spirit House) 360 video or Lytro’s Hallelujah experience have focused on experimenting with auditory experiences. Sound isn’t just a cue used to nudge the auditory experience in incremental ways — it’s something that can totally change the VR adventure or even affect the entire storyline.
Here are a few trending efforts that creators are using to push the envelope with sound.
The industry will embrace object-based audio for every kind of experience.
Just as a 360 camera captures an entire 3D scene from one point, an Ambisonics microphone is used to represent an audio snapshot of a 360 scene at a given time. Ambisonics signals are great for transmitting general ambiance, but they can be less than perfect for accurately pinpointing individual sounds, a shortcoming that has aggravated audio engineers for quite some time.
Starting earlier this year, more audio engineers began to use a mix of object-based audio and scene-based audio to create more realistic listening experiences. Essentially, object-based audio refers to an individual sound captured with a lavalier mic, while scene-based audio refers to the general sphere of sound captured with an Ambisonics mic. In physical reality, you hear sounds that are close or important to you with great detail (object-based audio), while everything else is perceived as background sound (scene-based audio). It’s only a matter of time until this combination of object and scene audio, and not exclusively Ambisonics, will be the industry standard for VR audio workflows.
Utilizing object-based audio also gives more creative freedom to content creators since it’s easier to manipulate post-production effects on a single sound — think of it as a single raw element as opposed to a big, messy sound glob. In addition, object-based audio works perfectly for 6DOF (six-degrees-of-freedom) VR content, which is rapidly growing in popularity.
6DOF content is just like a game — the character moves around within the space in every direction and has the agency to interact with objects in the environment. When the character does either of these things, the sound needs to change accordingly. Because it is better at pinpointing sound and easily reflecting the changes during gameplay, object-based audio has actually already been used in 3D game engines for quite some time. As more 6DOF content is being built on game engines, it’s plausible that more audio engineers will be forced to learn how to mix and master sound in game engines rather than their traditional Digital Audio Workstations.
Quality VR content will be published with more players embracing spatial audio.
Spatial audio is key in delivering lifelike VR experiences. Spatialized sound sources have three-dimensional positional data to make listeners feel like sounds are actually coming from those visual locations. As of right now, YouTube and Facebook are the only well-known player platforms that currently support spatial audio.
These limits on publishing platforms have discouraged content creators from fully embracing spatial audio in their productions. Still, as sound is met with increased appreciation, renderers or players will eventually have to support spatial audio. When Vimeo launched Vimeo 360 in March to support 360 content, a huge amount of the requests from users involved a desire for a spatial audio feature and the official help page states that they are planning to support spatial audio in the near future. Smaller players and platforms will follow the path laid out by Facebook, YouTube, and Vimeo. As user standards for VR content quality continue to rise, adoption for spatial audio will race to keep pace.
Out of the many reasons that have kept content publishing platforms from adopting spatial audio, the primary one has been the absence of a dedicated VR audio format along with a compatible renderer. With more emphasis being put on object-based audio, Ambisonics alone will be phased out as the standard format of the future.
In an attempt to unify spatial audio workflows, GAO format was introduced early this year to deliver superior sound localization and quality for VR audio. Developed by LA-based spatial audio company G’Audio Lab, GAO combines object and Ambisonics signals as well as stereo signals.
“We’ve seen a lot of people watch our demo pieces that start in regular stereo format. When they get to the part where the sound is spatialized and delivered in Ambisonics format, they’ll usually start to smile. Then when they watch the GAO portion, a lot of people will involuntarily let out a little laugh from being so pleasantly surprised.” said Brooklyn Earick, the director of business development at G’Audio Lab. Since GAO comes with a compatible renderer SDK, adopting spatial audio will be easier than ever before for platforms of all sizes.
Creators will push beyond post-production to create new listening experiences.
VR audio is already close to achieving accuracy when matching visuals with their corresponding sounds. The next step will be exploring exciting new techniques during post-production (or even earlier) in order to leverage the full spectrum of possibility exclusive to VR.
Sound is not just a storytelling cue that can be used to encourage VR users to look in a certain direction. In some new use cases, you are actually able to hear certain sounds over others within the same experience if you want. The following recorded 360 video, for example, lets users hear what they are looking at more clearly than the other instruments placed all around them.
This new wave of sound won’t just be part of an evolution of current techniques. In many cases, it will give way to revolutionary new forms of entertainment. The virtual canvas for artists is expanded 360 degrees horizontally and 360 degrees vertically beyond the physical dimensions of a stage in real life. Musicians will now be able to play with “virtual location,” along with their traditional considerations of pitch, loudness, and timing. They’re also learning how to exploit human auditory perception to influence these experiences at an even deeper level. Some psychoacoustic principles that you have already experienced in real life can be taken advantage of in VR to make each experience different at the individual level. While there is a whole lot to consider in that realm, our collective knowledge about it continues to grow.
Advances in VR audio are going to be key in perfecting the experience for mainstream consumers and for giving pioneers the tools they need to drive innovative experiences. These three trends will go a long way toward achieving both goals and will help shape the industry in the years to come.
