VFX professional Kert Gartner has been pioneering new ways to make capturing virtual worlds as compelling as filming in the real world. His latest technique makes a real virtual camera out of an extra Vive controller and a smartphone.

You may have actually seen Gartners work before, he was a major part of the new Space Pirate Trainer trailer production, and the mixed reality Fantastic Contraption trailer. His latest trick is a little bit of genius that expands upon much of his prior work with VR cinematography.

To get the feel of ‘real world’ cinematography in VR, you need to be able to manipulate the virtual camera just like real world cinematographers manipulate real cameras. Often times that means handheld cameras that are up close to the action. Gartner has been emulating that handheld camera look by smartly using a third Vive controller to manipulate a virtual camera inside the virtual world. But filming this way is difficult because the controller doesn’t have a viewfinder, meaning that seeing what it’s pointed at requires looking at some distant monitor on the periphery of the of the action.

Now Gartner has solved that problem by using a smartphone to act as the viewfinder for the virtual camera, allowing him to see exactly what the virtual camera sees as he films:

The foundation of the technique requires a custom build of whatever VR experience is being filmed; one which is able to project multiple camera views in a single frame, including an angle which will be the view of a virtual camera which is “attached” to an extra Vive controller. Rendering out a 4k frame divided into multiple views means that each view can still be captured at a sharp 1920×1080 by cropping away the other views. Capturing all views at once gives the editor the ability to chose from multiple viewpoints of the same shot in post processing, without the cinematographer needing to do multiple takes from different angles.

SEE ALSO
'Max Mustard' Review – An 'Astro Bot' Style VR Platformer That Cuts the Mustard

virtual-reality-camera-vr-camera-htc-vive-1The next step is to affix a smartphone to the Vive controller. In this case, Gartner whipped up a simple rig with a vertical grip for easy capturing, but I imagine it would be almost as easy to connect the smartphone directly to the Vive controller (just don’t block the tracking receivers!).

From there, Gartner employs the jsmpeg-vnc utility which allows him to stream a selectable crop of the monitor to a web server which allows it to be viewed easily by visiting a URL on the smartphone. This gives him a viewfinder which shows exactly what the virtual camera sees.

virtual-reality-camera-vr-camera-htc-vive-4Altogether, the rig now works like a handheld camera, except instead of filming the real world, his viewfinder shows a portal into the virtual reality world. Even rendering multiple views at 4k and running the screen streaming and capturing software, Gartner says the whole system is still hitting the crucial 90 FPS figure (no doubt thanks to a powerful PC managing it all).

Praise to Gartner indeed for this smart solution, though many will know that technique isn’t brand new; the use of real-time digital visuals with a virtual camera and viewfinder has been seen in high-end film and game production, notably on Avatar (2009), Halo Reach (2010), and Uncharted 4 (2016) but Gartner’s system is tremendously less expensive, using just a PC, HTC Vive, extra Vive controller, and some streaming software; that puts it in the realm of practicality for many VR developers.

We expect to see this technique further developed for the production of high-end VR trailers, and even for big livestreaming VR shoots. Hooking the Vive controller and smartphone viewfinder up to a Steadicam rig could give a cool high-end film look. We can also imagine a more realistic virtual camera model offering complex adjustments like focal length, exposure, and manipulation of lens characteristics, all controllable in real time from the touchscreen of the smartphone viewfinder.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • MW

    Great idea and great work. Sadly, it’s
    just idea for now, because we have to wait (maybe many years) for hmd with good enough screen and real vr fov for real hi-end vr movies:-(

    • dogtato

      This is for filming in VR, not for viewing in VR. The end result will be a traditional rectangular video.

      • Theo

        could be quite cool to create some scenes and then allow a camera man to just film the action from where ever as they walk around. only problem is the limitations of real time i guess.

  • Sponge Bob

    trying to understand the utility of this….

    virtual filming inside virtual world

    like subsampling digital data for virtual world to produce some type of limited dynamic view into that virtual world

    I say – who cares ?

    and why smartphone ? it’s all computed predetermined digital data anyway so you can manipulate it and view any way you want on any device – in virtual world or in real

    • pj530ii

      It’s not useful for players but will be great for people trying to create “mixed reality” content. Aside from actually trying the games, mixed reality is currently the best way to show off a VR game. Having the camera man be able to see what’s actually happening in the game world (as opposed to just looking at a person flailing around in front of a green screen) will allow them to create better videos/trailers to show off games.

  • Philip Anderson