Sony Establishes Virtual Production Collaboration Centre

Sony, Lux Machina reps discuss emerging VP tools for film production teams and the steps required to introduce the on-set solution

7
Sony Establishes Virtual Production Collaboration Centre
Augmented RealityNews Analysis

Published: May 22, 2023

Rgreenerheadshot

Rory Greener

Virtual Production (VP) technologies are a rising solution for film production professionals. The technology field leverages immersive hardware and software stacks, enabling on-set professionals to project realistic virtual environments as backdrops with immersive visualizations viewable by a human’s and a camera’s eye.

VP solutions look to replace or enhance pre-existing film production operations such as green screens and in-vehicle backdrops. The emerging XR tools allow studios to realistically project a real-world location without needing to travel or transport resources to the site – saving time, money, and CO2 emissions massively.

Moreover, VP tools can project a completely digital environment, such as a sci-fi planet ready for actors to see. VP walls also project light via LEDs for realistic and dynamic lighting based on an RT3D environment.

Sony’s VP Tools

Sony’s Crystal LED display technology and Venice camera capture technology create a unified RT3D backdrop solution which projects immersive and reactive visuals with the firm’s high brightness, wide colour gamut, and high contrast ratio promise.

Sony customers also leverage the widely adopted Unreal Engine, a powerful real-time 3D engine that can create immersive environments. Sony’s Virtual Production Toolsets professionals import and display real-time 3D environments behind a subject using Unreal Engine, allowing filmmakers to see the environment they are shooting in before they start filming.

Introducing Sony’s DMPC Institutions

Sony hosts two learning institutions the firm built for industry professionals and students. Sony’s Digital Media Production Centre Europe (DMPCE) provides a learning space to leverage VP tools for industry experts at Pinewood Studios in the UK, the home of many leading film productions from the likes of Disney and more. Moreover, Sony has a DMPC in Los Angeles to provide VP knowledge in the US.

During Sony’s VP showcase event at the DMPC suite at the famous Pinewood studio in Buckinghamshire, England, XR Today spoke to David Grey, Managing Director at Lux Machina and Claus Pfeifer, Head of Connected Content Acquisition at Sony, to discuss the role of introducing VP tools to industry professionals. Moreover, the duo spoke on how the technology can improve on-set, efficacy, collaboration, and creativity while helping to manage green and monetary outcomes.

Establishing the DMPC and VP Tools

Claus

Presented today is the combination of our Venice cinema camera and crystal LED screens. Both have been created specifically for the movie industry. For example, by creating the colour science between these two tools, you can easily use them together to create the maximum quality in your productions.

David

We’ve been working with Sony for a little bit now in collaboration, looking at virtual production workflows. We’ve helped create the system integration for the DMPC here, but also in Los Angeles and, the bigger picture, looking at what tools are useful on set. What’s actually happening in the world, and how concerning? Then create the best tools for filmmakers and technicians and everyone else involved.

Attacking On-Set Pain Points

Claus

So what we experienced was that there are a couple of pain points when producing or creating for virtual production. One pain point is a foray that appears simply by moving the camera too close to a screen.

You physically cannot avoid that completely or remove this completely, but you can avoid this by making sure that you are planning early and creating your set so that it is not shown.

So that was one idea behind the Virtual Production Toolset was to create a virtual Venice camera that can be loaded inside Unreal Engine. So you have this combination of the virtual world with the real world. Because you’re using a virtual Venice camera in the virtual world, this allows you to plan ahead [for using a real-life Venice camera].

The plug-in for the Virtual Production Toolset will also be free of charge when it is released in the summertime. Start to create your own content, and if you then are happy with the content, put this into the real world by going to a [Sony LED] volume and then simply reusing the setting that you have created [virtually]

The second paint point is colour consistency. By definition, with the same team working on Venice and Cyrstal LED, [consistency between shots] is very close, but still, there could be some certain differences. This is why we created the tool to match the colours. So again, we are supporting the industry by bringing down maybe 2 hours or 3 hours that you easily spend on matching the camera to the screen.

David

I think democratization of technology is essentially what we’re looking at, and that’s always important in getting the tools into more users’ hands like Unreal Engine, which is, you know, the backbone of a lot of 3D in-camera effects. [Unreal Engine] is a beast which is inherently a game engine, not a film production tool.

So if you can make it more comfortable for, say, a DOP. That translation layer is super useful, and that’s a big part of what we do on set. Put tools back in the hands of the people.

If you got a gaffer with a lighting desk, put him in control. Let him use his traditional tools to access Unreal Engine. That’s a translation layer that will help people get comfortable with a beast of a tool.

Leveraging Photogrammetry Scanning for On-Set ROI

Claus

So you can create your own scanning rig. We have done this by using a little turntable with one camera and creating small 3D objects you can put into a virtual space. But obviously, you can do that bigger by using full rigs.

Then take the real object, a real prop, and put this into the virtual world. Again [matching] the consistency of colours, the consistency of the quality of the capture between the real world and the virtual world is very important.

For example, we have one filmmaker that is doing a lot of commercials for toys. So they usually have a tabletop environment where they build up a little world with an LED screen behind them. So they would have something in the foreground, scan this and bring it into the background. Both of them need to match, obviously. So for them, it’s very important to move between the virtual and real world easily and quickly, and that’s what we are supporting.

David

It’s a difficult piece of the puzzle which isn’t spoken about often; anything we can do to make that content creation easier, whether it’s scanning an object or using the [Unreal] marketplace, it’s only going to make it easier for more people to get hands on the technology.

I think when people start to take a much more holistic, virtual production approach, that’s where you’ll see the cost-benefit. We’re still in a world where people are used to making decisions as late in the process as possible – “fix it in post.”

“Fixing it in post” is still a massive thing, right? But I think we need to have a bit of a mindset shift back to before visual effects days, where you made decisions about what you will see in camera[/on-site]. If you can spend a bit more time with your key talent making those decisions, then even you only save, say, 5 percent, 10 percent on the production. That’s obviously a huge amount of money, but it will take people who want to commit to that workflow and see the benefit of it to get to that point.

VP to Establish New Jobs and Opportunities

Claus

It will be very interesting for future filmmakers and young students because they can combine things and know-how that was not combined so much before.

Obviously, you still need a cinematographer that knows about the camera, knows about images, and knows about composition. Still, at the same time, you have people potentially coming from the game industry that have a completely different skill set.

Getting these people together is very interesting, but at the same time, you need to train them because of the game industry [influx]. People don’t know how the reality is on set and vice versa. So for us, it is very important as Sony to bring this know-how together. Enabling the community, young filmmakers grow, understand, and develop [VP tech]. So that’s one of the reasons why we have established this DMPC.

David

In terms of roles, there’s a whole new department on set now: the virtual art department.

On set, we’ve got, depending on the scale of production, as few as three people for a simple 2D playback job like car plating or 15 people on a AAA production. So having 15 new people on set with completely new roles is a big thing. We’ve got the virtual production supervisor, who is a translation layer. They know about unreal, they know about film production, they know about visual effects.

Environmental Concern

Claus

OK, so obviously, you need to, first of all, understand and check where CO2 emission is coming from. In our studies, most of the CO2 emissions come from travelling. Getting whole crews of 20/30/40 people, including all the equipment, over to a certain remote location is going to be very expensive and not positive for your carbon footprint.

Also, a volume uses a lot of energy, but compared to travelling, this is a lot less. At the same time, bringing up a set and then tearing it down the next day for nothing, only for the shoot [also consumes a lot of energy].

Also, there are good and bad when bringing everything to a virtual world, I don’t want to say that everything needs to be virtual. There are benefits of going to locations and getting the feeling for it. Telling a different story because you’re on location.

But not everything has to be on location. You can do things where you are. What is necessary? What can you do at home?

Sony’s behind supporting this, making it more feasible and commoditizing this so it can be used for everybody.

David

The beauty of this technology is that it’s also from the touring event world, right? So we can actually go to productions and build it for them. Sony has done that Greener Pictures report, which put some facts against this because I think there’s a lot of conjecture in virtual production.

More on Sony’s VP Push

To learn more on Sony’s Pinewood VP showcase, please refer to XR Today’s interview with Sony’s Content Acquisition Solutions Specialist Daniel Listh, who discussed how VP tools and integrated software could save production companies money and time.

EntertainmentEventImmersive CollaborationVisualization

Brands mentioned in this article.

Featured

Share This Post