By now, you've probably already seen Andy Serkis and his performance capture demo for Magic Leap. And sure, marrying motion capture acting and augmented reality sounds compelling, but how does it really work as an experience?
A few days ago, I had a chance to find out, thanks to The Imaginarium Studio's set-up in Los Angeles during the L.E.A.P. conference. That's right, I met Grishneck, as well as a few of his friends.
After putting on the Magic Leap One, I was directed to watch a flat screen TV monitor that has Grishneck trapped inside, begging for release. Seconds later, that screen dynamic moved to augmented reality, and then Grishneck jumped from that virtual screen onto the real table in front of me, all while continuing to beg me to take him home and save him from Hollywood obsolescence.
- Don't Miss: Magic Leap's Chief Content Officer Rio Caraeff Opens Up on Walled Gardens, Data Privacy, & Blockchain
The experience is clever, but since it's on rails, it's not really something I haven't seen before. But Grishneck was just the teaser. Just a few feet away, I was introduced to a motion capture actor wearing an array of sensors covering a black bodysuit. Nearby, a monitor displayed the actor as Magic Leap's astronaut character, which mirrored her every move.
However, that TV monitor was just for onlookers, the real fun began when I put on another Magic Leap One and suddenly saw a new character — a pixie elf. That character also mirrored the motion capture actor's movements, but instead of on a 2D screen, the character was standing right in front of me in 3D. In addition to the pixie elf, I was able to change the character to Grishneck, the Magic Leap astronaut, and a rock creature.
A flick of my thumb on the Magic Leap One's controller and I was able to scale the rock creature up to a massive 10 feet tall. I told the actor what I was seeing and asked her to circle me menacingly. It was great. And it felt as though the creature was towering over me in real space as the actor followed my every move in real time.
I can imagine a number of ways this combination of motion capture and the Magic Leap One could be used, but instead of guessing, I instead spoke to the CEO of The Imaginarium, Matthew Brown, to find out where he thinks this will take us in the very near future.
Next Reality: What is this all for? It's amazing, but what, specifically, will you be using this set up for?
Matthew Brown: It's a content creation tool. We can, through performance capture, authentically offer performances. As opposed to key frame animation, which is what you see in a lot of tradition VFX films, our actors are able to essentially puppeteer digital characters. And as part of the capture process, with your performers, you're able to get as close as possible to the final version of the performance. What we're excited to do is take that workflow and our knowledge of performance capture to a new platform.
Next Reality: While interacting with one of your performance capture actors, I asked her to pretend that we were facing off for a fight so that I could see what it would look like for the virtual rock monster in front of me — via the Magic Leap One — to look as though it was about to attack me. Is part of the goal of this set up to offer more realistic ways for actors to perform alongside digital characters instead of just having them imagine monsters against a green screen?
Brown: Yeah, that's completely possible. And with the tech that you just demoed, we could have an actor in our studio in London and the viewer can be anywhere on the planet, seeing that character streaming in real time.
Next Reality: I noticed that there was a bit of a lag during my demo.
Brown: That was basically the [event space's] Wi-Fi. All that motion data is going into our system and going into a piece of software called MotionBuilder. And then from Motion Builder it was streamed over Wi-Fi, competing with all the other Wi-Fi that's here, to then be rendered on the Magic Leap device in real time.
Next Reality: Can you have, let's say, two or three motion capture actors working together and viewable through the Magic Leap One using this set up?
Brown: Sure, this is just a very small [space] that we set up for L.E.A.P. con. Our [space] back in the UK is about 30 feet by about 25 feet, and we've done shoots where we've had up to a dozen people being captured at the same time. We work for film, TV, video games, all sorts of projects. We did all the cinematics for Battlefield 1, for example.
Next Reality: What about Magic Leap-specific experiments?
Brown: I think we're just at the start of how we use technology like Magic Leap. We've done tests with other motion capture data that we've already shot and have had up to nine people running in a scene [shown through the Magic Leap One].
We're keen to take the Grishneck experience and explore whether or not we can create a whole universe of characters. The idea with Grishneck is he's a piece of rejected concept art [based on Lord of the Rings' Orcs, but not actually from the film's production], and he didn't make it into a movie. Every artist has tons and tons of rejected pieces of concept art. We think there's a very interesting narrative idea to explore there with bringing those characters into your own world through Magic Leap.
Next Reality: Is your team helping other companies to work to develop this kind of Magic Leap content?
Brown: Not at the moment, but that's something I definitely want to do. As a studio, we do service work, and we do co-productions. We open our doors up, show them the studio, and show them what performance capture can do for their ideas.
Next Reality: Anything else in development for the Magic Leap platform?
Brown: We're working on another project with Magic Leap where we motion captured an entire musical. We're just at the beginning of taking that data and turning it into something that we can demonstrate. We have a few TV and film projects that we're working on as well.
Next Reality: Where do you see this kind of production, specifically for devices like the Magic Leap One, going in the next five to 10 years?
Brown: I think it will form another way that we experience the world. So [today] we have lean back activities like watching film and TV, and we have lean in things like games. I think Magic Leap and spatial computing is going to lead to a whole new vernacular for how you tell stories and how you present the world to people.
And one of the things that is really exciting about this for me is the potential of moving away from something like your smartphone, where you're looking down all the time, to doing a lot of those same things while you're looking up and you're still in the world.
Just updated your iPhone? You'll find new emoji, enhanced security, podcast transcripts, Apple Cash virtual numbers, and other useful features. There are even new additions hidden within Safari. Find out what's new and changed on your iPhone with the iOS 17.4 update.
Be the First to Comment
Share Your Thoughts