Eggscape is a mixed reality gaming prototype from the Argentinean 3Dar VR, film, & animation studio behind Gloomy Eyes and Paper Birds that leverages the black & white, mixed reality passthrough mode of the Meta Quest 2. It’s a has a really compelling 2.5 platformer game mechanic where you’re navigating a table-top scale egg character through a number of bridges and obstacles placed in a physical environment. It has a very similar vibe to Lucky’s Tale, but rather than just a 2.5 POV, you’re able to have the full 6DoF experience by moving your body through space in order to aid your 3D perspective judgments. It was a charming, innovative, & engaging enough experience to earn the 3rd place Special Jury Prize at Venice Immersive. I had a chance to sit down with co-founder German Heller to get the backstory of 3dar’s evolution from animation to film to VR, and a sneak peak at some of the new frontiers of co-located gameplay that’s enabled with these new mixed reality features on VR devices like the Meta Quest 2, Meta Quest Pro (coming soon), Lynx R1 (coming soon), and Pico 4 (also coming soon).
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to The Voices of VR Podcast. Well, it's been a busy week with a lot of VR news that I've been digging into, both the announcements from NVIDIA with their GTC keynote, where I had a lot of news from what they're doing with XR and the Metaverse. There's also a big announcement from Pico XR. They have the Pico 4, which is essentially the biggest Quest competitor that was just announced. It's a ByteDance which owns TikTok. There's a mixed reality mode for Pico 4, is worth mentioning in the context of today's episode, which is Eggscape. It's one of the winners from Venice Immersive, actually won the third place prize, which is the special jury prize. It's a mixed reality application, allows you to be immersed within Quest 2. They're really waiting for the MetaQuest Pro, which is going to be announced next month and released here soon, which is going to be this color mixed reality pass-through that I think is going to be the baseline for developing a lot of augmented reality applications. But this experience of Eggscape was really pushing the edge of what's possible with mixed reality. It's a little bit more of a game and prototype of a game that they're still developing. I think it's going to be a while before it actually has an actual release. My experience was that it was super compelling to actually be immersed within mixed reality and then have a sense of a spatial awareness of my environment around me. It gave me a little bit more freedom to kind of move around without worrying that I was going to smash up against a wall. As they're developing this as an application, they're starting to tap into some of the potentials for mixed reality. And I think that was probably a lot of what impressed the judges. Also, it's just a really fun game, compelling mechanics. I think when you start to think about blending interactions with storytelling, if you start with a really compelling game mechanic, then you can start to add different elements of the story on top of that. And I think that's the approach that 3DAR is going to be taking here. So Hermann Heller is one of the co-founders of 3DAR. They worked on Gloomy Eyes, which is a really amazing piece. I had an unpublished interview with them when they premiered it at Sundance and showed the final versions at Venice 2019. They also worked on Paper Birds, which is a really amazing project that they worked on and released throughout the course of the pandemic, and so that's available for release. So Gloomy Eyes, Paper Birds, definitely check out their prior work. But Eggscape is a really impressive experience that we start to impact both 3DAR's journey into VR, but also talk a bit about the process of designing for mixed reality. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Hermann happened on Friday, September 2nd, 2022. So with that, let's go ahead and dive right in.
[00:02:45.304] German Heller: My name is Germán Heller. I'm Argentinian. I work in a company called 3DAR that I co-founded like in 2004. And lately we've been working a lot with the first virtual reality, now augmented slash mixed reality. We also do tons of animation projects, like we write ideas and content in different forms using technology, like very generic, but that's what we do, a very broad kind of expression through art and technology.
[00:03:14.634] Kent Bye: Great. Maybe you could give a bit more context as to your background and your journey into working with the VR and AR mixed reality mediums.
[00:03:22.412] German Heller: So my background, like originally, like I was just like a crazy gamer when I was a kid and I was so obsessed with gaming in my childhood. And it was like that until I was like 20. Then I completely dropped and stopped playing for some reason. I became a software developer. I worked for like five years in that industry and then I switched to 3D and started this company. I was always really fascinated about technology and really what you can do with technology to bring the imagination to a more real and tangible space and create things. Like when I was a kid I was playing with mirrors and lights and kind of like seeing how the projection was creating magic. So I think technology is a way of magic, of creating like, I don't know, things that are not possible to do with the limits of the laws of physics and stuff. And okay, so virtual reality comes up, it's like basically the perfect tool. Not the perfect tool, but like it's almost the perfect tool. I think the perfect tool is mixed reality because virtual reality takes you out of this universe and I think that's actually something that most of the times you don't really desire. You want to add something but without removing everything else. So we were very excited about VR when it started. We even did a short film about VR before the Oculus headset even came out. And now that we have this new generation of headsets that are about to be launched with mixed reality, we're super excited to be playing and to be putting things in the real space and seeing the people next to you and playing together. And it's just like so many possibilities that it's really so interesting for us.
[00:05:11.079] Kent Bye: So you said you started 3DAR in 2004. What was happening from 2004 up to when you first started doing VR?
[00:05:19.575] German Heller: We started first, I was 23 at the time, and we were just trying to make a company work with whatever project we could get in the world of like 3D. So we started with architectural renderings and then we started playing with visual effects and we did some visual effects for commercials and animation, motion graphics and You know, like with a startup without any funding and getting like a team going, a team spirit. And it was kind of like a school for us. So if I have to put a label to that period, it was really like what I didn't do in college. I did with my own company and learning from the people that I originally hired and some even became partners afterwards. We were just growing with the technology because at that time V-Ray was just launching and photorealism was starting to be something that you could do with a computer at home. So in a way it was a really good environment for us to try things and learn and at the same time get paid with some projects and learn from working for clients and from the pressure and from the duties that you have to do and even learning English by working there. and in parallel like always keeping a little space for experimental projects trying things like going out and shooting a little short film or doing a 3d animation we did a short film about some graffitis destroying the city in 2007 because we were like in love with the graffitis and playing with that and then in 2011 we launched the first The first really recognized work that we ever done, which is the short film Shaved, about this little psycho monkey that shaves himself and goes to the city, becomes the president and does like a revolution only in 4 minutes and 14 seconds. And it's a very high pace, like stylized, with intense color palette animation that really put us out there and we got calls from even from Hollywood managers that represented us and they brought us in front of studios. So from 2004 to 2011 it was getting to that point. When we did this short film we realized that we wanted to do original content. But it was a long time before we actually started being able to do the shift because it was really difficult. Mainly because we were so ambitious and being in Argentina and you don't have access to funding or government help at all, like you're really on your own. So we did this short film about this psycho monkey and it costed us like almost $300,000 to do. And it was money that we get from doing commercials and without that money it would have been impossible to get to the quality that we wanted. So we almost go bankrupt to do this short film. Right after that we got like a bunch of work because of that. In 2015 happened the same but worse because we got to the point that we had to get a loan to survive and we paid it back afterwards when we did the short film about VR. And then in 2017 it started to stabilize and getting projects that are really high-end and rejecting other projects, doing our first VR series with Gloomy Eyes, then getting funding from Meta to do the second one with Paper Birds and other projects. We did the Earth project, that's this massive video with all these celebrities including Justin Bieber, Leo DiCaprio and Last year we did the video for Bad Bunny, this year actually, 2022. So we're in a point that we are fully determined to really put a lot of energy in creating content. We keep doing work for hire and high-end animation and working for big companies. And it's actually like a good thing that we don't want to stop doing. But in parallel, we're preparing products and ideas and brainstorming. So that's a long answer.
[00:09:24.105] Kent Bye: No, it's helpful because I remember seeing that short film, and I think I actually have an unpublished interview with one of the directors. Was that your brother?
[00:09:31.569] German Heller: Yeah, that's my brother and co-founder. Actually, I remember when we interviewed him, and we were just looking forward for this technology to come. And we thought about doing a dystopian version of when virtual reality really destroys humanity. But it was more like an expression of art, like a premonition, I think.
[00:09:55.343] Kent Bye: Yeah, yeah, I got it in my archive somewhere. I'll have to dig it up. And I've got some archival interviews about Gloomy Eyes as well. And I really enjoyed both Gloomy Eyes and Paper Birds. So you made the film, but you weren't into VR. What was the point when you made the pivot into VR? At what point did you come across and start to have some of the VR experiences and make this decision to start to make some immersive VR projects?
[00:10:18.952] German Heller: there was some like a huge hype like the VR revolution is happening and everything and but behind that was like the true desire to really do something that hasn't been done and that has like a soul of its own that is like our take on that technology a story that we wanted to tell something that is ours and and that is we wanted to get it done as beautiful as possible and And we did this meeting in our office. It was the core team of the company. It's like, OK, we have to do something with VR. Let's start brainstorming. And we started doing a pool of ideas. And there were a couple that were standing from the other ones. And at some point, the idea of this little zombie kid that was in a world that was no sun and was in love with a little girl. started taking momentum, and we started to prototype things. That was 2017, and Gloomy was released in Sundance in 19. So it took us some time to really play around with the technology. And at the same time, we were growing the IP, the story, the characters, the world. And then, yeah, it became real. And we released it. Then we got some funding to do episode two and three. and with Paperverse we already knew a little bit more like how the whole trail is so it was easier in that sense it was more difficult in the sense that there was a pandemic all the time and that was I mean I think both projects were equally difficult because the pandemic was kind of like a pain to deal with but in a way it affected the project in a good way, because Paperverse, if you've seen it, it's like it's a very very compatible mood with what we all experienced during the pandemic, this kind of like inner journey kind of thing. And now we're doing XK, which is the complete opposite. I mean, it's just really delirious and it's supposed to be just fun with a deep purpose, but it's not that... I mean, it doesn't take you to the sensitivity place or to this fragility or this inner journey. It's more like just laughing and enjoying and laughing at ourselves even as how we fail many times.
[00:12:43.382] Kent Bye: So you have the film that you had done about VR in 2015, you did The Gloomy Eyes, started in 2017 and premiered in 2019, but what point did you have your first VR experience and when was that?
[00:12:55.060] German Heller: I think we got the Oculus Quest in 2015 or something, the Quest, the Rift, the DK2 or something. And we tried many things, but there was something that we really liked. I don't remember the name of it, but it was like a car racing game that you would see it like on a tabletop kind of thing, and you would see the cars here going, and it was so well done. I wish I remember the name of that but it was like this is really something like to be able to see toys like this and moving and really with the 3D feeling that feels real and and doing good use of space it was like an off-road kind of like game and that was something that I was like oh we have to play around this like we can tell stories with this language And that is inspired a little bit like the scale of Gloomy Eyes. And even Paperverse and Xscape are like kind of like the same scale, like the toy scale, like the little things that you could move with your hands if you were playing and they were real objects.
[00:13:59.588] Kent Bye: So maybe you could tell me a bit about the origin point of Xscape, which is a mixed reality game that you're showing here for the first time in public at Venice Immersive 2022.
[00:14:10.294] German Heller: Sure, so Xscape. We were brainstorming with my brother Federico because we were always laughing at how it's funny that we're always trying to do things and we fail a hundred times before we do it right. and it was kind of like a concept that it's often fun to see failure because it's like you clearly see the disappointment but it's like if you take it heavily it could be really bad but if you actually look at like a comedy it's like It's really hilarious, like what happened to us when we tried to go in one direction and things don't work out. And we wanted to play with that feeling of being in a state of survival all the time, that the world is full of dangers and you don't know what's going to happen to you, you don't know when you're going to die or what's the next problem you're going to have. And at some point it was like this state of like, oh my God, I know this is not going to work. Your odds are really against you. So that's kind of like the tone, doesn't have to do anything with mixed reality, and it didn't start from the mixed reality at all. It started from that thing of these characters that were really fragile and the world full of hostility. And then we started playing, at first it was more like a puzzle, like strategy, kind of like this game for iPad, what's... Monument Valley. Yes, exactly, Monument Valley. This puzzle, I really like the aesthetic and the resources in there and we wanted to do something like that in VR, but then it felt like, I don't know, it felt like it wasn't fun enough. I mean, the puzzles is more difficult because if you're in danger but you have time to think, that kind of danger is not the one that you're running and there are missiles behind you and you have to save yourself somehow, you don't know how. So we did this piece of concept art that was this egg running from so many dangers. And it's like, we saw in that concept, like this is the mood that we have to achieve. Like this feeling of like, there is no way we survive, but we're going to try it anyways. And you don't have time to think, you know? So we played with three prototypes until we got to the mixed reality one. The first one was the puzzle that was in the middle age. Then we did another one in the future. that was like a platformer, more like a VR platformer, still not mixed reality. And this year we heard about mixed reality and that it was finally something that was going to happen. And the moment we heard that it was like, this is it. We're going to use this because the last platformer scene, it was fun. But it was like a Super Mario with some other, like an immersive Super Mario with death and some other little things, but it didn't feel enough. And we were like, okay, so what do we add? Gestures, what is it that we can do that brings something new to the table? Mixed reality was perfect for that because it also enables you to play next to somebody else and see the other person. And then we just reformulated the whole thing to be multiplayer mixed reality. And then we developed the World Builder 2 that you could build your own levels. And now we feel like it finally has taken shape because it has really a lot of elements that I haven't seen before. Maybe there is something out there, but I haven't run into anything that is neither mixed reality nor multiplayer in that sense. And it still has this soul, this essence of this X that are like freaking out all the time.
[00:17:43.888] Kent Bye: Yeah, there's something about being in mixed reality that if you're very familiar with the space that you're in, you know, it's a black and white camera, at least the version that you're showing here on the MetaQuest 2. So you have the black and white video, but then you have the color with all the different entities that these eggs are jumping around on and I found that when I first tried it in Tribeca, I was able to move around a space and feel like I had a little bit of freedom of knowing that I was able to move much more vast distances. But then also the gameplay mechanic reminded me a lot of like a very early platformer that released on Oculus Rift as a launch tile with Lucky's Tale, which it was basically you're moving around this fox and as you're jumping around, you have to, you almost like as you move your body around, you get more of a perspective for how things are actually going to land. And I found that one of the fun gameplay mechanics of this is that as you're jumping that it's kind of difficult sometimes to know if it's going to land without like moving your body to have a little bit more of the depth and the parallax and so it actually encourages you to move your body in order to understand the spatial dimensions as this egg is moving through space. And so it ends up being a real engaging embodied gameplay, but also the abstractions of using the controllers to be able to control it like a platformer, but you're using your body to have an additional depth and perspective as you're playing it.
[00:19:00.200] German Heller: Yeah, yeah. It's something that even not so long ago, we were trying to design what's the real challenge of the game. Because before, you would have things that would attack you, but you would just run. So it's like, OK, so you're going to die because somebody is going to hit you or something, and you have to run and escape. And then we found that the reason for which you would constantly die is because you fall down cliffs. You have platforms, and you miss a platform, and then you fall. And that was somehow frustrating, like falling constantly and dying in that sense wasn't rewarding. You get like a sense of like, oh damn, I missed it again. And we did like a whole month almost of brainstorming and playing around ideas. And then it came the idea that the ex can learn kung fu. and then they're going to fight enemies and they're going to do tricks so we took a little bit of weight off like falling in the right place we put like even like a hanging thing that if you miss it a little bit the egg hangs so it's not so difficult you don't have to be just like trying to do it perfectly all the time, but it's much more fun because you can do like Kung Fu like kicks and fight in funny ways and even punch your friends too. And there's still a lot of development to do that because we want to do combos, we want to do like different ways of like grabbing your enemy and like not even hurting them sometimes but just like mocking at them or like many many animation clips and we have two animators in the team that are really brilliant with coming up with really random style of animation and random scenes and And we plan to really develop that, to make the eggs feel like they're alive and they can surprise you and act really weird all of a sudden. So that's a big part of the game. And it's not only about going from a platform to the other, which is a little tiring after a while.
[00:20:59.655] Kent Bye: Yeah, when we did the demo here, you did a little bit of a multiplayer experience where they have a sandbox version where you can build your own levels, but then as you're building it and you're running through, you can have someone else who's in the mixed reality headset who's trying to introduce obstacles or make it a little bit more challenging. And so you have a little bit of that sandbox feel, but adding the gameplay element onto the sandbox creation aspect by having somebody else who's trying to obstruct you in some ways. So I thought that was potentially an interesting dynamic. I mean, it kind of requires having two people with the Quest headsets. I don't know if they need to be co-located or if you could do remote versions of that, but at least it's an interesting dynamic to be able to have two people at the same time and you have less latency issues and so you have more real-time interactions to try to have it a little bit more of a competitive aspect for someone trying to reach a goal and the other person trying to stop the other person from reaching the goal.
[00:21:50.050] German Heller: Yeah, totally. I was also playing the multiplayer here and discovering new ways of gameplay. For instance, I found that it's very funny when an egg runs away from another one. Just visually, one runs and jumps and the other one is chasing them. And you could grab objects, so it's this egg grabbing a barrel and chasing the other egg. It's such a hilarious scene. And that could be even a way of gameplay. We are playing and we're going to keep developing different modes and different missions. And there is a big inspiration for us to do that. It's a game that is not that old. It's called What the Golf for Apple Arcade, which I believe is brilliant in the sense that the fun of it is that it's always breaking the rules of his own game. is another game and then you do something else and it has some common elements between these ways of gameplay but it constantly breaks the rules of itself and we don't want to go as far as that but we really want to keep adding elements and keep adding ways of cooperative play or competitive play and yeah and the way I see it for the future is like when you have like a basketball game and it's like three players in one team and three players in another team and you have technology that you're using is just a ball but at some point everybody is gonna have their own like mixed reality glasses and people are going to get together to play sports or to play games and and you would play a full multiplayer it's like come by let's play Xscape together and it's gonna be four friends So I'm going to be the world master, so you create an adventure for all of them, and everybody has to run, and the one who gets there first is the one who wins, and there is somebody which is the god and is creating the level, you know? So, I mean, I'm really excited to keep working on improving the game and adding more things for years to come, you know? Like, the next generation can be like a completely different thing, and I'm so fascinated by the core, soul of the project of being like so fragile and with this feeling of like oh my god this is gonna go wrong that I can keep working on this for for five years for sure.
[00:24:10.002] Kent Bye: Well, here at Venice Immersive in 2022, you have your installation where you have things that are actually built out on the side of the wall that you talked about the tabletop nature of these games where they feel like little toys, but you have actually built out a physical representation of the scenes. Then when you go into the VR experience, then it's completely occluded and overlaid on top of that. So then you have the egg that's going through these different levels. But you have a little bit more level design that you have from what I saw from the first prototype at Tribeca, you have different level design and game design and mechanics. And so maybe you could talk about, as you're preparing for showing it here at Venice Immersive, translating the open sandbox game into more of a level design and more of a narrative experience that's unfolding through a spatial context that's unique to here at Venice.
[00:24:58.305] German Heller: Yeah yeah so what we did after the first real prototype we had that was showing like unofficially in Tribeca to some people was where would it be fun to see these eggs being harassed by enemies you know it's like so it's like a dangerous alley you know and then these bullies come and they're like hitting a little blue sheep and then they come after you like And then the basketball court that is very intimidating with really tall players playing and then you go over there and they go after you. Then we wanted to do the possibility to go inside an office and then you can break everything and enter in an office that looks a little bit like an office in the 80s. Had like a feeling that we liked. Then to cross the street, it's kind of like a little tribute to the Crossy Roads game that we really like. And then we wanted to do other scenarios too, like we wanted to do like a casino, and we want to take the eggs to a stadium, to Vegas, and really random situations. Because we're actually playing with this idea that we are all fragile eggs in the end, but like we have all this circus that we create as our personalities, our looks, and the crazy cities we built. But we always stay in this feeling like, oh my god, no.
[00:26:21.414] Kent Bye: So yeah. One of the challenges of designing an experience for mixed reality is that you have to work with whatever spatial context you're given. And so if people have a variety of different types of spaces, then you have to try to figure out how to create a modular system that could potentially adapt and adjust the game so that it fits into the spatial context that you're playing it in, which then when you do all that abstraction and generalization, then you have to make all these various trade-offs that may make different quality of the gameplay. Whereas in VR, you could just make it and you have complete control. You don't have to worry about that spatial context because you're not constrained by any of the physical places that people are playing in. So I'm curious to hear about how do you solve that dilemma of if you are going to make it truly mixed reality, then if you want to create those curated scenes, then is it only really focused? I mean, you're still in development, so this is still up in the air, but I imagine that part of the allure of the mixed reality is that you have the open sandbox dimensions, but to be able to design some more of those curated levels that you have like here, then either it's going to be pretty static and you could pretty much have the same experience in VR. Then the question is like, what is the mixed reality really giving to that as a experience? So I'd love to hear a bit about like, as you move forward, how do you address some of those issues?
[00:27:36.960] German Heller: No, absolutely. First, I think that you have much more freedom in mixed reality than in virtual reality. In virtual reality, you're constrained by the fact that you don't see anything. You're blind, and you don't see if you have a wall next to you. And I've seen people bashing their heads to walls because they don't see anything. And it's like you're exposed. And not even from a space perspective, you are limited. with the sense of relaxation that you lose by being blind. So it's like you are a little bit constrained because of that in virtual reality. And you have the Guardian, and you have a limited space, and you bump into objects. So when you're actually seeing the world, it really expands the possibilities that you can do around it. You only need to understand the topology and the shapes of things. And then you lay things on top of them. what we're planning to do is like you just draw a line and then recognizes the surfaces and then you're gonna have different platforms and connections and bridges and bouncing platforms that you're gonna be going from point A to point B. And it's like unlimited really because you can detect the shapes. Like there's this thing that I know that Meta SDK has a scene understanding and spatial anchors. It's all prepared from a software and hardware perspective to interact with the space and not to deny it like VR does.
[00:29:11.187] Kent Bye: Yeah, I know that Mark Zuckerberg was recently on the Joe Rogan podcast where he had shown Joe Rogan, presumably, a production build of what is probably going to eventually be called the MediQuest Pro, but has been colloquially referred to as the Project Camera, which is a full-color mixed reality headset with eye tracking, facial detection, but the pass-through camera with color is going to, I think, open up a lot of new opportunities for the future of mixed reality. I'm not sure if you are at freedom to say whether or not you've been able to have access or to see some of those, but I'm just curious to hear some of your thoughts on that as a headset and where that might go in terms of the project that you're working on and how that might fit into that, or if you feel like you're going to be targeting more of the consumer quest, MetaQuest 2, with the existing black and white pass-through.
[00:29:59.439] German Heller: Yeah, no, we're targeting the next headset. We're using the Quest 2 here because we don't have the next headset. I haven't even seen one yet and I did ask for it but I'm still waiting and I can't wait. I know many major hardware producers are going in the mixed reality route. I mean, it's nothing that I can confirm but I'm expecting Apple to do the same. I think Pico is doing the same. Meta is really focused on mixed reality too. So 2023, everybody is going to be in that space. And we're targeting to have this new project finished by the launch of the next Quest, the Quest 3, that we expect, we're projecting that it's going to be next year, end of next year.
[00:30:44.686] Kent Bye: OK, yeah, because I was surprised to hear Zuckerberg say on the Joe Rogan podcast that the MediQuest Pro or the Project Cambria could be launching as early as October. So it sounds like people that are getting the higher version may have access to some of that. But in terms of the consumer scale, you're going for the consumer launch rather than the more prosumer or the more expensive version, it sounds like.
[00:31:05.623] German Heller: Yeah, I think there's going to be like an adoption curve and there has to be also a process in which we have enough time to play with the Cambria or with whatever headset we get to our hands that is actually designed to be mixed reality because I haven't ever seen a full color pass-through and I don't know how that's going to play with animation because we chose this unlit style for Xscape that is all like painted textures that I love and look beautiful but what if we have a color pass through and they look like they don't integrate in the lighting maybe we can have dynamic lighting or not we're not using dynamic lighting almost We're just using a single light for XK because the pass-through also takes a lot of processing, like the pass-through the way it is now. So we are very handicapped with this headset that is not designed for mixed reality. But yet we are here presenting a mixed reality project and I believe we have something that shows the potential. I can't wait to have a Cambria in my hands or any headset that can give me a full mixed reality adventure.
[00:32:19.347] Kent Bye: Yeah, it'll be interesting to see if, uh, what the contrast between the existing style that you have with the black and white, because I actually think it works really well to have the black and white in the background with the color. It stands out. And so I, what I would wonder is that if you start to have the color and the AR, well now you would have less possibility because it doesn't look quite as real as when it has a black and white, you're not expecting it to be as certain fidelity and that once you go to the color, then it may create a less sense of embodied presence within that scene. So I feel like there might be some trade-offs. And I wonder if once they release the Cambria, if they provide the option to turn it to black and white, if you want to create more of a surrealistic scene that allows you to be aware of your surroundings without creating such a high level of photorealistic fidelity that you don't believe as much of the plausibility of it because it's not quite as photoreal as you would want it to be.
[00:33:10.548] German Heller: yeah i honestly many ideas we have we do them we try them and then we trash them because only by trying them we can realize whether they were right or wrong and i think with this color passthrough i just feel like i want to try it and see how it feels but i My intuition tells me that if you're going to do something that has to match a full color environment, it will need to match the lighting too. And so you will have to be more careful in the aesthetic that you choose and you're probably going to be limited by the amount of real-time rendering you can do. And right now even we are using only baked textures because of not having power to render like real-time shadows. So I don't know how Cambria is going to come and what ARM is going to give us from that performance standpoint but we have to get it and I mean I'm really waiting and sending emails a lot like come on guys it's one month before the launch just send us one we just want to do content for you guys.
[00:34:23.694] Kent Bye: Awesome. Well, just to give a brief mention to Gloomy Eyes and Paperboards, I think both of them are really innovative in their own rights in terms of the tabletop scale and use of space. And yeah, that's for people who haven't seen those, I definitely recommend them to check it out because I feel like there's a level of seeing what's possible with spatial storytelling. And so I'm also curious to see that now that you're doing this pivot into eggscape, how you're going to be potentially drawing in some of those different immersive story elements. Because that seems to be the big question here at Venice Immersive 2020 is how do you start to really integrate the narrative in the games?
[00:35:01.056] German Heller: Yeah, it's a great point. So it's like basically Gameplay comes first. I hate it when you have this. I mean when I was a kid like 95% of the games that I had cinematic I was like Clicking clicking clicking trying to spacebar spacebar trying to just like pass them and get to the game because either you're anxious to chew the adrenaline of the game And I would really enjoy cartoons and stories and everything, but it's just such a different mood. But it was 95%. There was a 5% of the games that I was playing that were more like an arcade, kind of what we're doing here, that had this wonderful cinematics that added drama and story to the characters that were fantastic. So once we have something that is like completely fully fleshed out in terms of gaming we're going to add more and more bites of stories or little surprises but it's always going to be in short like little situations or things that are going to surprise you so you don't lose the pace of the game. It's not like you're going to watch a cinematic of like 60 seconds and you go back to the game. That's not what we're going to do. But when there is this villain that comes out and it's a huge villain that's like this and punches and there is a lot of storytelling there. And the way you present it, it's not that you have to wait. It's like it's part of the game. So that's our approach there, but it's priority too because first we need a really fun gameplay and we always have to keep it simple. So yeah, that's how we're doing it.
[00:36:45.160] Kent Bye: It's very interesting to see Eggscape in the context of Venice Immersive because there's the Peaky Blinders, which is a narrative VR game, but again, it feels like what I've seen before in terms of adding context switches between the interaction and the puzzle adventure versus the little cutscenes and cinematics. With here, with Eggscape, it seems in some ways to be more of a pure game and a little out of place for what Venice usually focuses on, which is very focused on the stories. And so I think with your background coming from Paperbirds and Gloomy Eyes that there's sort of a trajectory of where that could go. But yeah, I'm excited to see where that continues to go. And also just to show the other immersive storytellers here that are at Venice to be able to see the potential of some of those different interactive and game mechanics. I totally agree that you have to have the core game mechanic really nailed down first before you start to build out the story. And so it seems like that's what you're doing here. And so I'm excited to see where it goes. Have you had a chance to see any of the other pieces here or any thoughts on... Maybe Monday.
[00:37:45.767] German Heller: Thank you for what you're saying is motivating too. Yeah I always like to see things after I'm kind of like settled and what I'm bringing to the festival really works and I get some feedback from the people it's like maybe a little bit selfish in a way because I'm not really open to see other works just now because I'm really focused on my own project but I'm sure I'm gonna find the time like closer to the ending days like the three four ending days because i'm interested to see what's up and it's also good to actually get the best recommendations of people you really trust it's like what is it that i can't miss and then i go and see all of that that's my secret
[00:38:27.050] Kent Bye: Yeah, it's a good chance for you to do a lot of user testing, but it seems like you also have an iPad to be able to get a window. But mixed reality as a medium feels like you're able to really be, in some ways, immersed in the same spatial context as you watch the users play, which I think will also be kind of interesting for you to, as you continue to do user testing, there seems to be some new opportunities to use the medium itself to be able to be co-present with people, to be able to watch and observe what they're doing in the imaginal space that's overlaid on top of the physical reality.
[00:38:52.162] German Heller: yeah yeah yeah and i'm getting really good ideas from here but i got really really really great ideas from playing it with my nephews like it was my brother that was playing with them but he was just telling them like what they were doing and how they were, like, mocking each other. Like, the older one was just, like, hiding treasures in the house for the little one to find, and then, like, hiding traps. And at the time they had only one headset, so they were looking with the iPad, with the casting, to see how he was doing. And it was, like, such an interesting way of gameplay, you know, with a single headset. And a phone, since it's a passthrough, it's a mixed reality, so you already have a multiplayer with a single headset. So it's just with the casting. So I think the PG-13 aspect of the rating that the Quest 2 has is wrong. This needs to be tested, of course, but, like, it's for younger kids. It's for, like, because my nephews are, like, nine and seven, and they were just, like, inventing ways of gameplay and having so much fun and playing for hours laughing. And it's such a great toy, and it's not that they're isolating. They're playing with each other, and then they were fighting, like they always do, but in very creative ways and also very inspiring to develop gameplays around it and give them more tools to create the gameplay they want.
[00:40:20.213] Kent Bye: Yeah, I think probably comes more down to COPA compliance, which means that for people who are less than 13, you can't collect any data on them. And so there's a lot of issues around the younger kids who are less than that, that there are youth accounts on, say, like Rec Room, but that's at 12. So there's other medical reasons as well in terms of the eyesight and everything. So there's some unsettled science on some of that. So I think they're probably leaning towards being more safe than causing a lot of permanent eye damage of a whole generation.
[00:40:49.930] German Heller: I don't want my nephews to go blind by playing Escape at all, but it's just like more like The way this is going is bringing a lot of possibilities like toys coming to life and when you buy Legos you just pay like $40 and then you get like a handful of pieces. Here you have unlimited possibilities and the pieces can move and there is no law of gravity and the characters can run and shoot to each other. It's like without isolating you, without getting you into a computer, without getting you to be seated and still like there is a lot of elements of like the way you would play in the world because you're not looking at a screen. It's just you have your hands free and you're like playing in space. So it's actually much healthier in many ways that like stationary gaming and people just like sitting and watching a TV. This is a very active way. And they were cracking laughing and they were just like playing the same way they would play like a basic frisbee play.
[00:41:54.439] Kent Bye: Awesome. And finally, what do you think is the ultimate potential of mixed reality and what it might be able to enable?
[00:42:02.605] German Heller: Well, I'm not like a mixed reality evangelist or like some fanatic that I think is going to change and save humanity. We're going to be the same. No technology is going to change that. But I think there was the cell phone, then the smartphone, and then mixed reality at that level. I think that's where it's going. It's like right now I have this rectangle in my pocket with a screen that I have to touch and everything and it's going to be like glasses and it's going to have like an interface that's going to be much easier and fast. than the smartphone and the screen you're going to have there. It's going to be integrated to the space. It's going to be the real place to be in the metaverse, like not a VR thing, but like a mixed reality thing. And I think the metaverse is going to work more like windows to the metaverse than just like going away from here and going somewhere else. And I think mixed reality is going to be the platform for all of that.
[00:43:04.083] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:43:08.815] German Heller: I don't know, there's like a little bit that's coming to mind now that it's like with these new games that have like some crypto aspect to it and the play to earn thing and the speculation and everything that happens when money is involved around it and everything. I was wondering, like, what is it that we have to learn from that to really integrate the money aspect into it without selling the soul for it, you know? So it's maybe like a loose thought in a sense, but it's like Uncanny Valley in itself was a short film about how people were making a living by playing games in virtual reality and how that destroys the universe, the world. And it's happening a little bit, not like in just a little hint of this, that people going to play games and it's like, I don't know, I think there is a fine line there when you're not playing to have fun and to share a good time but you're doing it to speculate and I don't know, there is a danger of this becoming something that people like with the Pokemon Go like and working on that to get money, to get awards and the competition I mean, it would be great to keep it in a sense that we can play and have fun, and it doesn't become stained with the speculation. That's just a loose thought.
[00:44:31.267] Kent Bye: Awesome. Well, I'm excited to hear more news about the Project Camera, probably the MetaQuest Pro that comes out within the next month or so, and excited to get a bit of a sneak peek of some of the different interactions and gameplay. And I can say that there is something distinctly different about the interactions that I have in Eggscape. I've saw the Lynx demo at the AWE which had full pass-through and it had like really close to the eyes and so just able to have the full color pass-through and the blending and blurring of the realities that the Lynx R1 had. And yeah, the Vario XR3 that they have as well had demos at AWE. So yeah, just to see the next phase that feels like this. mixed reality and then perhaps eventually just a pure augmented reality or maybe mixed reality pass-through with VR. It feels like it's the next chapter that's about to begin. So glad to get a little bit of a sneak peek and some of your thoughts today on the podcast. So thanks for joining me.
[00:45:23.206] German Heller: Thanks. Thanks. It was my pleasure, Kent, and always a pleasure to be talking about this with you.
[00:45:28.975] Kent Bye: So that was Hermann Heller, one of the co-founders of 3DAR, and Eggscape won the 3rd place prize of the Special Jury Prize at Venice Immersive 2022. So, I have a number of different takeaways about this interview, is that first of all, well, I had a chance to first see this back at Tribeca, and so being able to see what they were able to on since then here at Venice Immersive 2022. Like I said, it's a really compelling game. It reminds me a lot of Lucky's Tale, that kind of platformer where you're looking down at this 2.5D perspective. In this case, with mixed reality, you're able to actually move around into a 3D space. It's a little bit more than that 2.5 fixed perspectives that you get in something like Lucky's Tale, but you're able to actually move your body around the space rather than have like a floating camera. And yeah, I just found that when you have a mixed reality experience, you have a little bit of better spatial awareness of you moving around. In this instance, it's using the black and white cameras of the Quest 2. And so everything's black and white in the world around you. And then all the stuff that they have featured is going to be in color. Hermann talked a little bit about the specific shading, the lighting that they have in that consideration of being black and white. And so once you start to move into Full color pass-through then I start to wonder whether or not it's still gonna have that realistic enough look and feel because you're basically Comparing it to your understanding of the reality which has a pretty high bar from a plausibility perspective It has to be really really quite good So I wonder if as the MetaQuest Pro comes out and more of these mixed reality experiences come out, what kind of other things that people will need to do to balance that mental presence, that plausibility, and if there's anything that is not quite up to the same level of your experience of reality, then it's going to take you out of that experience. Yeah, as we get more information about the MetaQuest Pro and these mixed reality experiences, these are the types of things that people will be able to actually test and work out as they move towards what I see as kind of a prototype demo of proof of concept into a fully fledged game. And it was very interesting to hear from her mom that working with his nephews, that they're able to prototype different ideas of how this could be used into a asymmetrical collaborative game where maybe somebody goes into VR experience builds out a level and then comes out and is able to use the iPad to get a window into that world. And so being able to use existing devices that people already have and to be able to tap into this world that's been created, that's been broadcast from the Oculus Quest headset as a remote virtual camera to be able to have this collaborative asymmetrical type of play. If people start to have multiplayer games where you start to have real time interactions where the latency isn't so much of an issue, maybe you could start to have new types of gameplay mechanics of co-located mixed reality gameplay and a piece like eggscape. Really amazing character design. Very cute, very adorable. And actually, there's a fighting mechanic that was new that wasn't in the Tribeca version that I saw that was also pretty fun and reminded me of different aspects of Lucky Stelm. Yeah, if you enjoy that kind of platformer format, then thinking about how to abstract that out into more generalized spaces, to draw a line on a couch or whatnot, and to be able to have a variable amounts of difficulty in that open world sandbox that they had was a lot of fun to be able to play stuff out. Although I think it is limited, I think having a set of really solid predetermined levels that can maybe adapt somewhat to your world around you, I think is going to make the difference between something that's a mixed reality experience in a virtual reality experience. I mean, within VR, you can really control the level design, but in mixed reality, augmented reality, I think to really work with that, it needs to be somewhat adaptable. So having those generalized abilities to make it dynamic and to really fit with the world around you, I think is going to make it so much more compelling. So how to do the balancing of level difficulty and the game progression curve and all the different interplay of narrative elements that they want to start to throw in there and maybe have collaborative types of games as well. All that, I think, is to be determined. It's still very early days in their development cycle. But with their award, I hope to see that that's a seal of approval of the mixed reality as a medium and to see more and more experimentations with that. So, that's all I have for today, and I'd like to just thank you for listening to the Voices of VR podcast, and if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listeners-supported podcast, and I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
