Pilgrim is an AR audio documentary that I experienced at the IDFA DocLab premiere as I was walking through the streets of Amsterdam. Co-director Lauren Hutchinson captured audio stories of people as they were walking the Camino de Santiago trail in the north of Spain, and you have a chance to listen to their transformational stories as you’re walking through an urban environment. If you stop, then that pilgrim continues on and you wait for the next pilgram to come along with a new story. They used prototype AR glasses with spatialized audio that were connected to a phone, which used ARKit through camera to track your movement and rotations as your turning around.
I had a chance to talk with co-director Hutchinson, who collaborated with Saschka Unseld and the Tomorrow Never Knows team in order to create this location-based, interactive AR story set to a specific loop in downtown Amsterdam. Aside from the phone providing directions, the overall experience was screenless. It focused on augmenting your experience with interactive audio in order to recreate the serendipitous interactions that you might have if you were taking pilgrimage on the Camino de Santiago trail.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
Most people on the trail have some sort of reason for being there, and they captured a number of people who were in transition or had just gone through some sort of transformational experience. There is a very specific depth of sharing that happens in the context of refuge which was recontextualized, remixed, and overlayed on top of the context of a bustling urban environment. The creators wanted to highlight the vast differences of these different contexts, and so they embraced this chaotic balance between fate and free will where sometimes there are circumstances beyond your control that might trigger a change in the stories you end up receiving. It’s a form of branching narrative where you may not even be consciously thinking about the stream of consciousness segments you’ve receiving, which led to some profound serendipitious moments in my experience of it.
As the tracking technology and spatialized audio solutions on mobile AR glasses continues to improve, then we’re going to start seeing a lot more of the immersive theater type of stories where actors are physically moving their bodies through space and you have the ability to follow them. But the first iteration of location-based AR stories will likely be more like a guided tour or a more linear branching story with explicit opportunities to make choices through the experience.
But what’s made clear to me with the Pilgrim experience is that there will be all sorts of different location-based, AR audio stories that are overlaid on top of specific locations. And as spatialized audio solutions start to come out on AR glasse, then that’s going to open up more screenless options for creatives who won’t have to use visual components in order to tell a compelling story.
This is a listener-supported podcast through the Voices of VR Patreon.
Music: Fatality
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. Hello, my name is Kent Bye, and welcome to the Voices of VR Podcast. So in my last interview with Kasper Sonnen, I had a chance to give a brief overview of some of the projects I saw at the Idea Phase Doc Lab. And one of the projects I was really excited to see was Pilgrim. This was actually like a audio documentary that has been recontextualized to be this interactive narrative within this augmented reality experience that you experience by actually walking around an urban city and you get a taste of these different conversations. So I had a chance to experience this experience at the IDFA doc lab in my brief 23 hour layover in Amsterdam, where I went to the doc lab and did a bunch of experiences and talked to a number of the creators there. And I unpack it with the creator, Lauren Hutchinson, and then At the end, I'll be talking about what this means and where I see the future of this type of interactive augmented reality storytelling is going to go. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Lauren happened on Monday, November 19th, 2018 at the IDFA's DocLab in Amsterdam, Netherlands. So with that, let's go ahead and dive right in.
[00:01:21.108] Lauren Hutchinson: So my name's Lauren Hutchinson and I work in radio but for this project I've decided to work with augmented reality technology to work out how we can tell stories in a new way using sound and augmented reality.
[00:01:36.493] Kent Bye: Great, so we're here at the International Documentary Film in Amsterdam and there's a doc lab and so maybe you could tell me a bit about this experience that I just experienced called Pilgrim.
[00:01:47.768] Lauren Hutchinson: Yeah, so Pilgrim is inspired by the Camino de Santiago, which is a pilgrimage in the north of Spain. And it's over 800 kilometers long. And people normally walk it for about a month. And they walk it for lots of different reasons that they're coming across in their lives. And so Pilgrim is a app on your phone, which is using pass-through headphones. And it's a walk through Amsterdam that takes you on this augmented journey of the Camino.
[00:02:17.887] Kent Bye: Great. So how'd this project come about? I mean, you were talking a bit earlier that you had walked the Camino, and so maybe you could tell the first iteration of the story and then the evolution of it from there.
[00:02:26.891] Lauren Hutchinson: Yeah, sure. So in 2015, I went to start making a radio documentary about the Camino de Santiago. And I was interested in the meditative act of walking and letting things go. And so I spent a number of weeks walking along in Spain, interviewing people about their motivations for doing the walk. And it was while I was doing that walk that I realised that it didn't really work as a linear documentary because I felt like people should have to be walking themselves in order to hear the stories and they shouldn't be able to just sit back in their armchairs and hear these amazing feats that these people are doing and they should be also making themselves vulnerable in a way and walking themselves to hear the stories. And also on the Camino you meet people and then you let them go and you don't know if you're ever going to meet them again. So I wanted to think about how I could recreate that feeling of like losing someone and then a few days later you see them again but you say goodbye and you don't know what that means. And so after that, I started to research all the different ways that you could do that. And I started to realize that augmented reality was a way that you could potentially do that. And I was interested in how you could use augmented reality to augment your own journey. So it's not like a physical place, because a lot of people are using sound in places, and that's triggered by the place with augmented reality. Or they're using visual triggers, and the screens are kind of in your face. So I was really interested in how you can still be in the world and then it's your movement that becomes augmented.
[00:03:59.626] Kent Bye: Great, so you're using a phone and I guess I learned the hard way that you're using ARKit and a camera on the phone because it's in Amsterdam in the fall, winter time and I put the phone in my pocket and then it got stuck. So you have to have the phone out and so what is the phone doing? Is it just detecting if you're moving or like how are you using like the augmented reality and the ARKit because there's also spatial sound where you're able to kind of like I noticed that if I pinned the phone to my body and turned around, I had a little better effect that it was like, oh, if this is using the phone, like if I'm moving my head around, it's not necessarily detecting that as much. But if you're tying this rotation movement to the phone, then you get this extra specialized sound. So maybe you could talk a bit about the process of figuring all that out.
[00:04:44.098] Lauren Hutchinson: Yeah, so exactly, you've tapped into all the different important things there. And this is really the first iteration of the project. So it's the premiere here in Amsterdam. And so at the moment, the way we're doing it is we're using ARKit, which detects motion and your movement, and it does that through the camera. And that means it needs light. So yeah, by putting it in your pocket, you lose the light and it loses the tracking. So we're learning a lot about how this is used in practice. And so there's that, but then at the moment, we're also using GPS. as other information that the phone is using. And so what it's doing is it's tying those different bits of information together of both your real location and also your movement. And what it does as well is it tracks which is more accurate at a point in time, so which it can get most data from. And then it uses that to then drive the story forward and determine basically whether you stay latched to a pilgrim or whether you unlatch from a pilgrim.
[00:05:41.908] Kent Bye: Yeah, that was interesting because as you're walking if you suddenly get bored and you don't want to listen anymore you could just stop and then they walk away and then you find the next pilgrim that walks around. I could imagine in a real situation you might not so abruptly stop, or maybe you do. Maybe you find your ways to make an exit and maybe pause and then catch the next person that's walking along. but just that mechanic of walking and then stopping and then having the next one come along. And I noticed that there was the footsteps I think were a helpful cue because actually the first time I did it there was so many crowds on the street that I couldn't walk as fast as I wanted to and so I found myself actually wanting to hear the story but then there's too much traffic and there was like this frustration running into the blocks of the real world And I found myself a couple of instances like that where I was like, oh man, I have to find myself, like, keep moving, otherwise I'm going to lose this story, but I don't want to, like, run into people.
[00:06:38.058] Lauren Hutchinson: That was kind of partly an intentional thing we wanted, because we wanted to bring it to Amsterdam, but at the same time you're not on the trail, you know, you're not on the Camino de Santiago, and we're not pretending that you are. so we wanted you and I'm pleased you say you felt that and I know it might be frustrating but we kind of wanted that feeling of like I'm not actually on the trail and so I don't get to just walk with people as long as I want because I'm in a busy city and so that yeah sometimes you're stopped by other people in front of you sometimes you're stopped by a traffic light there's all these things that then you might lose this person you want to hear the story from but that's part of the experience that we wanted to create.
[00:07:13.472] Kent Bye: Yeah, maybe you could elaborate on the other ambient sound design that you have to give these various different clues, because you're trying to give information to the person who's experiencing this, which is like, are we tracking that you're walking? Because if you're stopping walking, then it switches. And so you kind of have these footsteps and other things. So what were the other aspects that you're trying to create? I mean, you have a screen that's giving some directions, but other than that, it's just an audio experience. And so you have to rely upon these sound designs and these audio cues in order to communicate with the audience. And so maybe you could talk a bit about that process and different things that you were trying to do there.
[00:07:47.073] Lauren Hutchinson: Yeah, definitely. So we work with the cue department on the sound as well because, as you say, it's a screenless, we want it to be as a screenless experience as possible but there is a lot of information that the listener needs to know. So we worked with where the pilgrim is coming up behind you, you don't hear them as clearly and then we adjusted the kind of volume when they latch to you so there's a kind of feedback of, okay I'm latched to this person and then when you let them go the volume goes down and we worked with specialized headphones to work with you being kind of orientated with where the pilgrim was. It has been a learning experience and as I said this is the first iteration and we have found that the result we have is slightly more abstract than we might have set out to go with because there are still limitations in the technology and with the spatialisation to get the accuracy of where the pilgrim is. So in a way you can see them as someone in the piece talks to them as a kind of trail angels where they come towards you and they float away from you. Then we also do have footsteps so we've carefully worked on the kind of the roll-off curve in Unity, so we used Unity as the game engine to work out when the footsteps are coming nearer towards you and when they're walking away. And also different people have different voices that kind of travel in different amounts of time, so we carefully worked out the like roll-off curves there with the voice and the footsteps to be going away. And we also designed it in a way because we know it is a new experience for people to be doing. No one's actually done this before through sound where they latch to something and it moves away from you in that way in a kind of podcast version. So we made chapter one a bit more simple and then the complexity built up in chapter two and chapter three where we thought hopefully the person using it would have begun to understand that relationship they have with the pilgrim as they then go on to chapter 2 and chapter 3 where we build up more of a kind of world and you go into like a different universe as you go into the different chapters through sound.
[00:09:47.727] Kent Bye: Were you taking the recordings that you had recorded from this previous project or did you use like ambisonic recordings? I've imagined that you take the single audio and put it in the Unity and maybe spatialize it within the game engine and so that you get this effect that as you turn your body you got this kind of spatialized sound and so I'm just curious to hear about that.
[00:10:05.944] Lauren Hutchinson: Yeah, so the actual interviews that we did, they are mono audio sources that we did and then we worked to spatialize them in Unity and also, but then the, as I said, we worked with Q Department who made sure that the music you're hearing was spatialized, but there's still somewhere to go with working on the spatialization, yeah.
[00:10:30.399] Kent Bye: And there was three different chapters and I'm wondering if you could describe the different emotional beats that you were trying to gather because you have all these interviews and if you had an arc with each of the stories, if there's different arcs, but what was the common thread between the chapter 1, chapter 2, and chapter 3 pilgrims?
[00:10:48.345] Lauren Hutchinson: Yeah, so that's something we spent a lot of time on because in this experience we have ten different pilgrims and we worked a lot with the editing to make sure that they were in three different emotional states in the different chapters. So chapter one, in a way we tried to abstractly follow the path people go on on the Camino where the first third of the Camino is really based on your body and you're consumed with your body and the aches and the pains of the practical process of walking. Then chapter two, it's more of the mind. And then chapter three is more your soul. So in this experience, chapter two, it's more, also you've got to know the pilgrims by now. So now they open up to you. So chapter two, they start telling you their kind of emotional reasons of why they're going on the Camino. And then we also created a kind of the music to accompany that, to bring that to it. And then chapter three is more about coming to terms with things and about your soul and coming back to this, Also the kind of wisdom that people have learned in their lives from walking 800 kilometers. And so chapter three, the pilgrims are in that kind of state. So we decided against having these kind of like hero stories, as you might say, where there's this like one story you're following through the whole thing. And instead it's really a story of the trail, of the Camino. so we wanted all these different voices and so that's what we do in Chapter 3. I don't want to say too much about it because I want people to try it but in Chapter 3 it's more of a chorus where it's a chorus of voices to make one story which is the trail.
[00:12:16.400] Kent Bye: Oh, interesting. Yeah, it feels like everybody's in some sort of liminal state, transitional state in their lives. They may have gone through something or like this turning point in a lot of people's lives that they're actually walking this Camino. And I like how the way that you actually laid it out in the walk is that you have the different segments where you're going through a similar arc of walking around in a circle, but Yeah, it's interesting to hear about the underlying framework of trying to do it in a way that is not necessarily the viewer's journey, but trying to see how each individual is connected to a larger whole, or that larger whole is the trail. So, what guidelines are you looking at to explore these new forms of storytelling, just to kind of help guide you as you're putting this together?
[00:12:57.902] Lauren Hutchinson: Well, I guess, I mean, we decided to make each pilgrim have a cohesive story within each chapter. So they are saying a few minutes of, you know, there were key things we wanted to make sure you got across. You wanted to make sure it was coherent. what they were trying to say in terms of whether they're talking about what it's like to go on this walk or whether they're talking about what it is about their life that brought them to this situation but it always linked back to the trail but I mean every interview is done on the trail so we also wanted to let it be slightly kind of exploratory in terms of kind of streams of consciousness of the kind of things that people just wanted to talk about what they were on the trail. So that's also why it can come across kind of loose, because it's also about the emotional states of people while they're there and just what they want to say. So we purposely didn't make the interviews overstructured because we wanted it to have that kind of randomness. And so we worked with what people talked about, you know, we didn't over structure it. We read, like we, you know, we listened and thought about, okay, what are they, what do they want to say? And so it does happen to be a lot of them are about heartbreak or loss, but that's what people wanted to talk about.
[00:14:10.723] Kent Bye: Great. And finally, what do you think is the ultimate potential of all these immersive technologies and what they might be able to enable?
[00:14:19.808] Lauren Hutchinson: Well, I mean, my focus is on audio and sound, and I just think we were so excited with this. So Sashka Ansad, who's the co-director of this project, who built the prototype, and we realized it really is possible to make it where you can be walking along to hear the story you stop and you hear the person go away from you and we did that on a walking project based on the Camino but we're so excited about the future of what we've built for telling lots of different stories in this way where you can just be walking you know on any journey but also the way you can be in flow and I think that's what really has excited us with this project is that it's kind of in some ways it has a branching narrative even if it is a cohesive story of the trail but you're not having to consciously think What direction do I want to, like, who do I want to listen to? It's more that you can be in a story that's crafted and shaped, but you're also in a kind of flow and you're navigating your own way through it. And I think that's what's quite exciting with these technologies. And it's screenless as well. You can do that while being in the world.
[00:15:24.809] Kent Bye: Great. Is there anything else that's left unsaid that you'd like to say to the immersive community?
[00:15:29.567] Lauren Hutchinson: just how I'm excited to see more people experimenting with Unity and audio because it's not an easy task, but it is a hard task. But I think there's just so much potential there. So I look forward to seeing what people are going to be coming up with. Yeah.
[00:15:48.171] Kent Bye: Awesome. Well, thank you so much.
[00:15:49.412] Lauren Hutchinson: Okay. Thank you.
[00:15:51.453] Kent Bye: So that was Lauren Hutchinson. She's the co-director of Pilgrim, which was debuting at the IDFA doc lab in Amsterdam. So I have a number of different takeaways about this interview is that first of all, well, just to quickly recount what's happening technically here, because you're holding a phone and that phone has a camera and that camera is using the AR kit and AR core. It's basically seeing where you're walking around in the streets. It's detecting the motion and it's able to detect when you're stopping. and if you're able to turn and so because the augmented reality glasses solution that they had had like spatialized audio but it didn't necessarily have the ability to detect which way that you were facing and so you can imagine if you were holding a phone on your chest and kind of moving your upper body you'd be able to use the camera of the phone to be able to detect where you were actually looking at. And so that was one of the limitations where they're able to do these types of hacks in terms of what's available with the technology, but the technology isn't necessarily designed and built to the point where they're using it. So there's certain things that you can do to get a better immersive experience. But if you hold your body connected to your torso and move your entire torso, then you get this sense of spatialized audio. So they're able to gather a bunch of this information from the context of walking down this Camino de Santiago, which in a lot of ways, people are at this really deep turning point. There's some really interesting stories for why people are taking as much as a month or more out of their lives to be able to do this really long walk. And they're really at this turning point of transformation. And so you're physically walking around Amsterdam and they break it up into these three different sections of your body, your mind and emotions, and then the deeper soul intentions of what your deeper motivations are. And they broke up these interviews into those three different segments. Now, there are a lot of different interesting things that happened to me while I was doing this, and I think it's worth unpacking some of them. So first of all, it's a screenless experience, which means that you do have a screen that you're looking at that is supposed to be giving you these different directions to turn right at certain moments. It's using GPS of the phone to be able to geolocate you to some extent, but I found myself, because I had done a lot of different turns, I think it either got confused or at some point I thought that it was going to turn when I was supposed to, but then I actually turned like a couple of walks before I was supposed to. So it was actually going down the wrong path. But at the same time, I was so immersed within the story and it was able to actually really get tuned in to a part of the story that I may have not otherwise heard that really struck me. It was like such a powerful, visceral moment. I was actually like in tears crying down the streets of Amsterdam listening to this moment within this story. It just really deeply resonated with me. And I think there's an important point here, which is that they're in some ways trying to capture the synchronicity and serendipity that happens when you do this type of thing. For anybody that's gone on vacation or gone to Burning Man, you have these moments where it's just at the right moment, the right time, you've made a series of decisions that your internal state is being reflected by whatever's happening in your external reality. It's what Jung would call a synchronicity. And so how do you cultivate and generate these types of serendipitous synchronicities within the context of telling a story? And I think a big way of doing that is actually providing the user some sort of opportunities to make different choices. You have the option to listen to a number of different people within the context of these stories, and based upon whether or not you're walking or not walking, is allowing you to edit what stories you're receiving. And so you're able to go through these three different phases of the walk and then in some ways start to have people that you've met before, you're revisiting them and getting different aspects of their story. And you're also dealing with the constraints of stuff that are way beyond your control. Like for me, for example, I was listening to a story and I was trying to keep walking, but there was so much traffic I had to stop. And then that cued the AR experience to then go into the next story, which is not the one that I wanted to make the conscious choice, but. It was almost like it was being faded to me that I had to start to listen to a different story, which ended up being the one that was really resonant with me and really struck a chord to the point that had me in tears walking down the streets of Amsterdam. So I feel like that there is some kind of magical component of the difference between the Kronos time and the Kairos time. The Kronos time is something where you've got everything scheduled and planned out and you know, you set a schedule to do an interview, but it sounds like the way that these interviews were captured where they were just walking down the trails of the Camino de Santiago, and that they're recording people who are randomly coming up with them, and they're able to serendipitously run into these same people, and those are the people that they're able to capture and edit and use in their story. And so, whatever those stories they captured were captured in the context that was very much like a serendipitous Kairos time of the moment. And I think there's something really interesting and magical of you also experiencing those serendipitous Kairos time moments in the context of recreating that same type of similar Kairos moment where you are in control to some degree, but there's other degrees of which that you are completely out of control and you have no decision to fully exert your free will and to make choices because there's sometimes just situations that are beyond your control. So I found that that actually produced some really magical moments for me. Even though I had taken a wrong turn and I actually went to go back and redo it, the time that I redid it, it wasn't as magical as the time that I had did it improperly and didn't trigger the third phase because I had, you know, basically walked down the wrong way. so there's still a lot of things to try to work out here in terms of like how do you actually tell a story that's screenless and how do you give people the types of information that you need and also from a production standpoint what are the best practices for how you're even capturing these types of information because anytime you have an interactive narrative you're trying to balance these two tensions between the authorial control where you're trying to really control the narrative tension of some sort of story that's unfolding, but at the same time, you want to allow the person who's experiencing it some sort of choices to be able to decide whether or not they're really resonating with whatever they're listening to. And if they're not, it's okay to stop, just like it's okay to stop if you're actually walking down the Camino de Santiago, and that if you wanted to actually have a different conversation, you could wait for the next person to come along and you could do that. You could stop and have the next story come along. And so they were really trying to recreate this feeling of what it was like to walk on the trail, but obviously you're not walking on the trail, you're walking down the streets of Amsterdam. And so they're taking this content that was of a completely other context, which was all these really deeply moving transformative moments in these people's lives. They're really like taking a time out of life and on this refuge and retreat and really at this turning point of transformation. And you're able to hear those types of stories as you're just walking around the streets of Amsterdam. So, you know, in some ways they're recreating this hero's journey where you're actually physically going out and walking around in a loop. And as you're, you're moving your body through space, you're hearing other people who are also moving their bodies through space, but also telling the metaphoric cycles of their journey and their transformation of going through talking about their physical feelings of what's happening inside of their body. their deeper emotions that are happening, what's happening within their mind, and their deeper soul intentions for what they actually want to do and accomplish in their lives. And I thought it was a really well done, interesting type of experience that was using technology that, you know, actually isn't really built or designed for any of this yet, but they were able to actually make it work. And I think the philosophical implications of what that means is that we're going to be able to start to overlay stories on top of geography. Perhaps we're gonna be able to have this type of immersive theater type of thing where you actually can go to specific parts around the city and maybe that will trigger different dimensions of either individual stories or an immersive theater or maybe at some point it'll get to the point where you're able to be walking down the street with two characters and you're able to listen to other characters and they're able to be walking around in this virtual world and you have the ability to somehow follow them and make this choice where there's this overlaying of story that is equivalent to something like an immersive theater where you are walking around a space and there's a story that's unfolding and that you're moving your body through that space to track that story. Maybe there's a way to kind of virtualize that and trigger the beginning of a story in a loop where you're able to then be able to trace different aspects of the story. And there's other experiences that have been starting to do that within virtual reality. I'll be talking to the creators of Fire Escape, which was a Daydream exclusive that really did this similar type of immersive theater type of thing where there's 12 different rooms and different characters and there's a murder mystery and you have this process of trying to track and zoom into individual stories, but you're trying to essentially figure out who did it, who was the culprit in terms of this murder that you're trying to solve. But The idea is that there's going to be different ways within these immersive technologies that you have these types of interactive immersive stories of a story that's unfolding through space. And this is basically like a very linearized version of the story, very early days in terms of what's possible to technology. But I can see where this is going, where you have these augmented reality glasses with like spatialized audio that eventually is going to get really good with head tracking and everything else. We're going to be able to do some really sophisticated types of storytelling with that. And maybe it'll be these podcasts and these immersive theater types of stories that will be similar to like a docent giving you a guided tour through a city, but it could be at layers of story. It could be your own individual stories. It could be this whole nonlinear way of aggregating context and meaning and story that as you move your body through space, you're able to have an immersive interactive experience with these layers of meaning and story that are being overlaid into this virtualized world that you're experiencing through the augmented reality and virtual reality technologies. So, that's all that I have for today, and I just wanted to thank you for listening to the Voices of VR podcast, and if you enjoyed the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a listener-supported podcast, and so I do rely upon your donations in order to continue to bring you this type of coverage. So, you can become a member and donate today at patreon.com slash voicesofvr. Thanks for listening.
