Microsoft unveiled an experiment this week that explores the future of always-on virtual reality. Building a system called DreamWalker, Microsoft researchers walk around in the physical world while still being fully immersed in VR, essentially taking the first steps into replacing your morning walk with something that’s not only reactive to your physical surroundings, but ultimately more interesting.

To do this, DreamWalker fuses a Windows VR headset’s inside-out tracking, two RGB depth sensors, and GPS locations. The system, the researchers maintain in their paper, can continuously and accurately position the user in the physical world, sense walkable paths and obstacles in real-time, and represent paths through a dynamically changing scene in VR to redirect the user towards the chosen destination.

Created by Jackie Yang, Eyal Ofek, Andy Wilson, and Christian Holz, the experiment shows it’s clearly still early days for always-on VR—the system requires a backpack-mounted computer and a load of other gear—however DreamWalker poses some interesting questions related to how such a system could (or rather should) be shaped around a dynamic world.

DreamWalker: created by Microsoft researchers Jackie Yang, Christian Holz, Eyal Ofek, Andrew D. Wilson, Image courtesy Microsoft Research

In the paper, the researchers pose a few methods of keeping you safely on your path, including obstacle avoidance techniques like digital pedestrians moving you away from potential danger and controlling a user’s paths with dynamic events such as vehicles being parked in front of you (video linked below). And while randomly spawning traffic cones or a wild gang of pedestrians herding you to your destination may seem like inelegant solutions for now, there’s no telling what a smarter, more integrated system may hold in the future.

SEE ALSO
Microsoft Files 'Virtual Reality Floor Mat' Patent, Possibly Aimed at an Xbox VR Headset

For all its marketing bloat, Magic Leap poses such a highly integrated system in its hypothetical Magicverse that would necessarily require a fairly complete understanding of the physical world, including pre-mapped and digitized streets, buildings—everything so you could potentially ‘reskin’ the world to a varying extent.

Microsoft’s method is decidedly less involved than Magic Leap’s moon-shot idea, which thus far has only been presented as a hypothesis more than object of active experimentation. The researchers instead take the user’s planned walking area and match it up as best they can with a digital map, introducing re-directed walking when needed to curb the user from veering off course.

A: the real-world path B: the digital map with planned redirection – Image courtesy Microsoft Research

Of course, neither AR or VR hardware is capable of doing all of this for now, however it’s not out of the question for future devices. Whatever off-the-shelf parts the researchers have strapped together are more than likely to find themselves on future headsets in some form, which includes greater rendering power, better computer vision, and on-board GPS for better location-based play.

Moreover, these early steps are predicated on the very real assumption that the better AR/VR gets, the more time we’ll spend interacting in digital environments, making something mundane like a walk through the park as exciting and novel as creators can make it.

Check out the four-minute video below to see the research group’s findings it in action:

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Scott Hayden
Well before the first modern XR products hit the market, Scott recognized the potential of the technology and set out to understand and document its growth. He has been professionally reporting on the space for nearly a decade as Editor at Road to VR, authoring more than 3,500 articles on the topic. Scott brings that seasoned insight to his reporting from major industry events across the globe.
  • Ardra Diva

    It could be a generation away but the scenario i imagine as being totally real some day is something like this: Your ride on a city bus looks like you’re in a pirate ship, because you want it to. The bus driver is actually an Orc, because you want him to be. Really interesting times ahead.

    • Honestly can’t see this ever becoming a real feature. The liability alone would be insurmountable if there was even a slight glitch users could end up in oncoming traffic – let alone if a hacker maliciously misguided them. It’s hard to imagine a company being so confident they would risk killing or maiming someone, not to mention the inevitable resultant lawsuit, for a feature with such questionable value.

      Then there’s the risk/benefit analysis for the user. What exactly are you getting out of this that you would risk death for it? Is pretending you’re walking through a fantasy with fake characters instead of the real world with actual people worth the risk? What is the psychology of this escape from reality, turning away from opportunities for real interaction and instead investing in an alternate reality?

      • m m

        look for “hyperreality” on vimeo or youtube

        • That’s augmented reality. I agree it will become a dominant technology as it matures, I don’t however see something like “DreamWalker”, a total replacement of all your surroundings, ever gaining a serious foothold.

          • I agree, this is more AR’s area.

          • dk

            a total replacement anywhere is pretty hard and unpractical ….but a small form factor headset capable of great quality ar and vr is doable like the old nvidia prototype lightfield display but the panel resolution and gpu and eye tracking has to improve to get a consumer headsets like that ….and u will go to full vr mode in a dedicated vr friendly area like a vr-park …not all the time

          • I think AR will become a more and more regular part of life the closer we get to being able simply add it as a layer to normal glasses during a visit to the optometrist. I can’t see anything like “Dreamwalker” becoming prevalent though. AR can add value to your life by overlaying useful information onto the real world, and VR can be a fun and interesting dive into other worlds, but attempting to completely replace the real world with something like “Dreamwalker” is not useful and frankly kind of creepy.

            I can understand wanting to escape reality occasionally, but all safety concerns aside I find the psychology that would inform an individual wanting to permanently replace it in everyday aspects of their life to be a bit disturbing – especially if it somehow actually became popular enough to justify industry investment to the point it would need to be to actually create a functioning consumer version that could pass muster from liability concerns.

      • alboradasa

        I suppose you could say the same thing for driverless vehicles, in terms of glitches causing accidents.

        • Kind of, except:

          – Cars are inherently incredibly dangerous whereas driverless cars are an improvement upon the existing technology that human drivers represent, thus making driving much safer

          – Walking around with your eyes open is inherently an incredibly safe activity that could only be made much more dangerous by completely covering your eyes

          – Given these two diametrically opposed realities this is not only a poor example, but illustrates why the liabilities involved with a company attempting to supply either technology are vastly different

          • alboradasa

            Hey no need to be so arrogant. Both examples involve putting your life in the hands of technology and 100% relying on it for your safety, whether it’s an overall improvement of safety or not-
            sure, the liabilities are different but that doesn’t mean it won’t ever happen. If the technology is good enough to convice people to get into 80mph driverless cars without issue (even though they have actually been hacked already), then it’s not too crazy to imagine that people might rely on the same tech to be able to walk around at their own pace.

          • Hey no need for insults. My avatar may look smarter than you but it’s just a cartoon don’t feel threatened. I’m just presenting the facts as I see them. These two examples in my opinion objectively represent fundamentally different value propositions. One is attempting to solve a problem by addressing one of the leading causes of preventable death in the world, potentially saving millions of lives each year.

            The other is a feature that solves no problems and could potentially make your life more dangerous. When you are on foot you’re not traversing purpose-built concourses protected by a metal cage. You’re a vulnerable bag of meat that’s relying on social cues to navigate a world of not just comparatively unpredictable physical obstacles but inter-personal ones. How are you going to hear or make eye contact with people to communicate or ascertain any of an innumerable amount of things from the mundane intricacies of coming to a doorway at the same time as another human, to the dangerous of sensing a person with ill will towards you?

            I’m not saying it’s “insane” to think such a thing could ever possibly pass into some form of legal use that doesn’t nearly instantly result in a cavalcade of lawsuits which bankrupt those companies, I’m saying I think it’s very unlikely that the specific hurdles it represents coupled with a very dubious value would ever grant it a large enough market to survive. VR as a dominant and relevant technology seems to me to be very much relegated to safe spaces such as ones own home or an arcade, and largely not because of technogical or insurance hurdles but social ones, though it certainly does also represent many more difficulties in those areas as well.

          • alboradasa

            You’re talking as if this is a product about to go to market, rather than the first steps into a technology that is probably decades form being used in everyday life. Technology will improve to the point where the advantages of using it will likely justify the risks. Just like driverless cars. I don’t think the hurdles it represents will be insurmountable with enough time, research and tech. And the value of it will no doubt come from entertainment and advertisersing industries- just because it’s not “solving problems” and makes your life (probably a tiny bit) more dangerous doesn’t mean there won’t be a market for it.

          • To clarify – are you talking about AR generally, or this “DreamWalker” type of technology specifically? I do see a number of valuable uses for AR and I’m certain it will relatively shortly become a widely used technology but I just can’t see it with something like “DreamWalker”.

            If you are talking about “DreamWalker”, even in the far future it seems incredibly unlikely to me, as I said mostly due to social hurdles rather than technological ones as those could potentially be addressed at some point, but the social hurdles do couple with technological ones in interesting ways where this is concerned as well.

    • Foreign Devil

      We can all experience schizophrenia.

    • dk
  • dk

    hmmm dual-band gps

  • Very interesting experiment. I wonder how many times the system provided the wrong path, though. In the videos we can see only when it works well… what is the failure rate, instead?

  • VrDreaming

    I can see a lot of practical purposes for this, not as an overlay to your everyday world which has some challenges already stated but more in an amusement park scenerio.

    All you need is a large private space that is completely mapped and monitored, then you pay for a weekend in this alternate reality – this overcomes one of the biggest issues in vr, locomotion. Here the play space would map to the physical space one to one, and for an added sense of realism and all the physical objects would match too; chairs, doors, guns etc. This would create an incredibly immersive experience. The costs would not be so high because you don’t see any of the physical space only feel it.

    By tracking and monitoring all the players it would provide for a very safe environment.

  • Mighty Pumkpin

    So pumped

  • Mythos88

    I think it would be more practical to take it out into nature. Imagine seeing yourself hiking along a treacherous mountain pass with beautiful vistas instead of just strolling along an everyday park trail.