kim nevelsteen metaverse

Kim Nevelsteen wants to bridge different virtual worlds to create the Metaverse

We all dream of the Metaverse, a fantastic online virtual world where we can all hang out with other people and do whatever we want. Something similar to what we have seen in the movie Ready Player One, or even better.

ready player one review
(Image by Warner Bros)

But nowadays, the situation is very different: we have some rough social VR experiences, that can already give us some satisfactions, but that feel incomplete, with limited interactions and unrealistic avateering. And even worse, these virtual worlds are like silos: they don’t talk to each other, they are separate the one from the others, they are not connected in any way. So you are a kind of virtual person in VR Chat and another one in Bigscreen, and there’s no way to retain what you do in one virtual world when you go to another one.

I find this situation a bit sad, and that’s why I was very intrigued when I found on New World Notes an article about Dr. Kim Nevelsteen, a researcher that is exactly working in creating a connected Metaverse. Or better: in connecting already existing virtual worlds to build a Metaverse.

The video was astonishing: he has been able to create a portal inside a Minecraft server, and if you enter the portal, you enter into another server, this time playing Doom. What is mind-blowing is that the two worlds are connected and consistent: if you take a gun in the Doom world, and get back through the portal into Minecraft, you will find yourself equipped with a Minecraft gun equivalent and viceversa.

This is exactly what we do need: a way to connect different virtual worlds, different ecosystems, so as to mantain a status of the user, a consistent experience during all the virtual journey. This is what also happens in Ready Player One, where Wade Watts can enter the portals of different games.

I wanted to interview Kim to ask him how he got to such a cool result, and here you are his answers on the topic.

Hello Kim, introduce yourself to my readers!

I’ve been pondering solutions for a viable Metaverse since at least 2004. My colleague and I assumed that the people at Second Life and There.com would figure it out, considering the massive investment each of the companies had in their technology. A decade and a PhD later, I have strayed away from the typical academic or programmer career, only to implement a prototype for a viable Metaverse as an Indie.

I have also launched a Kickstarter campaign to create a medical metaverse in 2016, but with little luck. Anyway, I launched it not to succeed, but to raise awareness to the topic of the Metaverse.

Kim Nevelsteen Metaverse
Kim Nevelsteen, while presenting its virtual worlds’ connection solution (Image by Kim Nevelsteen)
What is your job? And your scholastic background?

I’ve been coding since at least 1998, but I stopped my industry programming career to follow an academic path leading to the completion of my PhD. I used to program mainly medical software and still do sporadically, but I try and focus on next-gen virtual worlds. My PhD is in (distributed) pervasive games/applications and mixed reality.

I was impressed by seeing your video where you created a portal between Doom and Minecraft. Can you detail what you have done for people that have not seen this video yet?

What I have done is creating a way to teleport between Doom, Minecraft and old school MUD, without being invasive to those engines.

The concept that is proven, is that with a proper set of conventions (not standards), you can get interoperability between worlds without requiring developers to massively change their implementations. We already have massive amounts of virtual worlds, if we are going to get a Metaverse, these worlds should easily be able to connect to the Metaverse. The video is a recording of a proof of concept.

What is perhaps not obvious is the stages of planning that I’m working through. The rough plan is detailed in my first video. We are currently here:

The Doom/Minecraft/MUD integration was a feasibility test to verify if it is possible to link highly disparate systems without massive amounts of invasive changes to the original virtual worlds.

Since that video didn’t create enough of an impression for people to start asking how it was done and provoke an interest in linking worlds, I set out to the next phase. Find a viable candidate virtual world where the source code was readily available and the end product had a massive amount of users. Minecraft is a perfect choice for this (outside the fact that I haven’t been able to gain any interest from Mojang). The teleporting between servers in Minecraft is a step to make a Minecraft Metaverse. If I can push the Minecraft Metaverse into existence, the next step is to find another candidate willing to link into the Minecraft Metaverse, making it a 2 virtual world Metaverse. From there the Metaverse can expand with more and more virtual worlds until it evolves into a single unifying Metaverse.

There is another hurdle that must be leaped before we can get a 2-virtual-worlds Metaverse. How do we get a universal interface for the Metaverse and deal with various different graphics formats? I have solved this in the Doom/Minecraft/MUD video, but I have recently spoken with the people at Exokit and they have resolved the problem in an even more dynamic way. Combining my work with Exokit might brings us an even bigger step closer to a viable Metaverse with truly mixed virtual worlds.

I see another major hurdle for the Metaverse. That is the one of distribution. We don’t want to require the user to install each world they choose to visit manually.

In the beginning, this might be ok, but we need to gradually evolve a mechanism where a new virtual world is installed in the background and can be started by the universal client. A more distant solution would be where virtual world data is piecewise downloaded to the users’ system in the background on demand. The distribution problem is the reason why I would like to speak with those who have a store: Valve, Epic Games, EA and more.

How did you create this? Can you tell us some technical details?

All technical details will eventually be open to the public, but considering I am running Indie, I must reserve my implementation details until I find someone to invest or collaborate with in this project.

Does this mean that it is possible to create portals between every kind of already existing games? Can I connect Starcraft to Fortnite?

Yes, every single game and application can be linked to the Metaverse. The word “portal” is not always applicable. If you made an application that queried the Metaverse network to find all people currently chatting about “Ready Player One”, that wouldn’t be a portal.

Please also refer to the Kickstarter link above where I describe that we came up with our solution by linking medical software to the gaming world. This should answer your question to what scope can be linked.

How complicated is implementing these portals?

Depending on what you would like to achieve, implementation can be from relatively easy to difficult. Here is my answer to @slhamlet:

Enhancements to the code could be anywhere from super simple to extremely complex, depending on the functionality sought. It is a distributed system, and so the state space can be potentially large.
As an example, consider the difference between querying the network to see if a specific existing server is ready to accept a teleport and querying the network to find all servers open to teleportation. The latter example means dealing with: multiple responses, at different response times, limiting broadcast length (for large networks), and more, while providing a responsive interface to the user.

Also, with respect to (not just connecting worlds, but) mixing virtual worlds. If someone would like to mix the “worlds” of Minecraft and let’s say the Visual Studio IDE, this will require substantial work implementing the communication over the bridge, even though my solution provides the bridge. Exokit already does some of this work. I truly think we can have an industry emerge of companies making mashup technologies for the Metaverse similar to the mashups that are available for the web.

How does it work the system to take the inventory to/from the various games?

It is video game dependent. In the case of my Doom3/Minecraft integration, I treated both worlds has unwilling to accept changes. This meant that only a mapping was possible. A better scenario would be where worlds are willing to collaborate to a certain degree. This means some weapons could literally be in both worlds.

Add blockchain technology and some kind of OAuth, and you could have unique items attached to user accounts.

So, how can I map a bazooka in Doom with the bazooka in Minecraft?

Make a mod for Minecraft where you can have a bazooka, give it the same item id as in Doom; sync the user profile (no mapping needed).

Does this require a long list of mappings for all possible items?

The mapping system is only an example of how we can trivially handle discrepancies between worlds in the beginning. Once people start inter-connecting worlds, people will start devising interesting ways to share inventories between virtual world profiles e.g., blockchain tech. The nice thing about not having standards is that innovation is not stifled.

Doesn’t this create some problems for what concerns the game design? If a bazooka is a rare item in Minecraft, but a common item in Doom, people could all go to Doom taking bazookas and taking them back to Minecraft, ruining the game balancing…

No, absolutely not. The way worlds are inter-connected leaves each world with the power of determining its own physics, interface, avatars, weapons, etc. I think a common pitfall people stumble in when thinking about the design of a viable Metaverse, is assuming worlds must adopt some kind of standard. This is doomed to fail.

Instead of starting with Minecraft, why can’t we start with the new social VR ecosystems? Maybe they are more open towards the metaverse (see High Fidelity for instance)

As I mentioned in a previous question, Minecraft was chosen on the premise that it has massive amounts of users and that the source code was readily available. High Fidelity has still not reached the mainstream.

I am also of the opinion that people need something to do together in the Metaverse. If you have an open-world game like Minecraft you can still sit around and chat. If you have a social VR system like HiFi, then entertainment value beyond chat would still need to be built, if not present.

Is there already a place where people can try a demo of what you have done?

Currently, no. But, as I mentioned in a previous question, I plan on releasing the Minecraft teleportation software before the end of this year.

In your video, I saw the foundation of what I would like the metaverse to be. A world of connected worlds… not only some silos social VR worlds like today. Have you the same vision?

Yes.

IMHO, to make this vision fully complete, we also need a common identity system, so that I can be myself in all VR games I take part, going from one portal to another one. Do you agree?

We already have OAuth and OpenID. But, a common identity system is not even needed, it would just require more programming to provide the interoperablity. Note, that Minecraft and Doom in the demo don’t have the same login database/system.

new oculus avatar review
My current super-elegant Oculus avatar. Oculus avatars are consistent over all Oculus experiences (unless they are overridden by the developer)
The experiments with Minecraft and Doom are just examples…

Calling the Doom3/Minecraft integration just an example is somewhat of an understatement. These are proofs of concepts. It is similar to proving recursion using induction. We prove the base case holds, prove the first iteration holds and prove subsequent iterations hold using the same arguments.

…but can you tell us briefly how do you envision the metaverse of the future?

Some will access the Metaverse with VR/AR, some with mobile, desktop or other. Similar to how we surf different websites now, we shall be able to surf different virtual worlds in the future.

There will probably be at least one world that mirrors the real world virtually. If a user chooses to surf reality, various providers will offer a number of overlays to be displayed on top of the real world. People will have the option of having one or more virtual private places they call their own, which they can decorate as they choose.

People will be represented by one or more avatars that they can skin as they choose, provided by one of many providers. The world the person is visiting might limit which skins are allowed in that world. People might choose to have one or more avatars permanently online to represent them while they are offline e.g., one at work and one at home. There might be an AI that answers questions during an absence. In people’s private store of data, people might have a body scan of themselves so that they can go virtual shopping.

magic leap con round up
Layered structure of the MagicVerse, the metaverse envisioned by Magic Leap. Its structure reminds a lot what Kim is telling in this interview (Image by Magic Leap)

It is possible to go on like this, but the point should be that no one really knows what a Metaverse will offer. Obviously, we haven’t thought of all of it yet.

In your opinion, a future of connected worlds will be possible from a technical standpoint? What do we need to realize this vision?

I’m sure it is possible. There are only questions on in what fashion. For example, will individuals run instances? Will it be in the cloud? Will ISPs start providing instances? Geolocation will definitely start playing a larger role since that will be the limiting factor for latency.

Another example would be how individual worlds will be compensated for their work. I would imagine that some kind of monetary transaction would take place for the time a user spends in a certain world.

And from a business one? Do you think companies will allow that?

“Allow”. Copyright is a big problem. If I were to scrap results off Google and place it on a webpage together with other information, is that illegal? For the Metaverse, a “no” would be the best answer. As for the allow? In my latest video, I didn’t need Mineplex’s permission to link to their servers:

The videos I post seem like very simple demonstrations, but there is a lot going on that people don’t perhaps initially think about.

Why companies should adopt this? In a business world where money is all that matters, how can this take to more profitability for social VR startups?

“Social startups”… the Metaverse is not limited to this niche.

This question is somewhat irrelevant, even though it is not. If I would have chosen the path of High Fidelity, then this would be entirely relevant. High Fidelity has to convince people to use their platform.

Tens of Millions of people already play Minecraft. If I can just capture a small percentage of those people and let them inter-connect, perhaps a Minecraft Metaverse will exist. Getting a second world to inter-connect with the Minecraft Metaverse might still be a hurdle, but it should be easier thereafter. The aim would be that even worlds that are closed would want to open up at least partially, because either users request it, it generates traffic or that if they don’t, they become obsolete.

Instead of building a platform or new world, what I am offering is glue to inter-connect existing worlds. A question that is more relevant is if there will be worlds that actively avoid connecting e.g., perhaps Mineplex doesn’t like me linking to them and attempts to ban such a link.

If a Metaverse evolves into existence, should companies adopt it and can it be profitable? Apply this question to the Internet and the answer is clear. Even for Desktop, I can’t imagine it being confined to just 2D for many more years.

What is your opinion on virtual reality and augmented reality?

The Metaverse will naturally consist of VR and AR. I’m currently writing off VR/AR as “just” different interaction mediums for the Metaverse. In order to support VR and AR, adaptations will be required, but that is also the case for mobile, etc.

If we can get a Minecraft Metaverse started, we can continue building from there. The technology that I’m suggesting in the videos does not have any limitations with respect to catering to VR, AR or mobile.

What are your plans for the future?

Considering I’m running Indie (personal funds), my financial situation will dictate some of the directions I choose. I’m planning on releasing the Minecraft teleportation software.

Perhaps I will run another Kickstarter. Perhaps an investor will emerge. Perhaps I will find a research position that runs tangent with this subject matter. Who knows.

I do have a roadmap in my head on how to make the Metaverse emerge. I have outlined a bit of it in the answers above.

How can people help you in this effort of creating the metaverse?

Definitely… PUBLICITY! I’m currently just screaming out in the wind. It would be great if companies were to start taking notice: Valve, Epic Games, Bethesda, Mojang/Microsoft, etc…

Is there anything else you want to add to this interview?

From my research perspective, those who start talking about “standards” or a “platform” for a viable Metaverse are most likely doomed to fail. We have many examples of this in the past.


I really want to thank Kim for the time he has dedicated to me and I want to compliment with him for what he is trying to do. I think that we absolutely need a way to connect all the different social VR worlds that we have nowadays. The silos scenario is frustrating.

I hope he will get lots of support from the community, and if you want to help him, please get in touch with him on Twitter!


Disclaimer: this blog contains advertisement and affiliate links to sustain itself. If you click on an affiliate link, I'll be very happy because I'll earn a small commission on your purchase. You can find my boring full disclosure here.

Releated

playstation vr 2

The XR Week Peek (2024.03.26): Sony halts production of PSVR 2, Meta slashes the price of Quest 2, and more!

This has been the week of GDC and of the NVIDIA GTC, so we have a bit more pieces of news than the usual because there have been some interesting things announced there. Oh yes, also IEEE VR had its show. Actually, a lot of things happened for tech people in the US, which is […]

We need camera access to unleash the full potential of Mixed Reality

These days I’m carrying on some experiments with XR and other technologies. I had some wonderful ideas of Mixed Reality applications I would like to prototype, but most of them are impossible to do in this moment because of a decision that almost all VR/MR headset manufacturers have taken: preventing developers from accessing camera data. […]