Generative AI and Virtual Worlds
A few days ago, my team won grant prize the LA AI Virtual Worlds hackathon during the a16z Tech Week by creating a roleplaying game in one day.
This is the result of me spending the last few years obsessed with accelerating game development. A big part of that is generative AI — as well as the work we’re doing at Beamable to enable you to get the servers set up for entire worlds setup on “day zero.”
Along the way I’ve done a number of experiments, as well as conversations with hundreds of people (including a dozen recorded conversations for my video podcast, Building the Metaverse).
Read on for a summary of the most relevant articles and conversations I’ve had in the last 3 months.
By the way, did you know that I’m mostly writing on my Substack, Metavert Meditations? Please link over there to see the latest!
Semantic Programming and Software 2.0
This is the article that covers the programming methods that allowed my team to win the LA AI TechWeek Virtual Worlds hackathon, building a playable RPG in 1 day. We used Anthropic’s Claude LLM (with 100K context), Blockade Labs, Scenario, Beamable and Unity as the key technologies. Learn how we did it:
Open Source Experiments in Generative Gaming
Some of the earlier experiments — which we’ve open-sourced — by myself and my team at Beamable. This demonstrates the use of GPT into the development and real-time operation of games, as well as use of generative images from Scenario:
75 Lessons from Learned from Live Game Development
This isn’t strictly generative AI, but it is my own hard-won lessons from building and shipping games to over 20 million players — as well as some of the failures and learnings I collected along the way. If generative AI can accelerate development velocity, it supports one of the most important pillars of my thesis: more “shots on goal” to get your game right for your audience.
Digital Identity and the Evolution of Creativity
I presented at the MIT Media Lab’s Metaverse course, discussing how virtual worlds started as a place to manifest our digital identity — and are now a platform for our creativity… and soon, our will. The latter is enabled by generative AI agents.
Building the Metaverse
On this season of Building the Metaverse (Metavert.tv), I’ve been speaking with builders and leaders using generative AI to build their business. There’s a focus on games and virtual worlds, but many of these technologies could be applied to a wide range of use cases.
I provide direct links to the YouTube conversations below, but if you click on the titles, you’ll go to a summary article that contains supporting materials and show notes.
Text-to-World
My conversation with Kayla Comalli was about her startup that is using a text prompt to spawn an entire world to explore. We took a look at her product and the supporting technologies like Sloyd, Polyhive and Unreal Engine 5.
Creative Technology and Generative AI with Bilawal Sidhu
Bilawal Sidhu is doing some of the most interesting creative work with generative AI, neural radiance fields (NERFs) and photogrammetry. Some of his examples have been widely shared — and he has over 100M downloads of his augmented reality filters for TikTok. Learn about his work and methods in our conversation:
Generative AI and Voice Synthesis with Zach Johnson
I recorded a conversation with Zach Johnson about how generative AI will change the way we incorporate voices into games and simulations. We also show off some amazing demos of using Zach’s voice synthesis technology inside a flight simulator. We also talk about the nature of art, and the evolving creator economy around AI.
Generative Emotes for Avatars
Expression through avatars is one of the most complex frontiers for generative AI: it covers everything from modeling to rigging to animation. In my conversation with Yassine Tahi, we discussed how Kinetix is bringing together all of these elements and simplifying access through text prompting and a creator economy.
AI Storytelling and Narrative with Linus Ekenstam
Linus Ekenstam is creating a storytelling product using generative AI. We talked about how artificial intelligence is changing product design — and civilization itself.
ChatGPT and Game Development with Cameron Wills
Cameron Wills created the most-downloaded AI tool on the Unity Asset Store. We discussed his journey as an indie game developer, and what led him to make a generative AI tool to speed development:
AI Art, Creativity & Copyright with Kris Kashtanova
Kristina Kashtanova obtained the first copyright for an AI-generated work, her comic book: Zarya of the Dawn. We discussed her journey from photographer to AI artist, and what she’s learned from her quest to obtain copyright on her work.
Generative Graphics Workflow for Games with Jussi Kemppainen
Jussi-Petteri Kemppainen is building an indie game using generative AI. He’s created — and documented — a workflow involving tools that take art from Midjourney, creates projection maps, sets them up in Blender and then into Unity. Watch this video if you’re an artist or game developer who wants to learn how to get generative concept art right into your game engine.
Virtual Beings and Edward Saatchi
Inspired by the Culture Series by Iain Banks, Edward Saatchi’s vision is to create a virtual society with millions of AI-powered virtual beings. We discuss the importance of created simulated environment in which virtual beings take actions — not simply engage in conversations.
Generative Art Assets for Games: Emmanuel de Maistre
Scenario is creating generative AI tools that let you create game assets by training your own generators — and text-prompting individual assets into existence. We discuss how Scenario works, Emmanuel de Maistre’s vision for the future of game development, and his entrepreneurial journey.
Generative Games with Language Models: Hilary Mason
Hilary Mason leads Hidden Door, a startup building multiplayer games built around large language models (LLMs). Before Hidden Door, Hilary led machine learning for Cloudera — and was previously a professor of computer science. We discussed how language models will enable whole new categories of games. Some of the language models that may be put into practice may be simpler, CPU-driven models rather than the ChatGPT-style huge models you might be acquainted with. Learn more:
Further Reading
For the latest from me, please visit my Substack, Metavert Meditations.
