BETA
This is a BETA experience. You may opt-out by clicking here

More From Forbes

Edit Story

The Secrets Of Magic Leap

Following
This article is more than 5 years old.

Part one of three in a series.

Patrick Finley

There’s a crate in the corner of Magic Leap founder and CEO Rony Abovitz’s office. “Inside that box is Hiro's samurai sword from Snow Crash.” [Snow Crash is the seminal 1992 novel by Neal Stephenson which introduced the idea of the “metaverse” into popular culture.] “That is the one of one, actual sword, officially spec’d out by Neal,” said Abovitz, 48. Stephenson is now Magic Leap’s resident futurist. Abovitz promised to make him the sword when he joined the company in 2014, and it’s finally arrived. The gifting ceremony is pending. “That’s the nerdiest thing I think I’ve ever done,” Abovitz proudly says.

“It’s taken almost five years for a master swordsman to make this sword.” Abovitz continued. “It’s not a prop. It’s been folded over 20,000 times. You could go to war with that thing. Isn’t that awesome?” Abovitz beams beneath a mop of thinning, curly gray hair and a yarmulke. Behind him are other props, drawings, and a full-size ray gun model created for the first seminal Magic Leap experience, “Dr. Grordbort’s Invaders,” made by Magic Leap and Weta Workshop in New Zealand.

Fan Art by Joe Probition

The Beginning of The Beginning

According to his college classmates, Abovitz was a combination of semi-serious alternative artist (he was a cartoonist for the U. Miami paper, “The Hurricane”), and part serious engineer. He got his B.S. in Mechanical Engineering and his M.S. in Biomedical Engineering. Abovitz left the Ph.D. program at U. Miami (he continues to have a close relationship with the school) to start Z-Kat, an R&D think tank he created in his now-wife’s grad student apartment. Perhaps most unexpectedly, Abovitz was also a college athlete at Miami, which is a highly competitive Division I school. He walked onto the track team as a freshman and eventually was invited to join the team as a javelin thrower, one of his proudest accomplishments. In many ways, Abovitz is a prototypical nerd, but in other ways, not. There is always an unexpected twist.

“I used to haunt [computer graphics convention] SIGGRAPH when I was a grad student in the early 90s,” he says. VR was a hot topic then. Jaron Lanier, creator of VPL, famous for creating the “data glove,” and Bran Ferren, Disney’s then-VR guru, were pushing the edge of what could be accomplished with the bulky technology of the time, and they showed it off at SIGGRAPH. “The Imagineers were a huge inspiration. I was super inspired by books like Neal Stephenson’s Snow Crash and movies like The Matrix.” Years later, both Stephenson and John Gaeta, who won an Oscar for the mind-bending visuals of The Matrix, would join Magic Leap’s staff.

In the late 90s and early 2000s, Abovitz explored every VR system he could as a possible robotic surgery visualization solution. Doctors have long complained of “screen pollution,” where needed information requires them to look away from the patient to check a monitor. Surgical visualization of robotic instruments on another two-dimensional screen aggravated an existing problem. “Magic Leap would have been an amazing solution to have then and through the work of our amazing partners in the medical field it soon will be.”

One of the things I find most interesting about Abovitz is that he’s never worked for anyone else, other than his board of directors. He did odd jobs as a kid. He worked for his dad. He had an internship, but never a boss. Mako’s rise to a public company, and its subsequent acquisition by Stryker Surgical for $1.65 BN in 2013, was a sixteen-year marathon, punctuated by moments of terror when things were almost derailed, first by the terrorist attack of 9/11, then by the financial crisis in fall of 2008.  

For Abovitz, Stryker’s acquisition of Mako was a watershed event in many ways. First, it gave him long-awaited financial security, because CEOs of public companies like Mako can only sell shares slowly and in certain windows. Following the acquisition, Abovitz was liquid. He used the money to pay off student loans and to buy a comfortable house, where he built a music studio. Abovitz spent long hours there, playing music, and thinking about what he really wanted to do. He painted Magic Leap on the wall when it was nothing more than a compelling idea.

“When I was at Mako, we did robotic surgery, really serious stuff. FDA-level. But to me, it was like I was doing Star Wars droids. So my mindset was really different from all the other people in the field,” Abovitz explained. “I’m making Star Wars droids, and I’m inside the med-tech world, trying to mentally integrate where I’m coming from. I would go from SIGGRAPH, to bone and joint meetings, and neuroscience meetings. But my brain wasn’t there, it was in the movie world… the medical device world doesn’t really blend creativity and technology in the same way.”

[Author’s note: in 2010, Dr. Arthur Kobrine used Mako’s robotic microsurgery tools at Sibley Hospital in Washington, DC, to save my life when three discs in my neck collapsed. I never thought to ask Dr. Kobrine about the tools he used until I heard of Mako.]

Abovitz characterizes the period of 2011-2013 as “serious garage days,” but emphasized he was not alone in his explorations. He had contacted Weta Workshop in 2011, which he describes as “adopting him.” With Richard Taylor (who later became a Magic Leap board member) and Greg Broadmore, Abovitz began to develop what would become “Dr. Grordbort’s Invaders,” which had a great influence on the development of the Magic Leap One on the other side of the world.

Magic Leap

After the acquisition of Mako, things began to happen quickly. “No matter who you are, when you start something, including Z-Kat, you can’t do it alone. Entrepreneurs have to have partners. They have teams of people. The early people are so important.” Even though the company is made in his image, Abovitz insists the credit for creating Magic Leap lies with his collaborators more than himself.

As the Magic Leap idea began to coalesce, Abovitz undertook a quixotic journey, iterating his ideas about spatial computing with artists, filmmakers, authors, engineers, and geeky friends who spent hours with him on the phone and in the divey restaurants he prefers for their anonymity. As he refined his vision, Abovitz carefully broke the problems into pieces, and looked for solutions and workarounds, and the people who could help figure it out. When pressed about this… well, magic leap... to a broader conception of AR, Abovitz demurs. “I just wanted to make something really cool, and thought it would be awesome to really do it.”

Photo: Charlie Fink

The Four Wise Men

Abovitz credits four people for helping him refine the Magic Leap vision into reality in those early days: Richard Taylor of Weta Workshop, Sam Miller, formerly of NASA, his friend Graham Macnamara, a physicist who went to Caltech and Scott Hassan, who has been characterized as “Google’s unknown third creator.” He introduced the companies in 2014.

“I got to give credit to my cynical friend [Graham Macnamara],” said Abovitz. “We’d get into these eight-hour debates about things... And he’s like, well, you got to do the real thing if you’re going to do it.” Macnamara, now Magic Leap’s Chief Creative Scientist, has been friends with Abovitz since ninth grade. He went on to study theoretical physics at the California Institute of Technology and the University of Miami but became disillusioned with academia. He wanted to do something more practical than physics but resisted Abovitz’s entreaties to join Mako. Macnamara was exploring an alternative career in architecture when Abovitz started to puzzle through how Magic Leap might work with him.

Magic Leap

“The brilliant thing with Graham was, at some point, we stopped arguing about, is it physics or biology? We decided it was somehow both. Which I don’t think our industry still understands. And that’s probably the most important thing about Magic Leap… our number one science obsession in the company,” Abovitz says.

Macnamara and Abovitz set out to “to properly fool the brain” in order to create multi-planar spatial computing. To do this they broke down problems to be solved. “We tackled every problem, laid them all out,” Macnamara told me. Unsurprisingly, hardware presents a number of challenges. “Optics are the last hurdle. Only a true multi-planar light field display can merge information and abstraction with the real world convincingly,” said Macnamara. At the time their ideas seemed strange to him, almost psychedelic. “Rony’s brain defaults to the future and then he works backward.”

Macnamara was already working on Magic Leap’s first patents when Abovitz connected with a NASA scientist, Sam Miller. There emerged a white paper that “pulled together all the ideas about spatial computing that have been science fiction stuff for years and turned it into a product,” said Macnamara. After a year of working on the initial technical patents as a contractor, he became one of Magic Leap’s first twenty-five employees.

Abovitz describes an apocryphal visit to a friend in Hollywood who ran a music label and guided him to meet “the right kind of people, who wouldn’t eat me alive,” not those would smell his fanboy enthusiasm and pick his pocket. Abovitz bemusedly relates how he prepared for the trip by binging Entourage and watching Swimming with Sharks. Storytelling and movies are deeply embedded in Abovitz’ psyche, even though his own system is not yet fully capable of telling the kinds of stories that inspire him. In addition to John Gaeta, a lot of the senior creative people at Magic Leap have a background in entertainment and movies.

Photo: Magic Leap

No one compares to the relationship with and influence of Richard Taylor of Weta. “I had found kindred souls on the other side of the world,” Abovitz told me. Weta is the design and special effects shop founded by director Peter Jackson to create the award-winning special effects in movies like The Hobbit. Based in Wellington, New Zealand, Weta now hosts a team of embedded Magic Leap employees, mostly technical, who work shoulder to shoulder with the creatives there. “Here at Magic Leap, we have a lot of ultra high-end tech and software people,” said Abovitz. “But they need to be combined with creative minds. They are in New Zealand, far away — off the grid in a way — and we want that. They don’t limit what they are attempting for financial reasons, necessarily. When they go for things, they do it because they want the best,” he said.

Abovitz met Sam Miller at the Conference on the Future of Engineering Software, which is presented by Miller’s grandfather, the legendary software designer Joel Orr. Miller was a ten-year NASA veteran, having started as an intern and worked his way up to Creative Scientist. A Ph.D. with a knack for hacking the government from the inside, Miller worked on a range of projects, from cybersecurity to rockets and robots, which made him a perfect collaborator for Abovitz, who seems to have a talent for finding talent exactly when he needs it. When I talked to Miller in December 2018, he told me our conversation was taking place exactly seven years to the date he met Abovitz and found a mission big enough, hard enough, and compelling enough to entice him away from NASA.

Miller was dragged into the Hollywood phase of ideation as well. “Some of those people are crazy. I mean really crazy. Crazier than we are,” said Miller, describing his conversations with Abovitz’s non-technical creative advisors.

Like Macnamara, like Abovitz, Miller has a romantic side and a practical, engineering side. It took a year of talking about spatial computing with Abovitz, of breaking it down into can I do this? “Talking is exciting, but you don’t leave the best job in the world, drag your family to Florida, literally change your life for an idea. You have to get there technologically. It’s world-class talent, inspired by vision, and execution. There are hundreds of difficult little pieces, but by breaking it down we make impossible merely complicated,” he said. I hear that the phrase, “make the impossible merely difficult,” from a lot of Leapers.

Art: Magic Leap

As Convergence prepped for print in January 2019, Abovitz and I traded messages on Twitter. “You’ve only got 20% of the story,” he said. To really tell the Magic Leap story with the depth and nuance he would like, I’d need to write a whole book.

End of part one of three. Here is a link to part two. Part three will be published tomorrow. Together, the three parts are a chapter in my upcoming book, Convergence, How The World Will Be Painted with Data, which will be released March 12, 2019.