AWE 2023 Day 3: Ultraleap new sensor, LetinAR, and the end of the event

It’s a wrap: AWE 2023 has finished. It has probably been the most intense AWE of the last three years, and this is a very positive thing because it means that the industry is growing. Compliments to Ori Inbar and all the team for having made it possible. On the last day, I tried many interesting technologies, but I selected two for this summary, which are the first two that I’ve tried: the new Leap Motion Controller by Ultraleap and the AR technology by LetinAR. Then, if you want to read something less technical, the last paragraph will be about some memories I will carry on about this AWE.

Ultraleap’s new Leap Motion Controller

I couldn’t leave AWE without trying the new product of my friends at Ultraleap. So the first booth I visited on the last day at the event was their one, and I had a test of the new Leap Motion Controller.

Design

The controller is smaller than I was expecting: it is thin (probably half of the original one), and quite small in the other directions, too. You can see how it is in this picture, where I’ve put my hand for scale.

The upper part is black and the lower part is grey, and overall it has a quite elegant design. On the side, there is a USB-C port for connecting it to the host device. On top, there are the two cameras and the two IR LEDs that illuminate the scene with IR light for the cameras. I’ve been told the device can be so short because the LEDs have been put between the cameras now, not on the external sides like in the IR 170 sensor.

Tracking performance

The tracking offered by this sensor was fucking stable (this is the technical term used by the tracking scientists in these cases).

I first tested the sensor on a Lenovo VRX headset and did some tests on the tracking performances:

• Moving my hands and fingers in a natural way in front of me resulted in a perfect pose detection of my hand and my fingers
• Crossing the fingers of the two hands together resulted in the tracking keeping working and the pose of the fingers being almost correct
• If I put my hand longitudinally in front of the sensor (so that the fingers will all occlude each other). In this condition, moving the fingers resulted still in mostly correct tracking, with sometimes the pinky and the ring finger being swapped (e.g. I bent the pinky and the runtime detected a bent ring finger)
• If I put one hand in front of the other, the tracking of the occluded one could get lost (of course, I would say)
• Doing fast punches resulted in the tracking continuing to work, with just some occasional hand loss that was recovered immediately after. It is starting to get useful for boxing games
• As for the tracking FOV, as soon as the hand entered the FOV of the headset, the Ultraleap system was already tracking it. I tried also to interact with objects outside my visual FOV, and the interactions seemed to happen, but with less precision than when in my viewing FOV.

The tracking was so fast and reliable. I was impressed.

I then tested it on top of the Digilens AR glasses. It’s the first time I could test Ultraleap on AR smart glasses. The demo here was just about using gestures to make an engine 3D model to be exploded into its component. It worked fairly well. Ultraleap people invited me to keep my hands low while doing the exercise to prove that you can interact with AR elements without having your hands in front of you, which is tiresome in the long run (and leads to the Gorilla Arm syndrome). I have to say that I could keep my hand kinda low-ish, but not really at rest, because the vertical tracking FOV was not enough for that. Ultraleap people told me they are working to improve this, too.

Final impression

Overall, Ultraleap confirmed it is a company that offers very solid solutions for hand tracking, maybe the best on the market, at the moment. And I’m sure the new sensor will allow new versions of the runtime to come, too.

LetinAR

LetinAR is a Korean company working on AR glasses that have very special pinhole optics.

Pinhole glasses

LetinAR features AR glasses with lenses that have little holes on them. The holes are actually reflective mirrors that receive the virtual image that you should see, and they reflect it into your eyes. Since the lens is transparent and has only some tiny holes on it where you have the mirrors, you can have augmented reality. You see the virtual elements through the pinhole mirrors and you see the real world on all the rest of the transparent lens.

I tried this system and it kinda worked. It was hard for me to evaluate exactly how well it worked because instead of having a demo in 3DOF or 6DOF, the only thing I could see was a video projected in front of my eyes as if the glasses were just an additional screen for the smartphone with no tracking. The video chosen was, as for all the Korean companies, a video of a female k-pop group performing some of their charming dances.

The FOV was more or less in line with the other AR glasses of the moment (I have been told it was around 45°), the resolution was good (1080p) and the colors were bright. The video appeared as not fully opaque in front of me, and in the beginning, I could see no sign of the holes that composed the image… it was just a full virtual image for me. But then, moving the eyes, rotating my head, etc… I noticed that in some situations I could see the dark-ish halos of the holes in front of the video. So, most of the time, composing the virtual image through many pinhole mirrors worked perfectly, but in some cases, I could still see the circles that composed the image. Also in some other contexts, I saw some small reflective artifacts in my peripheral view, but they weren’t disturbing much my experience.

Ben Lang

While I was demoing these glasses, Ben Lang, my review hero from Road To VR, arrived. I’m always happy to see him since he’s one of my favorite journalists.

While I was focused on the K-pop music video, Ben started instead asking intelligent questions (like if there was a way to make the holes smaller and more numerous, but the LetinAR guy told us that the one shown was the optimal pattern they found). Then, since he is a celebrity, he was taken to a private room to try the new version of the glasses, and I could go with them.

Reflecting bars

In the private room, we were shown a prototype of a new LetinAR technology that instead of using many pinholes, uses a few reflective bars: it is as if every line of the pinhole grid was united in a single reflective bar. I guess the idea is to have a virtual image that is more cohesive.

Also this solution worked, but it had one drawback: its colors appeared less vivid. The reason is that since the bar occludes totally your vision (it has not transparent parts between the holes anymore), it must be only semi-reflective (technically speaking, they are “partial mirrors”) to let you see also a bit of the real world in front of you so that you can have AR. So part of the light of the virtual image gets lost.

LetinAR is going to commercialize the pinhole-based lenses first, and in the meanwhile, work on improving this next-gen solution.

Final impressions

LetinAR proposes an AR optical technology that is different than the one of many other companies, but it works fairly well. I think it still needs some refinement, but it works much better than I was expecting. Anyway, as my friend Ben Lang, was saying, we need more intense tests with real tracked AR content to evaluate it better.

The end of AWE

Before going to AWE, I wrote a post about what I expected to do at this AWE. Now that AWE is over, I would like to say what I actually did there:

• I hugged a looot of people
• I met lots of friends and made new ones. Thank you to everyone that made my AWE special!
• I shook the hand of Neal Stephenson, which had the facial expression of “who the fuck is this guy”
• I had people stopping me in the corridors saying “hey, are you Tony?” and in those moments I had always the facial expression of “Who the fuck is this guy”
• I did not recognize lots of people by their faces, but I had to recognize them by the name on the badge we had around our necks. This is the weird thing about knowing people from social media and not from real life: if their real face is not similar enough to their profile picture, I recognize no one. The best moment happened with Mat Pawluczuk, who I didn’t recognize at all until he said “look, I’m the guy with this profile picture on Twitter” and showed me a brain picture. At that moment I had an epiphany and said “aaaaah this is you!”. Recognizing people from Twitter profile pictures and not by their names or faces is the lowest point that humanity could reach

• Since I recognized people only by their names, and I was also interested in speaking with people from certain companies, I kept looking at the badges of all the people around me. The problem was, badges were laid more or less on our chests. So there was this awkward situation where I looked at the badge of some women around me, but they were moving, so I needed to stare for more seconds to read properly. Turned out that from the outside I seemed a creep that stared at their boobs with much attention, and I received many bad looks from females around me. I guess when the police will arrive to arrest me for sexual harassment, the excuse “I was just looking at their badges” is not going to work (I guess everyone says so)

• I met Tom Furness, which is a legend in the field and have been inspired by his word (thanks Linda for making it possible)
• I met Jazmin Cano in person for the first time, I met Macey from Qualcomm after five years I speak with her via e-mail!

• I tried to have a business meeting with a guy from a certain company while staying in the corridors, but every two words someone interrupted us saying “Hey, Tony, how is it going?” and hugging me. In the end, we had to escape outside the venue to be able to say two sentences in a row without being interrupted. After 10 minutes, anyway, someone came anyway and said “Hey Tony”. There’s no way you can escape from the hugs at AWE! This is fun!
• When people didn’t know what to ask when meeting others, the jolly questions were always “How has been your AWE?” or “What have you seen here that surprised you?”. After these two questions, no one knew what else to say and people looked at each other uncomfortably for minutes. I guess that conversations on social media are easier
• I exchanged lots of business cards with people that will never read them. I think that after AWE there should be a bonfire party where everyone burns the business cards that received and that will never use

• I’ve been to parties where the free food finished before I arrived. This is biggest capital sin of this AWE, and if it won’t be fixed next year, there will be a protest bigger than the French Revolution. Com’on Ori, we all go to AWE just for the free food, and the expo and the conferences are just there to give a context for the free food. If we don’t find food, why should we go to AWE?

• If I arrived at a party and there was still food, I complained for the food taste, because I’m Italian and I have to always complain about food abroad

• For the first time in my life, I’ve been asked to present an award: I was tasked to give one of the Auggie Awards of the event. The winner didn’t even show himself. I do not take this as a positive sign: I guess he preferred not to receive the award than to get it from me. But at least it was a fun experience!

#AuggieAwards 🏆- then a huge congrats to @SkarredGhost for the best Art of Film Award 🤣🤣👏👏👏👏 pic.twitter.com/mDfJND2htV

— Wisear 🧠🎧 at #CES2024 (@Wiseario) June 2, 2023

• I tried a lot of hardware, and I’m happy of having experimented with the current cutting-edge technology.

And that’s it for this AWE! Let’s meet again at the next one! And please, to facilitate things for me, put your social media profile picture on your face so that I can recognize you better…