Wisear lets you control XR with your jaw

I’m a big fan of products that try to bring something new to XR and that’s why I have been very happy of trying Wisear AWE 2023 in Santa Clara. When I was contacted by the company, from the name “WiseAR” I thought it was the nth solution for no-code development of augmented reality applications, but then I actually discovered that it was something much more interesting than that…

Wisear

Wisear is on a mission to find new ways with which people can interact with technology. Its dream is to remove devices like the mouse and the keyboard, which feel dated and also not suitable for new technologies like AR and VR, and to substitute them with something more natural. The idea is not the one of creating a next-gen XR controller or using hand tracking, but to try to detect the intention of the user from its biosignals. The final goal is so to create a sort of brain-computer-interface (BCI), a technology I’m personally a big fan of because we all know that it will be the final form of human-machine-interactions. This is how Wisear defines its vision on the website:

We are connecting humans and computers like never before by being the next big step in human-machine interfaces. We are making this technology available to millions of people by integrating into everyday devices. 5 years from now, we believe most of our everyday devices will be equipped with biosensing solutions, and we plan to become the neural-interface solution that is used in any device to read, process & interpret your bioelectrical-activity to make your life easier.

Fascinating, isn’t it?

Wisear earbuds

The first product through which Wisear wants to fulfill its vision is a set of smart earbuds that you can use to control electronic devices. These earbuds are something like a next-generation “mouse” you can use to “click” on a computer, smartphone, or XR device. I’ve had the occasion to discover them by meeting at AWE Yacine and Lauren from the company. They showed me a demo of the device, which was for me one of the most interesting surprises of the whole event.

The earbuds were quite bulky, but because they feature much more than the standard earphone. First of all, the rubber part is covered with sensors, to detect the bio-signals, in particular the signals that are sent to the muscles of the face to contract them. Then, right after the sensors, there is a little board to analyze the data coming from them. Having a local data analysis system makes the device a bit more complex and big, but at the same time makes it safer from a privacy standpoint, because the raw data never leaves the device; and also faster, because the big stream of raw data is analyzed directly on hardware, and to the host device is sent just the final result of it (click or no click), which doesn’t require much bandwidth.

The two earbuds for the two ears were connected through a wire, which is used to detect the difference in electric potential between the two ears. If you remember what you studied in physics in high school, what counts in electricity is the difference of potential between two areas, and it is this difference of potential that makes electricity flow. The wire is useful to calculate this difference of potential between your left and right part, which is the metric to evaluate the strength of the electric signals that are sent to the left and right parts of your face.

The system reads this raw electrical data and then looks for some specific patterns, in particular where the signal shows a peak in certain frequencies. When these peaks are found, then the system confirms that a “click” has happened. A click, in the current implementation, is when you contract the muscles at the base of your jaw, a bit like if you had to grind your teeth, pressing the lower teeth against the upper ones. The system can actually also detect double clicks and triple clicks, which are made by a small succession of these grinding movements.

Yacine showed me the system at work. He connected the earbuds to his phone via Bluetooth and then opened a video streaming service. He played a video, then he made a quite invisible movement with his jaw, and the video paused. He did the movement again, and the video restarted. It was cool that he did not have to fully grind his teeth, he just had to do a small contraction of his jaw muscles. Then he did like a “double click” and the application went to the next video. He was able to control this app without using his hands but with just micro movements of his face. This was cool.

As usual, I couldn’t stop myself from saying a joke: “It should be hell to watch videos while you are eating, because this would send a million click signals to the app”. Yacine actually answered that since the system works by analyzing an exact pattern of frequencies of the electrical signals, it is immune to this problem. Chewing when you are eating is very different from having the contraption needed to have a “click”, so it is just detected as noise. He showed me: he started chewing normally, and while the data analysis tool showed a lot of spikes of the electrical signals, nothing was actually detected as a “click”. That was pretty cool, again.

Eye tracking

Yacine went on to say that Wisear is interested in all forms of natural interactions, and so he wanted also to show me a prototype of something that may be coming. He said that Wisear is also working, among other things, on eye-based interactions. In particular, he showed me a tech demo that was reacting to the movement of his eyes to the left and to the right. He said it is still a work in progress, but he wanted to give me a preview anyway. He so showed me that when he was moving the eyes completely to the left, the app could detect it and react accordingly, and so was happening by moving the eyes to the right. Having tried eye-tracking systems since the aGlass DK2 of 7Invensun, I was a bit rolling my eyes (pun intended), because this was nothing special compared to other eye-tracking demos I had in the past.

Then he said, “of course, this eye tracking is still performed via our earbuds“. At that moment my brain exploded.

“Earbuds that can track the eyes” was not on the list of things I was expecting to try that day. It was kinda mindblowing to think about it. And of course, it makes total sense: if the earbuds can detect the electrical impulses sent to muscles, since the eyes have a lot of muscles around them, it’s obvious that Wisear can track them. Of course, they can not “track the eye” pupil reliably as Tobii does, but they can just detect when a certain movement of the eyes (e.g. rotation to the left) is happening and this can still have some interesting applications.

While it was still a prototype, so I can’t comment on its reliability, this was one of the most surprising moments of my whole AWE.

Hands-on Wisear

Of course, after having seen so much awesomeness, I wanted to do something with it myself. So Yacine let me try the earbuds. It has been very romantic sharing our ear wax.

They let me use their data visualization app to teach me how to perform the “click” gesture with my jaw. I could see the electric data read from the sensor and then when I performed the right gesture, the graph line changed color. Yacine tried hard to teach me, but it was quite difficult for me to reproduce the right movement. It was weird because even if I grinded my teeth, the gesture was not always recognized unless I applied a lot of force to my jaw. Yacine gave me a bubble gum and strangely, using the bubble gum while doing the gesture resulted in better click detection. I was quite surprised by that. I guess the bubble gum forced my jaw to apply pressure in a slightly different way, which resulted in those exact electrical pattern the recognition system was waiting for.

I was so able to try the video app, play/pause videos and go to the next one by just using my jaw. Anyway, even with the bubble gum, I had some problems with performing the clicking movement, and especially many “double” and “triple” click went misdetected. This was strange because instead when Yacine was doing that, his gestures were detected immediately and in a super-natural way.

When it worked, anyway, it was very cool for me using it. I had never controlled a device with my jaw!

Partnerships and vision

Wisear announced many partnerships with XR companies at AWE: for instance, there were common announcements made with both Pico and Digilens. The idea is to let people control XR devices in an innovative way. One of the possible final goals is to let people type on XR devices by just mimicking the speech movement. Imagine pretending to speak, but without really speaking, and without doing the full speaking facial expressions, just a subtle version of them. That would be fantastic, so you would have all the advantages of voice commands, but without its disadvantages (i.e. screaming in a room). Their mission reminds me a bit of the one of CTRL+LABS, the startup acquired by Facebook/Meta that was able to use subtle hand movement to command electronic devices. It’s all very fascinating.

In the short term, instead, the idea is to miniaturize the earbuds. They are a bit bulky, because Wisear has the BMI expertise, not the expertise of miniaturizing earbuds. So they are looking for a partner with the expertise in building such devices that can help them in performing the miniaturization. In case you know someone that could be interested…

Final impressions

Wisear has been one of my highlights of this AWE. I love how this company is trying to create new natural ways for us to interact with technology, and the choice of performing gesture tracking with earbuds is very smart because many people already spend their days with the AirPods always on. Imagine being able to reject a call while you are in the crowded metro by just using a small jaw grinding… this would be very convenient. And I love even more the vision of typing in VR by faking to speak, this would be ideal to remove keyboards from XR, where we know they don’t work well.

I liked a lot of this company, but I’m also aware that the road in front of them is long and complicated. When I tried the system, I wasn’t able to perform the required gestures in a natural way, meaning that in this phase a per-user calibration and a training phase are needed. This friction is not good to onboard new users. And of course, we can’t force people to eat chewing gum every time :). False detections should also be totally removed, because it shouldn’t happen that at a certain moment I’m stressed, so I grind my teeth, and this is detected as a “click”. Then the device has to be polished, miniaturized, etc… etc…. I think you get what all my points are. Seeing this system working flawlessly while Yacine was using it with the demo app was cool, but we need much more than that.

I hope this company will stay around for the next years because I’m very curious to see how the Wisear solution could evolve over time. I see a lot of potential, and I would like to see it fulfilled in an actual commercial product.

(Header image by Wisear)