Exclusive: New Magic Leap Patents
Predicting how their AR Glasses, Helmet & Belt Pack will work
I. Overview
On November 17, 2016, two new patent filings from Magic Leap were released by the USPTO. I’ve gone through them and extracted out the salient points. Included below are the actual drawings of the main device and peripheral devices that will run the technology, as well as my prediction of where the video industry is headed when viewing Augmented Reality out into the distant future.
If you read the November 2016 Fortune piece of the founder, Rony Abovitz, you know that they began debugging their high-volume production line (Made in America!) over the summer with likely plans to sell the product to consumers for the holiday 2016 shopping season.
However, considering it’s nearly Thanksgiving, that goal looks doubtful and we’re probably looking at a mid-2017 launch date. Of course, when you raise $1.4 billion (with a ‘b’) in a few short years and have hundreds of people working out of sunny Florida, anything is possible I suppose.
II. The Two Patent Filings & Historical Rhythms
Magic Leap’s latest two patent filings cover two topics that are likely to get more press in the coming months when the rest of the industry begins to hear about it, though there isn’t much in here that hasn’t been talked about already. They are:
- Biometric User Recognition: think Facebook face detection, but in the real world. It’s using a neural network approach to identifying people through
- Cameras on AR Displays: think Snapchat Spectacles. The device has multiple sensors but the camera is detachable. Hmm.
It’s pretty clear that these two patents and technologies fit together. One is about the camera as a sensor feeding images and the other is about recognizing a friend.
One thing is made abundantly clear about Magic Leap, however, that will set it apart from much of the industry. That this system is for being mobile. While the rest of the VR industry requires tethering via a cord to a very powerful desktop, the AR devices need to be fully self-contained while also communicating back to the cloud in order to function as desired.
And it appears from the filing that instead of keeping all the processing in the “helmet”, which makes things you wear on your head bulky and hard to wear for extended periods, Magic Leap will do something different. They will include a tether (i.e., headphones) that link down to a “belt pack”. I liken it to the old cassette tape players that Sony produced in the 80s.
Whether history repeats or rhymes, it matters not.
There is much to be learned about humanizing tech in these photos. In the first, a couple does a mobile activity together while listening to the same song. AirPods anyone?
It also shows a guy wearing, get this, glasses while working on with a big rectangle on his hip. If that doesn’t show you the inspiration for Magic Leap, I don’t know what does.
Imagine how both of these very human experiences could be improved with AR glasses instead of standard sunglasses or eyeglasses. The couple could be playing a Mario Kart like game with each other overlayed on the sidewalk while they exercise. The gentleman could be racing Lance Armstrong, and beating him (!) at his local YMCA.
So, reading the tea leaves, as well as these patent filings, lets me predict that Magic Leap is also working on multi-user agent interactivity. Much like how Facebook showed the multiple VR avatars sitting around a table, the real winning use case of AR won’t be about doing solo work, but about doing it together. Instead of cross-continental flights for face-to-face meetings, you can now manipulate a 3D object in front of each of you in AR space.
The user recognition aspect of this could be an encryption and security system. For example, imagine you’re getting on a conference call with your global team working on a new proprietary propulsion system. You pull up the 3-dimensional model in the middle of the table you’re all sitting around, virtually of course. Only the people with the Magic Leap glasses on can see the object, and only those people who have been specifically invited and authenticated by the user recognition system are allowed to see the physical prototype.
You know how secretive Apple is. The problem is you can’t let the hardware people see the software and vice versa. So, this would allow you to all sit around the same table and talk about the same project where only certain people could view certain parts of the product.
Weird, I know. But an apt use case.
III. The Patent Drawings
The interesting part about the drawings is how it shows a multi-part device. It appears that one is Oakley-style glasses, another is a helmet that covers most of the head aside from the glasses, and the final piece is a little cassette player-sized box you attach to your belt and wear on your hip.
That says to me that there will likely be two versions:
- Lite: Glasses plus the Belt Pack for processing power for lightweight, less graphics intensive use cases for everyday wear. Includes only 1 sensor, like a camera for mapping your environment.
- Pro: Glasses plus Helmet plus Belt Pack to add computing power for a richer and more immersive VR-like experience. Includes multiple detachable sensors and, interestingly, the option to affix the “belt pack” to the helmet. There is also the wired attachment for, as they describe, “high bandwidth applications”.
The next aspect of the patent drawings has more to do with what’s happening with the software vision system projected onto the eye, rather than what the hardware looks like and how it will work.
What it is describing is the use of deep learning techniques called a neural network that will take sensory input from the video cameras, detect what objects you’re looking at (e.g., a lamp, a desk, a chair), and then pro-actively project the augmented visuals onto your retina at the same time as if you were adding a photoshop layer to a real scene as you moved your head around the world we live in.
It’s not so much a magic leap as it is software, services, and hardware all working together really really well. That magic just cost about $1.4B in funding and took hundreds of really smart people years and years to pull it off. So, well, I guess it is kind of magical when you put it that way :)
IV. What It All Means
As we’ve often stated in Humanizing Tech, there is a continuum from real reality to augmented reality to virtual reality. Magic Leap’s device appears to take a page out of that playbook. On the Lite side, it’s more towards the lightweight AR applications like email or perhaps voice/text messaging, but on the Pro side, it’s more towards the immersive side which I would think means high frame-rate gaming.
You’ll also notice in some of the drawings connections to the cloud and between the various Magic Leap devices (glasses, helmet, belt pack). One important component here is the focus on mobility. So there is some talk of ultra-wideband (UWB) connectivity which would allow for high-throughput communication between the glasses, headphones, and the belt pack, for instance, without getting tangled in wires.
So, imagine the Sony Walkman again, only without the wires connecting the headphones. You can move your head around violently (festivals, anyone?) without the fear that you’ll get tangled up. Apparently the belt pack can also be attached to a backpack, helmet, and other places as long as its within a close distance to you. So, imagine keeping that in a purse while you just wear sunglasses during the day, and can remove the camera from the front.
Essentially, Magic Leap is trying to solve the Google Glass and even the Snap Spectacles problem where other people know you have a weird computer on your face. This means that, out of the box, you could buy their product and no one would know that you’re working with visual overlays on top of your glasses.
The data this thing will be pulling in from the various sensors is insane, but nothing much more than a smartphone is already doing: compass, gyroscope, GPS, accelerometer. The difference here is its all being fed into a rendering engine to first understand your pose, positioning, and eyesight direction, and then to overlay graphics that mirror and match the environment and sight angle.
We already know that Magic Leap’s visual system is projecting an image onto your retina. It’s not something inside the display but rather inside your eyeball, so it looks and feels as if it’s truly real.
Guess what else it has? Infrared cameras. Know what that’s for? You guessed it, we already wrote about it. That, plus depth cameras will help give a bit more context to creatures that are living and objects that are inanimate.
In terms of privacy, which is the title of the filing, they speak generally about an architecture where the only data from sensors passed to your mobile device for processing is geometry for pose and positioning and not the photos or videos of something while you’re in the bathroom. Yes, they actually say this in a patent filing. This “gating mechanism” in addition to the conspicuous, detachable camera allows you to selectively decide how much information to record and store.
IV. Anthropic Implications
There’s this wonderful concept that has roots where philosophy crosses physics called The Anthropic Principle. It’s a bit of a complex topic but it’s a bit like “if a tree falls in a forest and nobody is around to hear it, does it exist?”.
It’s a similar thought with our universe. Humanity, and life, evolved out of a universe that was created to enable it. We emerged from the fundamental building blocks. And because of that, we are able to consciously observe the universe and contemplate it.
But if intelligent life didn’t exist in the universe to contemplate it, then would the universe exist?
If a universe was created and intelligence was not around to consciously experience it, then does it exist?
— Sean
