Report: Meta in Talks with Magic Leap for Multiyear AR Headset Tech Deal

A report from the Financial Times maintains Meta is currently in talks with AR headset creator Magic Leap to strike a multiyear deal, which could include intellectual property licensing and contract manufacturing of AR headsets in North America.

The AR unicorn is said to possess valuable IP regarding custom components, including its optics, waveguides, and software.

It’s said a potential deal may also allow Meta to lessen its reliance on China for component manufacturing. In 2019, Magic Leap partnered with manufacturing solutions company Jabil to create a Guadalajara, Mexico plant which the report maintains can assemble headsets in “the tens of thousands a year.”

Magic Leap 2 | Photo by Road to VR

Citing people with knowledge of the talks, the report maintains however a specific joint Meta-Magic Leap headset isn’t expected.

While both companies didn’t comment on a potential partnership, Magic Leap said this to the Financial Times:

“Given the complexities of developing true augmented reality technologies and the intricacies involved with manufacturing these optics, as well as the issues many companies experience with overseas supply chain dependencies, we have entered into several non-exclusive IP licensing and manufacturing partnerships with companies looking to enter the AR market or expand their current position.”

Since it exited stealth in 2014, Magic Leap has released two AR headsets, Magic Leap 1 and Magic Leap 2, which have been compared in functionality to Microsoft’s HoloLens AR headsets.

The company has raised over $4 billion, with minority investors including Google, Alibaba, Qualcomm, AT&T, and Axel Springer. Its majority stakeholder is Saudi Arabia’s state-owned sovereign wealth fund.

In addition to Quest Pro mixed reality headset, Meta has confirmed it’s currently working on its next iteration of Quest, likely Quest 3, as well as its own AR glasses. Meta started real-world testing of Project Aria in 2020, a platform for training its AR perception systems and asses public perception of the technology.

Whether Hit or Flop, Apple’s Entrance Will Be a Pivotal Moment for XR

If the avalanche of recent reports can indicate anything at all, it seems Apple is entering the VR/AR headset market fairly soon, bringing along with it the most inflated expectations the industry has ever seen. It’s probably going to be expensive, but whether it flops or becomes a big hit, the mere existence of Apple in the space is set to change a lot about how things are done.

The iPhone wasn’t the first smartphone. That award goes to an obscure PDA device called the IBM Simon, released in limited numbers in 1994. The Apple Watch wasn’t the first smartwatch either. That was debatably the Seiko Raputer, which was released in 1998 in Japan. Its monochrome LCD wasn’t capable of touch, instead offering up a tiny eight-direction joystick and six function buttons to browse files, play games, and set calendar appointments. Similarly, iPad wasn’t the first tablet. Mac wasn’t the first home computer. iPod wasn’t the first MP3 player. But all of these products have become nothing short of iconic. There’s very little benefit to being first, at least as far as Apple is concerned.

And while it seems the company’s first mixed reality headset could finally debut at its Worldwide Developers Conference (WWDC) in June, like all of its other products, it won’t be the first MR headset. Just the same, like everything else the fruit company makes, it’s going to be the one everyone is talking about—for better or worse.

In case you haven’t noticed, Apple is a big deal. It has an ecosystem of products which connect to each other, design-forward hardware that has helped it maintain brand name cache, and a philosophy that puts user-friendliness at the core of its software experience. Oh, and it’s the most valuable company in the world.

And while the irrational exuberance for successive device generations has mostly petered out since its heydays in the early 2000s, reducing its famed long-line launch extravaganzas to more chill online pre-order releases, becoming an Apple apostate is still unthinkable to many. Once you’re in, you’re in. You buy the phone, the laptop, the headphones, and now, maybe you’ll get the newfangled headset too. Maybe. Let’s put aside the rumors for now. Forget about the spec breakdowns, hardware design leaks, software capabilities, etc. There are plenty of them out there, and you can read about those here. The only thing we know for sure is Apple is… well… Apple. Here’s what you, and probably everyone else is expecting.

Apple’s BKC Store in Mumbai, India | Image courtesy Apple

For Better: What Should Happen

Unless the company is making a drastic departure here, its first mixed reality headset should be built with this same level of user friendliness as all of its other devices, which means it should connect to the Apple ecosystem easily, and have a simple and intuitive UI. Log in with Apple ID. No muss, no fuss (whatever ‘muss’ is). Privacy should be a giant focus for the headset from the outset, since it will almost certainly pack eye-tracking in addition to a host of cameras to get a glimpse of the inside of your immediate surroundings, messiness and all. Apple has its fair share of data collection scandals, yet it seems to inspire enough confidence for privacy to be a big historical selling point for all of its devices.

If you want to avoid drawing the ire of tech reviewers everywhere though, wearing it should be fairly simple and very comfortable, and the experiences within should be of high enough value to overcome that inherent friction of charging it, putting it on, setting up a tracking volume, and wearing it for extended periods of time—everything we expect from any mixed reality headset at this point. It should fit most people, and offer up a clear picture to people with heads and eyes of all shapes and sizes.

Meta Quest Pro | Image courtesy Meta

An obvious analogue here is Meta Quest Pro, which is relatively low friction, but things like a halo strap that forces too much weight on your brow, or a passthrough that’s just a little too grainy, or a display that doesn’t have a high enough pixel per degree (ppd) for staring at text—all of these things make it less appealing to users in the day-to-day, introducing what you might call accumulative friction. You use it a bunch at first until you figure out all of the niggles, at which point you may revert to traditional computing standards like using a laptop or smartphone. The thing isn’t really the all-purpose device you hoped it would be, and the company thinks twice about when to send the better, more improved version down the pipeline.

One would hope that Apple’s headset, on the other hand, should have a mature design language and have obviously useful features from day one. While there’s bound to be some stutters, like with the first Apple Watch, which was critiqued for its slow software, short battery life, and lack of customization, it should all be there, and not require a ton of feature updates to enhance after the big launch day.

It should sell well out of the gate—at least by the standards of the existing XR industry—even if everything isn’t perfect. And it should be so cool that it’s copied. Like a lot. And it should drag top-level studios into the XR scene to start making innovative and useful apps that aren’t just straight ports of ARkit or ARcore apps made for mobile, but things people need and want to use in-headset. A big win from Apple should not only spur its new mixed reality product category, but kick off a buzz among developers, which would include those who currently work in the XR industry and Apple’s existing cohort of dedicated iOS developers.

But more than merely being the latest shiny new headset within the existing XR industry, Apple’s entrance into the field has a real chance of radically expanding the industry itself, by showing that the world’s most iconic tech company now thinks the medium is worth pursuing. That’s the way it happened when Apple jumped into MP3 players, smartphones, tablets, wireless earbuds, and more.

As the saying goes, a rising tide lifts all boats. The inverse is also true though….

For Worse: What Could Happen

Apple’s headset is reportedly (okay, maybe just one rumor) priced somewhere near $3,000, so it probably won’t be the sort of accessory that initially attracts people to the ecosystem; that would be the job of a peripheral like Apple Watch. It will likely rely on the pool of built-in Apple users. Despite the price, the first iteration very likely won’t offer the sort of power you’d expect from a workhorse like Apple MacBook Pro either.

At the outset, any sustained draw from prosumers will invariably hinge on how well it can manage general computing tasks, like you might have with an iPad or MacBook, and everything else current mixed reality headset should do too, namely VR and AR stuff. That includes a large swath of things like fitness apps, both AR and VR games and experiences, productivity apps, standard work apps, everything. Basically, it has to be the Quest Pro that Meta wanted to release but didn’t.

AR turn-by-turn directions on an iPhone | Image courtesy Apple

And if not, it leaves Apple in a pretty precarious situation. If their headset can’t find a proper foothold within its ecosystem and attract enough users, it could lead to low adoption rates and a lack of interest in the technology as a whole. Mixed reality is largely seen as valuable steppingstone to what many consider the true moneymaker: all-day AR glasses. And despite some very glassses-shaped AR headsets out there, we’re still not there yet. Even if Apple is willing to take a hit with a bulky device in service of pushing use cases for its AR glasses yet to come, the short term may not look very bright.

And perhaps most importantly for the industry as a whole are the (metaphorical) optics.

After all, if the iconic Apple can’t manage to make MR something that everybody wants, the rest of the world watching from the sidelines may think the concept just can’t be conquered. In turn, it may mean capital investment in the space will dry up until ‘real’ AR headsets are a thing—the all-day glasses that will let you play Pokémon Go in the park, do turn-by-turn directions, and remind you the name of that person you met last week. The steppingstone of mixed reality may get waterlogged. Those are a lot of ifs, coulds, shoulds, and won’ts though. The only thing truly certain is we’re in for a very interesting few months, which you can of course follow at Road to VR.

Apple’s entrance into XR has the potential to expand the industry by demonstrating its viability, just as Apple has done with previous technologies. It stands a good chance at carving out a sizeable claim in the space, but it’s a gamble that could equally backfire if both sales and public perception aren’t on their side.


Is Apple’s XR headset going to be the “one more thing?” we’ve all been waiting for at WWDC this year? Will it live up to the Apple name, or be an expensive dev kit? Let us know in the comments below!

Xiaomi Unveils Wireless AR Glasses Prototype, Powered by Same Chipset as Meta Quest Pro

Chinese tech giant Xiaomi today showed off a prototype AR headset at Mobile World Congress (MWC) that wirelessly connects to the user’s smartphone, making for what the company calls its “first wireless AR glasses to utilize distributed computing.”

Called Xiaomi Wireless AR Glass Discovery Edition, the device is built upon the same Qualcomm Snapdragon XR2 Gen 1 chipset as Meta’s recently released Quest Pro VR standalone.

While specs are still thin on the ground, the company did offer some info on headline features. For now, Xiaomi is couching it as a “concept technology achievement,” so it may be a while until we see a full spec sheet.

Packing two microOLED displays, the company is boasting “retina-level” resolution, saying its AR glasses pack in 58 pixels per degree (PPD). For reference, Meta Quest Pro has a PPD of 22, while enterprise headset Varjo XR-3 cites a PPD of 70.

The company hasn’t announced the headset’s field of view (FOV), however it says its free-form light-guiding prisms “minimizes light loss and produces clear and bright images with a to-eye brightness of up to 1200nit.”

Electrochromic lenses are also said to adapt the final image to different lighting conditions, even including a full ‘blackout mode’ that ostensibly allows it to work as a VR headset as well.

Image courtesy Xiaomi

As for input, Xiaomi Wireless AR Glass includes onboard hand-tracking in addition to smartphone-based touch controls. Xiaomi says its optical hand-tracking is designed to let users to do things like select and open apps, swipe through pages, and exit apps.

As a prototype, there’s no pricing or availability on the table, however Xiaomi says the lightweight glasses (at 126g) will be available in a titanium-colored design with support for three sizes of nosepieces. An attachable glasses clip will also be available for near-sighted users.

In an exclusive hands-on, XDA Developers surmised it felt near production-ready, however one of the issues noted during a seemingly bump-free demo was battery life; the headset had to be charged in the middle of the 30-minute demo. Xiaomi apparently is incorporating a self-developed silicon-oxygen anode battery that is supposedly smaller than a typical lithium-ion battery. While there’s an onboard Snapdragon XR 2 Gen 1 chipset, XDA Developers also notes it doesn’t offer any storage, making a compatible smartphone requisite to playing AR content.

This isn’t the company’s first stab at XR tech; last summer Xiaomi showed off a pair of consumer smartglasses, called Mijia Glasses Camera, that featured a single heads-up display. Xiaomi’s Wireless AR Glass is however much closer in function to the concept it teased in late 2021, albeit with chunkier free-form light-guiding prisms than the more advanced-looking waveguides teased two years ago.

Xiaomi is actively working closely with chipmaker Qualcomm to ensure compatibility with Snapdragon Spaces-ready smartphones, which include Xiaomi 13 and OnePlus 11 5G. Possible other future contributions from Lenovo and Motorola, which have also announced their intentions to support Snapdragon Spaces.

Qualcomm announced Snapdragon Spaces in late 2021, a software tool kit which focuses on performance and low power devices which allows developers to create head-worn AR experiences from the ground-up, or add head-worn AR to existing smartphone apps.

Qualcomm Partners with 7 Major Telecoms to Advance Smartphone-tethered AR Glasses

Qualcomm announced at Mobile World Congress (MWC) today it’s partnering with seven global telecommunication companies in preparation for the next generation of AR glasses which are set to work directly with the user’s smartphone.

Partners include CMCC, Deutsche Telekom, KDDI Corporation, NTT QONOQ, T-Mobile, Telefonica, and Vodafone, which are said to currently be working with Qualcomm on new XR devices, experiences, and developer initiatives, including Qualcomm’s Snapdragon Spaces XR developer platform.

Qualcomm announced Snapdragon Spaces in late 2021, a software tool kit which focuses on performance and low power devices which allows developers to create head-worn AR experiences from the ground-up or by adding head-worn AR to existing smartphone apps.

Qualcomm and Japan’s KDDI Corporation also announced a multi-year collaboration which it says will focus on the expansion of XR use cases and creation of a developer program in Japan.

Meanwhile, Qualcomm says OEMs are designing “a new wave of devices for operators and beyond” such as the newly unveiled Xiaomi Wireless AR Glass Discovery Edition, OPPO’s new Mixed Reality device and OnePlus 11 5G smartphone.

At least in Xiaomi’s case, its Wireless AR Glass headset streams data from compatible smartphones. Effectively offloading computation to the smartphone, the company’s 126g headset boasts a wireless latency of as low as 3ms between the smartphone device to the glasses, and a wireless connection with full link latency as low as 50ms which is comparable to wired solution.

Niantic Launches Visual Positioning System For ‘Global Scale’ AR Experiences

Niantic‘s new Lightship Visual Positioning System (VPS) will facilitate interactions with ‘global scale’ persistent and synced AR content on mobile devices.

Niantic launched Lightship during its developer conference this week and you can see some footage in the video embedded above showing some phone-based AR apps using its new features starting from the 50:20 mark. The system is essentially a new type of map that developers can use for AR experiences, with the aim of providing location-based persistent content that’s synced up for all users.

Niantic is building the map from scanned visual data, which Niantic says will offer “centimeter-level” accuracy when pinpointing the location and orientation of users (or multiple users, in relation to each other) at a given location. The technology is similar to large-scale visual positioning systems in active development at Google and Snap.

While the promise of the system is to work globally, it’s not quite there just yet — as of launch yesterday, Niantic’s VPS system has around 30,000 public locations where VPS is available for developers to hook into. These locations are mainly spread across six key cities — San Francisco, London, Tokyo, Los Angeles, New York City and Seattle — and include “parks, paths, landmarks, local businesses and more.”

To expand the map, Niantic developed the Wayfarer app which allows developers to scan in new locations using their phones, available now in public beta. Niantic has also launched a surveyor program in the aforementioned six key launch cities to expedite the process.

“With only a single image frame from the end user’s camera, Lightship VPS swiftly and accurately determines a user’s precise, six-dimensional location,” according to a Niantic blog post.

Scaling VPS to a global level is a lofty goal for Niantic, but could improve mobile AR experiences which could seem to unlock far more interesting content with accurate maps pinning content to real world locations.

You can read more about Lightship VPS over on the Niantic blog.

Magic Leap is Selling Its First AR Headset for Just $550

It looks like Magic Leap is holding a barn burner of a sale on its first AR headset, Magic Leap 1, as the one-time $2,300 device can now be had for $550.

As first reported by GMW3, Magic Leap appears to be flushing excess stock of the 2018-era AR headset via the Amazon-owned online retailer Woot. 

The listing (find it here) is for a brand new Magic Leap 1, including the headset’s hip-worn compute unit and single controller. The sale is happening from now until June 1st, and features a three-unit limit per customer. Amazon US Prime members qualify for free shipping, which ought to arrive to those of you in the lower 48 in early June.

If you’re tempted, there’s a few things you should know before hitting the ‘buy now’ button. Users should be warned that since Magic Leap pivoted to service only enterprise users, that its Magic Leap World online app store isn’t likely to see any new apps outside of the handful that were released between 2018-2020.

Image courtesy Magic Leapgic leap

Still, there are a mix of apps such as Spotify or room-scale shooter Dr. Grordbort’s Invaders which might be better suited as tech demos, giving prospective augmented reality devs a sense of what you might create for a bona fide AR headset, ostensibly in preparation for what devices may come—we’re looking at Apple, Google, and Meta in the near future for mixed reality headsets capable of both VR and passthrough AR.

Launched in 2018, Magic Leap straddled an uneasy rift between enterprise and prosumers with ML 1 (known then as ‘ML One’). Reception by consumers for its $2,300 AR headset was lukewarm, and messaging didn’t seem focused enough to give either developers or consumers hope that a more accessible bit of ML hardware was yet to come. Then in mid 2020, company founder and CEO Rony Abovitz stepped down, giving way to former Microsoft exec Peggy Johnson to take the reigns, who has thus far positioned the company to solely target enterprise with its latest Magic Leap 2 headset.

Here’s the full spec sheet below:

CPU & GPU

NVIDIA® Parker SOC
CPU: 2 Denver 2.0 64-bit cores + 4 ARM Cortex A57 64-bit cores (2 A57’s and 1 Denver accessible to applications)
GPU: NVIDIA Pascal™, 256 CUDA cores
Graphic APIs: OpenGL 4.5, Vulkan, OpenGL ES 3.1+AEP

RAM: 8 GB

Storage Capacity: 128 GB (actual available storage capacity 95 GB)

Power
Built-in rechargeable lithium-ion battery. Up to 3.5 hours continuous use. Battery life can vary based on use cases. Power level will be sustained when connected to an AC outlet. 45-watt USB-C Power Delivery (PD) charger

Audio Input
Voice (speech to text) + real world audio (ambient)

Audio Output
Onboard speakers and 3.5mm jack with audio spatialization processing

Connectivity
Bluetooth 4.2, WiFi 802.11ac/b/g/n, USB-C

Haptics: LRA Haptic Device

Tracking: 6DoF (position and orientation)

Touchpad: Touch sensitive

LEDs: 12-LED (RGB) ring with diffuser

Power: Built-in rechargeable lithium-ion battery. Up to 7.5 hours continuous use. 15-watt USB-C charger

Other Inputs
8-bit resolution Trigger Button
Digital Bumper Button
Digital Home Button

The post Magic Leap is Selling Its First AR Headset for Just $550 appeared first on Road to VR.

Qualcomm’s Latest AR Glasses Reference Design Drops the Tether, Keeps the Compute

Qualcomm has revealed its latest AR glasses reference design, which it offers up to other companies as a blueprint for building their own AR devices. The reference design, which gives us a strong hint at the specs and capabilities of upcoming products, continues to lean on a smartphone to do the heavy compute, but this time is based on a wireless design.

Qualcomm’s prior AR glasses reference design was based on the Snapdragon XR1 chip and called for a wired connection between a smartphone and the glasses, allowing the system to split rendering tasks between the two devices.

Now the company’s latest design, based on Snapdragon XR2, takes the wire out of the equation. But instead of going fully standalone, the new reference design continues to rely on the smartphone to handle most of the heavy rendering, but now does so over a wireless connection between the devices.

Image courtesy Qualcomm

In addition to Snapdragon XR2, the AR glasses include Qualcomm’s FastConnect 6900 chip which equips it with Wi-Fi 6E and Bluetooth 5.3. The company says the chip is designed for “ultra-low latency,” and manages less than 3ms of latency between the headset and the smartphone. The company has also announced XR-specific software for controlling its FastConnect 6900, allowing device makers to tune the wireless traffic between the devices to prioritize the most time-critical data in order to reduce instances of lag or jitter due to wireless interference.

Though a connected smartphone seems like the most obvious use-case, Qualcomm also says the glasses could just as well be paired to a Windows PC or “processing puck.”

Beyond the extra wireless tech, the company says the latest design is 40% thinner than its previous reference design. The latest version has a 1,920 × 1,080 (2MP) per-eye resolution at 90Hz. The microdisplays include a ‘no-motion-blur’ feature—which sounds like a low persistence mode designed to prevent blurring of the image during head movement. A pair of monochrome cameras are used for 6DOF tracking and an RGB camera for video or photo capture. The company didn’t mention the device’s field-of-view, so it’s unlikely to be any larger than the prior reference design at 45° diagonal.

Like its many prior reference designs, Qualcomm isn’t actually going to make and sell the AR glasses. Instead, it offers up the design and underlying technology for other companies to use as a blueprint to build their own devices (hopefully using Qualcomm’s chips!). Companies that build on Qualcomm’s blueprint usually introduce their own industrial design and custom software offering; some even customize the hardware itself, like using different displays or optics.

That makes this AR glasses reference design a pretty good snapshot of the current state of AR glasses that can be mass produced, and a glimpse of what some companies will be offering in the near future.

Qualcomm says its latest AR glasses reference design is “available for select partners,” as of today, and plans to make it more widely available “in the coming months.”

The post Qualcomm’s Latest AR Glasses Reference Design Drops the Tether, Keeps the Compute appeared first on Road to VR.

Qualcomm’s Latest AR Glasses Reference Design Drops the Tether, Keeps the Compute

Qualcomm has revealed its latest AR glasses reference design, which it offers up to other companies as a blueprint for building their own AR devices. The reference design, which gives us a strong hint at the specs and capabilities of upcoming products, continues to lean on a smartphone to do the heavy compute, but this time is based on a wireless design.

Qualcomm’s prior AR glasses reference design was based on the Snapdragon XR1 chip and called for a wired connection between a smartphone and the glasses, allowing the system to split rendering tasks between the two devices.

Now the company’s latest design, based on Snapdragon XR2, takes the wire out of the equation. But instead of going fully standalone, the new reference design continues to rely on the smartphone to handle most of the heavy rendering, but now does so over a wireless connection between the devices.

Image courtesy Qualcomm

In addition to Snapdragon XR2, the AR glasses include Qualcomm’s FastConnect 6900 chip which equips it with Wi-Fi 6E and Bluetooth 5.3. The company says the chip is designed for “ultra-low latency,” and manages less than 3ms of latency between the headset and the smartphone. The company has also announced XR-specific software for controlling its FastConnect 6900, allowing device makers to tune the wireless traffic between the devices to prioritize the most time-critical data in order to reduce instances of lag or jitter due to wireless interference.

Though a connected smartphone seems like the most obvious use-case, Qualcomm also says the glasses could just as well be paired to a Windows PC or “processing puck.”

Beyond the extra wireless tech, the company says the latest design is 40% thinner than its previous reference design. The latest version has a 1,920 × 1,080 (2MP) per-eye resolution at 90Hz. The microdisplays include a ‘no-motion-blur’ feature—which sounds like a low persistence mode designed to prevent blurring of the image during head movement. A pair of monochrome cameras are used for 6DOF tracking and an RGB camera for video or photo capture. The company didn’t mention the device’s field-of-view, so it’s unlikely to be any larger than the prior reference design at 45° diagonal.

Like its many prior reference designs, Qualcomm isn’t actually going to make and sell the AR glasses. Instead, it offers up the design and underlying technology for other companies to use as a blueprint to build their own devices (hopefully using Qualcomm’s chips!). Companies that build on Qualcomm’s blueprint usually introduce their own industrial design and custom software offering; some even customize the hardware itself, like using different displays or optics.

That makes this AR glasses reference design a pretty good snapshot of the current state of AR glasses that can be mass produced, and a glimpse of what some companies will be offering in the near future.

Qualcomm says its latest AR glasses reference design is “available for select partners,” as of today, and plans to make it more widely available “in the coming months.”

The post Qualcomm’s Latest AR Glasses Reference Design Drops the Tether, Keeps the Compute appeared first on Road to VR.

Meta CEO Teases Project Cambria’s Color Passthrough in New Video

Meta CEO Mark Zuckerberg talked a bit about the company’s upcoming high-end VR headset Project Cambria today and what it can do. The headset is still basically under wraps, however Zuckerberg showed off a bit of the experience of using the device, giving us a good look at Cambria’s color passthrough function, which allows it do some pretty convincing augmented reality tasks.

Zuckerberg released a video on his Facebook profile today where he sat down with Jesse Schell, founder and CEO of pioneering VR studio Schell Games.

Schell Games is currently developing what Zuckerberg described as a “hypothetical follow-up to I Expect You to Die” using Project Cambria, which is due to launch sometime this year. Instead of being sealed off from the physical world, like a typical VR headset might, Cambria’s color passthrough allows for augmented reality interactions using your own living room as the backdrop.

In a separate video (seen below), the Meta CEO tries on Project Cambria, tightens a back-mounted ratcheting knob, and plays an experience called The World Beyond, which focuses on mixing both virtual and physical spaces and using hand tracking for input. The full-color version will only be available on Cambria, Zuckerberg says, however Quest 2 users will be able to try it soon on App Lab, the experimental app library.

Schell Games is developing its I Expect You to Die-style AR experience on Meta’s Presence Platform, a suite of SDKs that, starting today, will allow any interested developer to build more advanced mixed reality applications.

Speaking to Schell, Zuckerberg talked a bit about the headset’s various sensors. It includes “a bunch of new sensors,” Zuckerberg says, including high resolution, outward-facing cameras that allow for color passthrough experiences as well as depth sensors which will no doubt help with room-tracking fidelity and establishing spatial anchors.

Schell also spoke a bit about the studio’s experience working with Cambria’s color passthrough:

“I’m really excited about the color passthrough, because the black and white passthrough […] gives you sense of where you’re going, but when you have things actually in color, it’s a lot more exciting. It seems much more real, particularly when objects are blending. They key is, when you get the lighting right on the virtual objects so that they match up with the ones that are in your scene, you have these [magic moments] when you’re not sure what’s real and what’s real and what’s not, and you kind of have to take the headset off and check for a second for what is there and what isn’t there.”

In the chat with Schell, Zuckerberg describes Project Cambria as having a “somewhat tighter formfactor than Quest 2.” Although gaming and fitness is likely to be an early focus, Zuckerberg says the headset is also targeting work, which he says will be better for productivity, co-creating things with other headset users, and having meetings virtually.

There’s still no precise launch information yet. As confirmed by Meta, Project Cambria is said to be priced “significantly higher” than $800, which likely positions it more for developers and prosumers. Recent reports however contend that Meta will be releasing Cambria in September, and three more VR headsets by 2024.

You can catch the full 27-minute chat between Zuckerberg and Schell below:

The post Meta CEO Teases Project Cambria’s Color Passthrough in New Video appeared first on Road to VR.

Reality Labs Chief Scientist Outlines a New Compute Architecture for True AR Glasses

Speaking at the IEDM conference late last year, Meta Reality Labs’ Chief Scientist Michael Abrash laid out the company’s analysis of how contemporary compute architectures will need to evolve to make possible the AR glasses of our sci-fi conceptualizations.

While there’s some AR ‘glasses’ on the market today, none of them are truly the size of a normal pair of glasses (even a bulky pair). The best AR headsets available today—the likes of HoloLens 2 and Magic Leap 2—are still closer to goggles than glasses and are too heavy to be worn all day (not to mention the looks you’d get from the crowd).

If we’re going to build AR glasses that are truly glasses-sized, with all-day battery life and the features needed for compelling AR experiences, it’s going to take require a “range of radical improvements—and in some cases paradigm shifts—in both hardware […] and software,” says Michael Abrash, Chief Scientist at Reality Labs, Meta’s XR organization.

That is to say: Meta doesn’t believe that its current technology—or anyone’s for that matter—is capable of delivering those sci-fi glasses that every AR concept video envisions.

But, the company thinks it knows where things need to head in order for that to happen.

Abrash, speaking at the IEDM 2021 conference late last year, laid out the case for a new compute architecture that could meet the needs of truly glasses-sized AR devices.

Follow the Power

The core reason to rethink how computing should be handled on these devices comes from a need to drastically reduce power consumption to meet battery life and heat requirements.

“How can we improve the power efficiency [of mobile computing devices] radically by a factor of 100 or even 1,000?” he asks. “That will require a deep system-level rethinking of the full stack, with end-to-end co-design of hardware and software. And the place to start that rethinking is by looking at where power is going today.”

To that end, Abrash laid out a graph comparing the power consumption of low-level computing operations.

Image courtesy Meta

As the chart highlights, the most energy intensive computing operations are in data transfer. And that doesn’t mean just wireless data transfer, but even transferring data from one chip inside the device to another. What’s more, the chart uses a logarithmic scale; according to the chart, transferring data to RAM uses 12,000 times the power of the base unit (which in this case is adding two numbers together).

Bringing it all together, the circular graphs on the right show that techniques essential to AR—SLAM and hand-tracking—use most of their power simply moving data to and from RAM.

“Clearly, for low power applications [such as in lightweight AR glasses], it is critical to reduce the amount of data transfer as much as possible,” says Abrash.

To make that happen, he says a new compute architecture will be required which—rather than shuffling large quantities of data between centralized computing hubs—more broadly distributes the computing operations across the system in order to minimize wasteful data transfer.

Compute Where You Least Expect It

A starting point for a distributed computing architecture, Abrash says, could begin with the many cameras that AR glasses need for sensing the world around the user. This would involve doing some preliminary computation on the camera sensor itself before sending only the most vital data across power hungry data transfer lanes.

Image courtesy Meta

To make that possible Abrash says it’ll take co-designed hardware and software, such that the hardware is designed with a specific algorithm in mind that is essentially hardwired into the camera sensor itself—allowing some operations to be taken care of before any data even leaves the sensor.

Image courtesy Meta

“The combination of requirements for lowest power, best requirements, and smallest possible form-factor, make XR sensors the new frontier in the image sensor industry,” Abrash says.

Continue on Page 2: Domain Specific Sensors »

The post Reality Labs Chief Scientist Outlines a New Compute Architecture for True AR Glasses appeared first on Road to VR.