Meta CEO Teases Project Cambria’s Color Passthrough in New Video

Meta CEO Mark Zuckerberg talked a bit about the company’s upcoming high-end VR headset Project Cambria today and what it can do. The headset is still basically under wraps, however Zuckerberg showed off a bit of the experience of using the device, giving us a good look at Cambria’s color passthrough function, which allows it do some pretty convincing augmented reality tasks.

Zuckerberg released a video on his Facebook profile today where he sat down with Jesse Schell, founder and CEO of pioneering VR studio Schell Games.

Schell Games is currently developing what Zuckerberg described as a “hypothetical follow-up to I Expect You to Die” using Project Cambria, which is due to launch sometime this year. Instead of being sealed off from the physical world, like a typical VR headset might, Cambria’s color passthrough allows for augmented reality interactions using your own living room as the backdrop.

In a separate video (seen below), the Meta CEO tries on Project Cambria, tightens a back-mounted ratcheting knob, and plays an experience called The World Beyond, which focuses on mixing both virtual and physical spaces and using hand tracking for input. The full-color version will only be available on Cambria, Zuckerberg says, however Quest 2 users will be able to try it soon on App Lab, the experimental app library.

Schell Games is developing its I Expect You to Die-style AR experience on Meta’s Presence Platform, a suite of SDKs that, starting today, will allow any interested developer to build more advanced mixed reality applications.

Speaking to Schell, Zuckerberg talked a bit about the headset’s various sensors. It includes “a bunch of new sensors,” Zuckerberg says, including high resolution, outward-facing cameras that allow for color passthrough experiences as well as depth sensors which will no doubt help with room-tracking fidelity and establishing spatial anchors.

Schell also spoke a bit about the studio’s experience working with Cambria’s color passthrough:

“I’m really excited about the color passthrough, because the black and white passthrough […] gives you sense of where you’re going, but when you have things actually in color, it’s a lot more exciting. It seems much more real, particularly when objects are blending. They key is, when you get the lighting right on the virtual objects so that they match up with the ones that are in your scene, you have these [magic moments] when you’re not sure what’s real and what’s real and what’s not, and you kind of have to take the headset off and check for a second for what is there and what isn’t there.”

In the chat with Schell, Zuckerberg describes Project Cambria as having a “somewhat tighter formfactor than Quest 2.” Although gaming and fitness is likely to be an early focus, Zuckerberg says the headset is also targeting work, which he says will be better for productivity, co-creating things with other headset users, and having meetings virtually.

There’s still no precise launch information yet. As confirmed by Meta, Project Cambria is said to be priced “significantly higher” than $800, which likely positions it more for developers and prosumers. Recent reports however contend that Meta will be releasing Cambria in September, and three more VR headsets by 2024.

You can catch the full 27-minute chat between Zuckerberg and Schell below:

The post Meta CEO Teases Project Cambria’s Color Passthrough in New Video appeared first on Road to VR.

Reality Labs Chief Scientist Outlines a New Compute Architecture for True AR Glasses

Speaking at the IEDM conference late last year, Meta Reality Labs’ Chief Scientist Michael Abrash laid out the company’s analysis of how contemporary compute architectures will need to evolve to make possible the AR glasses of our sci-fi conceptualizations.

While there’s some AR ‘glasses’ on the market today, none of them are truly the size of a normal pair of glasses (even a bulky pair). The best AR headsets available today—the likes of HoloLens 2 and Magic Leap 2—are still closer to goggles than glasses and are too heavy to be worn all day (not to mention the looks you’d get from the crowd).

If we’re going to build AR glasses that are truly glasses-sized, with all-day battery life and the features needed for compelling AR experiences, it’s going to take require a “range of radical improvements—and in some cases paradigm shifts—in both hardware […] and software,” says Michael Abrash, Chief Scientist at Reality Labs, Meta’s XR organization.

That is to say: Meta doesn’t believe that its current technology—or anyone’s for that matter—is capable of delivering those sci-fi glasses that every AR concept video envisions.

But, the company thinks it knows where things need to head in order for that to happen.

Abrash, speaking at the IEDM 2021 conference late last year, laid out the case for a new compute architecture that could meet the needs of truly glasses-sized AR devices.

Follow the Power

The core reason to rethink how computing should be handled on these devices comes from a need to drastically reduce power consumption to meet battery life and heat requirements.

“How can we improve the power efficiency [of mobile computing devices] radically by a factor of 100 or even 1,000?” he asks. “That will require a deep system-level rethinking of the full stack, with end-to-end co-design of hardware and software. And the place to start that rethinking is by looking at where power is going today.”

To that end, Abrash laid out a graph comparing the power consumption of low-level computing operations.

Image courtesy Meta

As the chart highlights, the most energy intensive computing operations are in data transfer. And that doesn’t mean just wireless data transfer, but even transferring data from one chip inside the device to another. What’s more, the chart uses a logarithmic scale; according to the chart, transferring data to RAM uses 12,000 times the power of the base unit (which in this case is adding two numbers together).

Bringing it all together, the circular graphs on the right show that techniques essential to AR—SLAM and hand-tracking—use most of their power simply moving data to and from RAM.

“Clearly, for low power applications [such as in lightweight AR glasses], it is critical to reduce the amount of data transfer as much as possible,” says Abrash.

To make that happen, he says a new compute architecture will be required which—rather than shuffling large quantities of data between centralized computing hubs—more broadly distributes the computing operations across the system in order to minimize wasteful data transfer.

Compute Where You Least Expect It

A starting point for a distributed computing architecture, Abrash says, could begin with the many cameras that AR glasses need for sensing the world around the user. This would involve doing some preliminary computation on the camera sensor itself before sending only the most vital data across power hungry data transfer lanes.

Image courtesy Meta

To make that possible Abrash says it’ll take co-designed hardware and software, such that the hardware is designed with a specific algorithm in mind that is essentially hardwired into the camera sensor itself—allowing some operations to be taken care of before any data even leaves the sensor.

Image courtesy Meta

“The combination of requirements for lowest power, best requirements, and smallest possible form-factor, make XR sensors the new frontier in the image sensor industry,” Abrash says.

Continue on Page 2: Domain Specific Sensors »

The post Reality Labs Chief Scientist Outlines a New Compute Architecture for True AR Glasses appeared first on Road to VR.

Canon Announces MREAL X1 Enterprise Headset with Larger FOV

Canon announced its next enterprise XR headset, MREAL X1. The device, which offers passthrough AR, appears to be a modestly upgraded version of the company’s 2021 model, MREAL S1, as it boasts a field of view (FOV) that’s “2.5 times larger.”

The headset, which is detailed in an English language video (seen below), is said to make use of Canon’s proprietary display panels and optical technology, something the company says has allowed it to expand from MREAL S1’s quoted 45° × 34° to MREAL X1’s 58° × 60°.

The display resolution has also increased from 1,600 × 1,200 per eye in MREAL S1 to 1,920 × 2,160 per eye in the new X1 model.

Image courtesy Canon

It appears the company is also offering an optional grip-style handheld display configuration, effectively working like the one for Canon MREAL Display MD-20, which was introduced in 2020. The halo headset strap system, with front-facing adjustment knob, appears to be very similar to last year’s model as well.

“In particular, by expanding the longitudinal viewing angle, which was often requested by users, it is possible to check the entire view area without moving the head greatly,” Canon says in its Japanese language site, underlining the headset’s biggest upgrade.

Although the announcement video is dubbed in English, Canon has not made supporting information available in anything but the Japanese language, which suggests it’s again primarily targeting its domestic market like it has with other MREAL headsets in the past.

Additionally, the headset’s price is marked as “open,” which means you’ll likely need to partner with Canon Japan to get a pricing list. That said, if the steeply priced MD-20 model is any indication, it’s probably in the range of 10s of thousands of U.S. dollars.

Unlike standalone AR headsets such as HoloLens 2 or Magic Leap 2, which feature wave guide-based see-through optics, Canon’s tethered headset is a passthrough XR device, which literally passes real-world imagery through a pair of front-facing camera sensors and blends it with virtual images on a completely opaque conventional display.

If you want to learn more about different AR display methods, check out our primer on the differences between smartglasses and AR glasses.

Specs

MREAL X1
Connection
MREAL Display Body dedicated terminals for interface cables (×1)
interface board Dedicated Interface Cable Terminal (×1)
Mini DisplayPort Terminal (×1)
PCI Express 2.0 (Gen3) ×4 Card Edge (×1)
interface box Interface Cable Dedicated Terminal (×1)
AC Adapter Terminal (×1)
PC Connection Cable Terminal (×1)
Size
MREAL Display body
(head mount unit, interface cable not included)
Approx. 158 mm (W) ×86 mm (H) ×38 mm (D)
MREAL Display body + head mount unit
(minimum state, interface cable not included)
Approx. 186 mm (W) × 150 mm (H) × 250 mm (D)
mass
MREAL Display body
(head mount unit, interface cable not included)
approximately 158g
MREAL Display body + head mount unit
(minimum state, interface cable not included)
approximately 359g
Cable Length
interface cable approx. 10m (20m optional accessory)
display system
display angle of view approx. 58° (horizontal) ×60° (vertical)
display resolution*4 approx. 3840 × 2160
(1920×2160 images on each left and right display)
display frequency Approximately 120Hz*5
IPD Support Range
IPD range approx. 55 to 78 mm
(adjustment range) approx. 57 mm to 76 mm

The post Canon Announces MREAL X1 Enterprise Headset with Larger FOV appeared first on Road to VR.

Epic Games Offers 3D Scanning On Smartphones Via App In Limited Beta

Epic Games unveiled its new 3D scanning app for smartphones called RealityScan.

The app uses smartphone cameras and photos to create high-fidelity 3D photogrammetric models of real-world objects for use on digital platforms. You can take a closer look at how it works in Epic’s new promotional video, embedded below.

In the video, the user takes a number of photos of an object — in this instance, the armchair — which then allows the app to create a 3D model that can be used in digital experiences and scaled and positioned as required.

Epic says that the app “walks users through the scanning experience with interactive feedback, AR guidance, and data quality-checks” and can then create a model “almost instantly.” The resulting models can be uploaded to Sketchfab (which Epic acquired mid-last year) and used across many platforms, including VR and AR.

The app was developed by Epic in collaboration with CapturingReality (acquired by Epic last year) and Quixel. It is now in limited beta on iOS — the first 10,000 users will be granted access on a first-come, first-served basis with wider access rolling out later in Spring.

This isn’t the first app to offer a form of 3D scanning on smartphone devices, but it is perhaps the most high-profile crack at the concept yet. 3D object capture will likely play a big role in VR and AR’s future. Headsets like the LiDAR-equipped Varjo XR-3 allows users to scan their environment and present it to others in real-time while  games like Puzzling Places showcase the creative potential of photogrammetric data as well, offering puzzles composed of real-world objects and places, scanned into the game as 3D models.

You can join the limited beta for RealityScan on iOS now, while spots last, via TestFlight. Android support will arrive later this year. You can read more about RealityScan here.

GDC Day 4: ARVORE, Hyper Dash, Emerge Wave 1 Haptics & More

The fourth and final day of GDC 2022 has come and gone. Don’t be too sad though — we’ve got lots of interesting interviews with VR developers straight from the show floor to cheer you up.

It was a great week at GDC last week, with lots of interesting news over the course of the four days at the show. Alex and Skeeva from Between Realities were checking it all out for us as UploadVR Correspondents, pulling some fantastic developers aside for interviews each day.

On day one, they spoke to Walkabout Mini Golf developers Mighty Coconut, Zenith developer Ramen VR and more.

Day two saw them speak to Polyarc about Moss: Book 2, along with Fast Travel Games on Cities VR and Virtuoso. Day three brought some hardware into the mix, including demos and talks with the developers of the upcoming Lynx R1 mixed reality headset. They also caught up Tilt Five and Owlchemy Labs, developers of Cosmonious High (releasing later this week).

For the fourth and final day, Alex and Skeeva first checked in with ARVORE, developer of last year’s Yuki and the Pixel Ripped series. When questioned about any new Pixel Ripped content or releases in the near future, Rodrigo Terra from ARVORE was tight lipped but did mention an upcoming collaboration with Holoride (who make VR experiences designed to take place inside moving cars) that might satisfy fans of the series.

Rodrigo also said that the studio is working on a few new projects, which could release this year or next, so keep an eye out.

Alex and Skeeva also spoke to the developers of Hyper Dash, who revealed a new free game mode will release for the title on April 1, called ‘Ball’. Triangle Factory CEO and Co-Found Timothy Vanherbergen insisted it wasn’t a joke, despite the release date, and described the mode as “Rocket League but with guns.”

Last but not least, there were some interesting discussions with the developers of the Emerge Wave 1 haptic device, which uses sound and vibrations to provide a new kind of haptic feedback, and the developer of Finger Guns, an FPS shooter using hand tracking technology coming to Quest this year.

What was your favorite news or reveal from this year’s GDC? Let us know in the comments below.

GDC Day 3: Cosmonious High, Lynx Mixed Reality Headset & More

Another day, another round of GDC 2022 coverage. Today is day three and the Between Realities crew hit the show floor again to bring you more interviews with VR/AR developers.

If you missed the previous two days, it’s been pretty jam packed already. Day one saw Alex and Skeeva talk to the developers of Walkabout Mini Golf, Zenith VR and more, and day two brought us interviews with Polyarc (Moss Book 2) Fast Travel Games (Cities VR and Virtuoso) and others.

Alex and Skeeva kept up the incredible pace today, speaking first to Owlchemy Labs (Job Simulator, Vacation Simulator) about their new game Cosmonious High, which releases next week.

They also caught up with the teams behind Patchworld: Sound of the Metaverse, Altair Breaker and Snapdragon Spaces.

Last, but definitely not least, Alex and Skeeva gave the upcoming Lynx R1 mixed reality headset a try and spoke to Stan Larroque from Lynx about the hardware.

When asked how far along everything was, Larroque said that things were “pretty mature” on the software side and they were “in the process of manufacturing” the hardware at the moment. The headsets were meant to ship next month in April, but Lynx has been affected by the ongoing global supply chain issues, which will mean a short delay.

“We were supposed to deliver in April but we’re going to face some issues with the supply chain,” said Larroque. “I think you can expect the first headsets to come between June and July. It’s a matter of weeks, we have some weeks of delays here.”

Keep an eye out for our GDC wrap-up show tomorrow, where Skeeva and Alex from Between Realities will join Ian live in the UploadVR virtual studio to discuss their hands-on experiences over the last few days.

You can catch that live on our YouTube channel tomorrow at 4pm Pacific.

Watch: New Look At Magic Leap 2 Headset & Controllers

A video shared by Magic Leap earlier this month gives us our most comprehensive look at the design of the company’s upcoming Magic Leap 2 AR headset yet.

It shows us almost every angle imaginable of the headset and its controllers.

As reported in late January, the Magic Leap 2 specs suggest it will be a best-in-class AR headset, aimed at the enterprise market. Compared to the Magic Leap 1, it’s lighter in weight, twice as powerful and features an eye box that is twice as large. This is just the tip of the iceberg — you can read more spec specifics here.

We had previously seen photos of Magic Leap 2, but this new video gives a full 360 degree overview. Plus, it gives a clearer look at the headset’s accompanying controllers. As reported earlier this month, the controllers feature cameras on the sides, used for onboard inside-out tracking.

We had seen some unofficial pictures of the controllers at the time, but this new video gives us our first official look. The two cameras are present on the sides, but you can also see what looks to be a trackpad on the top of the controller.

This style of inside-out tracking, using cameras on the controllers themselves, is being employed by other companies as well — leaked images from last September suggest that Meta will use a similar onboard camera design with its controllers for Project Cambria.

Magic Leap 2 will target enterprise markets on release, but specific pricing info and release window details have yet to be revealed.

Final Fantasy Publisher Square Enix Working On AR Gaming For Qualcomm’s Dev Kit

Final Fantasy publisher Square Enix is working with Qualcomm on its new AR developer kit, Snapdragon Spaces.

The pair announced the collaboration today as Qualcomm also revealed a $100 million ‘Snapdragon Metaverse Fund‘. It will see Square Enix explore the possibilities for “immersive gaming experiences” on the platform.

Specifics for the partnership aren’t known but, in a prepared statement, Ben Taylor, Technical Director at Square Enix noted that the company thinks “the time is right with XR to innovate on games of a classic genre we are especially known for, and we look forward to sharing them with the world”.

First announced back in November 2021, Snapdragon Spaces consists of a pair of head-worn AR glasses that are tethered to an external power source such as a smartphone. Similar to other glasses — many of which are powered by Qualcomm’s own XR2 platform — the device is capable of anchoring virtual images in the real world when viewed through the lenses. It also features positional tracking and hand tracking for input.

Square Enix, meanwhile, is known for some of the biggest brands in gaming, including several Japanese role-playing series such as the Final Fantasy games. Taylor’s note on a “classic genre” suggests that the company could be looking to bring this style of experience to the platform.

Qualcomm previously noted that Snapdragon Spaces would see general availability sometime in spring 2022, but it’s currently unclear what this rollout will look like.

What would you want to see Square Enix bring to AR? Let us know in the comments below!

Report: Microsoft Braces for Negative Field Tests of Military HoloLens

Microsoft is supposedly gearing up to field test its HoloLens-based military AR headset, however a new report contends the company is bracing for impact, as it’s expecting negative feedback from soldiers.

Last year, Microsoft announced it had won a United States Army defense contract worth up to $22 billion which would see the development of a so-called Integrated Visual Augmentation System (IVAS), a tactical AR headset for soldiers based on HoloLens 2 technology.

A Business Insider report, citing a leaked internal email, maintains that Microsoft has low expectations for its latest version of IVAS, which is set to begin real-world operational tests with the US Army in May.

Prototype testing (2019), Image courtesy CNBC

Microsoft’s IVAS contract has allegedly seen delays and quality problems. A separate Business Insider report from last month alleges its enterprise-focused HoloLens 3 may also be at risk due to internal issues within Microsoft’s mixed reality division surrounding whether HoloLens should serve consumers or continue courting enterprise companies.

A purported Microsoft Teams message from Mixed Reality division head and HoloLens co-inventor Alex Kipman paints a pretty depressing story:

“So depressed, so demoralized, so broken. I’m sure by now you’ve read or heard about one or two of the Business Insider articles that were published on us. On our private roadmap. On our customers’ confidential data … as a consequence of these articles and these individuals shameful actions, someone from finance already came to me to ask if we should lock down and not share so openly our numbers. Someone from marketing already came to me and asked if we should lock down and not share so openly our roadmap. Someone for from our National Intelligence and Security Team already came to me to ask if we should lock down our IVAS work.”

Kipman rebuffed the previously report of unrest, saying “don’t believe what you read on the internet.”

It’s said that soldiers may take issue with the device’s low light and thermal imaging performance, and that user impressions will “continue to be negative as reliability improvements have been minimal from previous events.”

That $22 billion is an upper target and not the full amount granted to Microsoft at present. And it seems confidence in the project isn’t very high at the moment, as US Congress has allegedly frozen $394 million from the Army’s IVAS budget, which Business Insider notes leaves only $405 million—around $200 million shy of what Microsoft supposedly needs to recover development costs.

Additionally, it’s also said some close to the project fear the Army will simply walk away from the contract.

The post Report: Microsoft Braces for Negative Field Tests of Military HoloLens appeared first on Road to VR.

Apple Leaks ‘realityOS’, Rumored For Upcoming Headset

App Store logs and GitHub code from Apple confirm the existence of realityOS, expected to be used in Apple’s upcoming headset.

The existence of realityOS was first reported by Bloomberg all the way back in 2017. In 2021 BloombergThe Information, and supply chain analyst Ming-Chi Kuo released reports claiming Apple is preparing to release a premium headset for VR and AR with high resolution color passthrough. Recent notes from Kuo claim this headset will weigh significantly less than Meta’s Quest 2, feature dual 4K OLED microdisplays, and use a new chip with “similar computing power as the M1 for Mac”.

The Information Apple VR

Three weeks ago, iOS Developer Rens Verhoeven spotted a new platform “com.apple.platform.realityos” in the App Store app upload logs. Apple’s existing operating systems include iOS (com.apple.platform.iphoneos), iPadOS, watchOS (com.apple.platform.watchos), macOS, and tvOS.

This week, “award-winning git repository surgeon” Nicolás Álvarez spotted Apple committing code to its open source GitHub repository referencing ‘TARGET_FEATURE_REALITYOS’ and ‘realityOS_simulator’ – the latter likely a feature to allow developers without the headset to test building AR or VR applications. Álvarez says Apple quickly force-pushed the repo to try & hide the change, suggesting making this public was a mistake.

This isn’t the first public confirmation of Apple working on AR & VR software. In December a job listing was posted for ‘AR/VR Frameworks Engineer’, with the role described as “developing an entirely new application paradigm” for “software that is deeply integrated into our operating systems”.

If Apple can pull off putting an M1-tier chip in a slim headset, it could deliver a significantly higher fidelity experience than Quest 2, and even Meta’s own upcoming take on a premium headset, Project Cambria. Last month Bloomberg reported the product release may have slipped to next year, and claimed Apple has “weighed prices north of $2000”.