A new video from Meta shows new projects from series like I Expect You To Die and developers like Resolution Games.
The video was released as part of a look at Quest 2 and the upcoming Project Cambria headset’s mixed reality capabilities. Mixed reality experiences are those that use a VR headset’s cameras to view the real world, but bring in virtual elements like virtual screens or 3D assets. The effect is similar to that of AR glasses, but AR uses transparent lenses to keep you in direct view of the real world. Though this will one day allow for much more seamless AR, current headsets are limited in field of view and other areas compared to VR.
At one point in the video we see a project called Fish Under Our Feet, developed by Resolution Games. In the clip, the player uses hand tracking to pick up a virtual hammer, smash a hole in their living room floor and then fish from a pond that lies beneath. They can even dive into the pond and have fish swim around them.
Following on from that, there’s footage of a new title from Schell Games called I Expect You To Die: Home Sweet Home. I Expect You To Die is Schell’s long-running VR puzzle series with escape room-style challenges. The footage here showed a user picking up a phone from a virtual telephone box that appears in their room, and opening virtual gates. There’s also a look at VR creation tool, Gravity Sketch, being used in MR.
Neither Schell nor Resolution has formally announced these projects and it’s quite possible that neither will be officially released on the Quest store. Schell Games told us that Home Sweet Home is a one-level demo intended as a showcase, but there’s a possibility that it could see wider release in the future. Meta itself is releasing its own MR experiment, The World Beyond, on App Lab next week.
Meta launched a demo called The World Beyond on App Lab to demonstrate to Quest 2 owners some of the most advanced features of the company’s development platforms for VR and mixed reality.
“This demo was created using Presence Platform, which we built to help developers build mixed reality experiences that blend physical and virtual worlds,” Zuckerberg posted. “It’s even better with full color passthrough and the other advanced technologies we’re adding to Project Cambria.”
The video shows Zuckerberg using a range of hand gestures to interact with a character and throw a virtual ball to see it bounce off the physical walls of his room. The character looks like Oppy, which was shown during Meta’s Connect conference late last year using the black and white passthrough mode on Quest 2. The World Beyond is meant as a demonstration of Meta’s “Presence Platform”, which refers to a collection of tools for developers encompassing voice input, passthrough, and interactions.
Amazon job listings reference a “new-to-world” AR/VR consumer product.
Spotted by Protocol, one listing explains “You will develop an advanced XR research concept into a magical and useful new-to-world consumer product” while another references “developing code for early prototypes through mass production.”
Another job listing describes the role as working on “the core system interface along with end-user applications spanning from multi-modal interfaces to 3D AR entertainment experiences”.
Protocol also spotted that in March Amazon hired Kharis O’Connell to lead a “Futures Design” group, described as “helping Amazon experience what it’s like to live in the future, today”. O’Connell once worked for the now defunct Meta View startup, and then worked on Google’s AR operating system.
Amazon is the only consumer tech giant with no announced or rumored AR or VR headset product. Meta has its Quest VR line and is working on AR glasses too. Microsoft has its HoloLens AR headsets. Multiple reliable sources claim Apple is working on a mixed reality headset, and The Verge reported Google is too.
The company currently sells “smart glasses” called Echo Frames, but these lack any display system or cameras – the use cases are talking to Alexa, taking calls, and playing music. It’s possible – even arguably likely – that Amazon intends to develop this product line into AR glasses in the long term future.
If you’ve got information to share with UploadVR you can email us via tips@uploadvr.com.
YouTuber Bradley Lynch claims to have seen Meta’s upcoming Project Cambria headset, and collaborated with a product designer to produce detailed renders.
Cambria was announced in late October at Connect 2021. Cambria isn’t Quest 3 – CEO Mark Zuckerberg described it as “a completely new advanced and high end product” positioned “at the higher end of the price spectrum”. Cambria will be sold alongside Quest 2 as a higher end alternative.
Cambria appears to have a much more balanced design than Quest 2, with a significantly smaller frontbox and a strap resembling Quest 2’s elite strap accessory, potentially housing the battery. Meta says this is achieved through the use of multi-element pancake lenses used instead of fresnel lenses. Whereas Quest 2 has grainy black & white cameras, Cambria will have high resolution color passthrough for mixed reality experiences. It will also include eye tracking and face tracking to drive avatars in social VR.
At the time of the announcement, Meta confirmed it had sent development kits out.
Past ‘Quest Pro’ Leaks
Cambria was seemingly once known as “Quest Pro”.
In February 2021 now-CTO Andrew Bosworth was asked about the prospect of ‘Quest Pro’. He replied “Quest Pro, huh… Interesting…” and winked at the camera. By April 2021 Bosworth’s stance on “Quest Pro” became more solid. In a public conversation with “Consulting CTO” John Carmack, he remarked “I did hint at an AMA earlier this year about Quest Pro because we do have a lot of things in development where we want to introduce new functionality to the headset”.
In September references to ‘Quest Pro’ were found in the Quest firmware, as well as eye and face tracking calibration steps. Just two weeks later, the leaks significantly intensified.
Screenshots of a video call leaked showing Touch controllers with onboard cameras instead of an LED tracking ring. But days before this, a redditor going by the username Samulia had posted low resolution renders of these same controllers, alongside detailed apparent specs of the headset.
In that reddit post Samulia claimed:
The headset’s codename is “Seacliff” and the controllers codename is “Starlet”.
The display is a Dual-Cell LCD with the same resolution as Quest 2 but an advanced backlight with pixel-level control, enabling OLED-like black levels without OLED’s black smear or manufacturability issues.
Three sensors are on the exterior of the headset: a 4K 120FPS RGB camera for color passthrough & mixed reality and two side-facing 1K near-infrared 120FPS cameras.
Some form of laser pattern projection guides controller tracking and potentially aids hand tracking.
Internally there are 480p 120FPS eye tracking cameras and two 400p 120FPS face tracking cameras.
Days after this YouTuber Basti564 – known for finding upcoming features in Quest firmware on multiple occasions – posted a new video backing up Samulia with firmware findings revealing the same sensor configuration as claimed and the same “Seacliff” codename. That suggested either Samulia had insider knowledge, or was using the same firmware decompiling methodology as Basti. Basti also found references to Seacliff having two cooling fans, up from one in Quest 2. The primary CPU cores in Quest 2 are actually underclocked. Better cooling could allow for significant improvement in CPU clock speed, and perhaps even GPU overclocking too.
In October, just days before Cambria was teased by Meta, Basti discovered tutorial videos in the firmware giving us the first look at the headset’s design.
By November, just weeks after the dark teaser video, Basti had found textures of Cambria and its controllers, and formed them into a 3D model.
“Don’t expect Cambria to be something that different. It’s different, but not that much.”
The source apparently claimed Cambria has a resolution of 2160×2160 per eye, up from Quest 2’s 1832×1920 per eye, and that the field of view feels “very similar” to Quest 2.
Yesterday Lynch posted a new video in which he claims his source showed him real images of the Cambria headset via an app which prevents screenshots. Lynch worked with product designer Marcus Kane in the VR app Gravity Sketch to produce a detailed 3D model of what he saw for renders, seen above.
The renders look essentially identical to Meta’s teaser video and the leaked tutorial video, with one notable exception. Underneath the front of the headset is two cameras near the bottom (in a similar position to Quest 2’s) as well as an apparent sensor suite directly in the middle. It’s unclear whether these sensors are a recent addition to Cambria, were simply not included in the 3D models and dark renders we’ve seen to date, or whether the reconstruction reflects the actual sensor layout of the upcoming device. Lynch speculates these may be time-of-flight sensors for hand tracking, though it may also be the laser pattern projectors Samulia claims.
Meta hasn’t given a specific release window for Project Cambria, but as recently as December confirmed it’s still on track to launch in 2022.
Samsung is working on its own Android-based AR headset separate to Microsoft, South Korea’s Electronic Times reports.
Four weeks ago, a report from Insider’s Ashley Stewart claimed Microsoft shelved HoloLens 3 last year in favor of letting Samsung build hardware powered by Windows mixed reality software. One of Stewart’s sources described the partnership as a “shit show”. But the Electronic Times report claims Samsung is planning “its own AR devices and the AR devices developed with Microsoft”.
The report suggests Samsung will use one of its own Exynos chips in the device, rather than sourcing from Qualcomm. Samsung has apparently completed a prototype, and “is deciding the release date”.
From 2014 Samsung partnered with Facebook on the phone-based Gear VR, technically the first widely shipped consumer VR product, but by 2019 then-CTO John Carmack declared it dead as standalone headsets took over. In January 2020 at CES, Samsung showed off a concept of AR glasses, but didn’t actually say whether this was a planned product. That same month China’s intellectual property office awarded Samsung a patent for a VR headset with four tracking cameras, but no product emerged from this either.
It’s unclear what exactly would distinguish Samsung’s own headset from its partnership with Microsoft, but if the report is to believed Samsung is finally ready to re-enter this market again.
Squingle, the psychedelic puzzle game for PC VR and Quest, is one of those “so out there, it’s difficult to explain” type of experiences. And as VR games creatively explode in every possible direction, the more vital it becomes to pop a human into frame to visually explain just what the hell is going on and why it’s so cool.
Squingle is admittedly difficult to pigeonhole just by looking at some screenshots or even a few gifs. We already have a good 450 words on the subject too, and it still may not be enough. It’s not that Squingle is overly complex, it’s just that the clever little ball maze-puzzle-thing is simply better understood in-headset.
To see what we mean, take a look at the original pre-release trailer:
Maybe it’s the undulating intestinal pathways, or the shimmering patterns that seem to change with every touch as you lead orbs around its increasingly circuitous routes. It’s so otherworldly and seemingly chaotic that you don’t implicitly know what to expect when looking at Squingle, whereas a different VR title might be easily explained with something as simple as a screenshot of a zombie and a gun. It’s a lot to take in.
If you were looking for some quick clarification though, the game’s new mixed reality video does a much better awesome job of shedding more light on why Squingle is so cool.
Mixed reality trailers like this aren’t new; Valve tossed out one of the best marketing pieces for VR we’ve seen, and that was way back in 2016 when they were hyping the general public for the original HTC Vive. In the following years, mixed reality capture has grown leaps and bounds, with many studios getting wise to third-party software like LIV—an easy to integrate mixed reality capture platform that many (many) studios use today. Still, there are plenty of games out there that would benefit from this sort of immersive, third-person view.
This is precisely why Meta is starting to push its own mobile mixed reality setup, which critically doesn’t require specialized equipment outside of a Quest and an iPhone. It’s easier for streamers looking for views, and anyone hoping to show their grandma what all the fuss is about. It’s not as good, or as full-featured as LIV, but the utility here is clear.
This will undoubtedly bring a wider swath of would-be VR developers, some of which likely want to make something they’ve never seen or experienced before—typically meaning unique, but difficult-to-explain mechanics. And using an actual human being as an anchor to visually tell what’s happening just seems more relatable than a ghostly avatar outline or claustrophobic first-person view. The protagonist is clear, so the objective becomes more apparent.
If you’re curious to learn more about Squingle, indie developer Benjamin Outram has published it on Quest via App Lab and on PC VR via Steam. You can also download the Quest demo or SteamVR demo for free if you’re still skeptical.
– – — ––
The Squingle mixed reality video above was taken by TougeVR, featuring Twitch streamer and sometimes Vtuber ‘Dubu’ (@dubusaeyo).
TougeVR is quietly making some of the highest quality mixed reality cinematics out there, and has a tutorial series called ‘Mixed Reality Masterclass’ which guides you through the process of mixed reality filmmaking. Check out TougeVR’s channel for more.
App Store logs and GitHub code from Apple confirm the existence of realityOS, expected to be used in Apple’s upcoming headset.
The existence of realityOS was first reported by Bloomberg all the way back in 2017. In 2021 Bloomberg, The Information, and supply chain analyst Ming-Chi Kuo released reports claiming Apple is preparing to release a premium headset for VR and AR with high resolution color passthrough. Recent notes from Kuo claim this headset will weigh significantly less than Meta’s Quest 2, feature dual 4K OLED microdisplays, and use a new chip with “similar computing power as the M1 for Mac”.
Three weeks ago, iOS Developer Rens Verhoeven spotted a new platform “com.apple.platform.realityos” in the App Store app upload logs. Apple’s existing operating systems include iOS (com.apple.platform.iphoneos), iPadOS, watchOS (com.apple.platform.watchos), macOS, and tvOS.
This week, “award-winning git repository surgeon” Nicolás Álvarez spotted Apple committing code to its open source GitHub repository referencing ‘TARGET_FEATURE_REALITYOS’ and ‘realityOS_simulator’ – the latter likely a feature to allow developers without the headset to test building AR or VR applications. Álvarez says Apple quickly force-pushed the repo to try & hide the change, suggesting making this public was a mistake.
This isn’t the first public confirmation of Apple working on AR & VR software. In December a job listing was posted for ‘AR/VR Frameworks Engineer’, with the role described as “developing an entirely new application paradigm” for “software that is deeply integrated into our operating systems”.
If Apple can pull off putting an M1-tier chip in a slim headset, it could deliver a significantly higher fidelity experience than Quest 2, and even Meta’s own upcoming take on a premium headset, Project Cambria. Last month Bloomberg reported the product release may have slipped to next year, and claimed Apple has “weighed prices north of $2000”.
Earlier this year the Unity Labs team shared an incredible proof-of-concept mixed reality demo that shows the power of blending the real and virtual worlds together. Now the developers behind the project say you’ll be able to get your hands on the experiment on Quest sometime next year.
At Facebook Connect at the end of October, the Unity Labs team revealed Unity Slices: Table, a proof-of-concept social mixed reality app that seamlessly connects people—both local and remote—into a shared experience centered around a game board.
It’s tough to explain so let’s jump right to a video example:
Take a look at the video above. We can see two other users around the table, and then we see the view transitioning seamlessly between the passthrough view of the real world and the virtual world. But do you notice anything else interesting?
Of the two users we see, one is actually there, and the other is not. As the virtual view is wiped away in favor of the real-world view, the local player’s real body becomes visible, but the virtual player’s body remains as an avatar (because they aren’t actually there in the real room).
As Eric Provencher, one of the developers behind the project, explained in a breakdown of the demo, one goal of Unity Slices: Table is to dissolve the barrier between local and remote users by making either kind of player feel equally present in the experience.
This is why the core of the experience is built around a virtual chess board which serves as a central anchor for everyone in the scene, whether they are actually in the same room or on opposite sides of the world. Beyond bringing everyone together around a shared point in space, the chess board rests atop a real surface, which turns it into a sort of virtual touchscreen with real haptics (thanks to the real surface underneath). Everyone (local or remote) collectively ‘touches’ the same board within the same spatial frame of reference, making it too feel like a shared piece of reality.
“It took us awhile to get to a system that worked smoothly, but the moment we first hopped into a networked session with expressive avatars, and could both see and hear the other person tapping our table over the voice chat as if we were in the same room, was truly mind-blowing,” wrote Provencher. “It felt almost magical to bring this tangible part of our reality into a shared experience.”
This week Provencher affirmed that Unity Slices: Table will be released as a demo on Oculus Quest for anyone to try.
“My team is still hard at work polishing up Unity Slices: Table up for release, look for it on App lab in 2022!”
Beyond being an incredible mixed reality demo, hopefully it’ll also be a fully functional multiplayer chess app. We look forward to mixing reality ourselves next year.
The latest update for Oculus Quest, v35, is rolling out to headsets starting today and brings with it mixed reality casting, global cloud saves, and Facebook Messenger calls in VR.
Mixed Reality Casting on Quest
Update v35 is bringing mixed reality casting to both Quest and Quest 2. This feature allows you to cast the VR experience to a smartphone while seeing your real body right inside of the virtual world.
This kind of mixed reality capture usually requires a complicated greenscreen setup with dedicated cameras and PC hardware, but having it built directly into Quest and the Oculus smartphone app stands to make it easy for anyone to do. Here’s an example shared by Meta showing how the feature looks with Beat Saber:
It even works with a moving camera view! While this is far from the production-ready quality you can achieve with a dedicated setup, it’s pretty amazing to see this capability built right into the headset and the companion app. Only six Quest apps are supported at present—Beat Saber, Superhot, Richie’s Plank Experience, Pistol Whip, Synth Riders, and Gravity Sketch—though we imagine developers will be encouraged to add support now that this capability is easier for players to use.
iPhone-only, For Now
Mixed reality casting on Quest only supports modern iPhone models and no Android phones, tablets, or iPads, yet. There’s no word on Android support, but in the past Meta has brought iPhone-only features to Android phones as well, so we hope this won’t be an exception. Here’s the complete list of devices which currently support mixed reality casting on Quest:
Mobile Device
Supported
Not Currently Supported
iPhone XR
iPhone XS
iPhone XS Max
iPhone SE – 2nd version
iPhone 11
iPhone 11 Pro
iPhone 11 Pro Max
iPhone 12
iPhone 12 Pro
iPhone 12 Pro Max
iPhone 12 mini
iPhone 13
iPhone 13 Pro
iPhone 13 Pro Max
iPhone 13 Mini
iPad (all models)
Android phone (all models)
Android tablet (all models)
Wasn’t This Already Added?
You might recall that a prior Quest update added a similar feature called Live Overlay, but it’s actually very different than mixed reality casting. Live Overlay simply superimposes the player on top of the first-person view that you normally see when casting with Quest. Mixed reality casting, on the other hand, actually shows the player in the virtual environment with a one-to-one correspondence.
The new cloud save system automatically backs up game saves so that players can retain their game progress even if they uninstall an app or change headsets. The feature works by default with all applications unless a developer chooses to opt-out.
Although v35 is required for global cloud saves on Quest, Meta says it will be rolling out the feature gradually “to make sure it works correctly,” so be patient if you don’t see it right away.
Facebook Messenger Calling in VR
Image courtesy Meta
Another feature that Meta announced last month is also rolling out alongside v35—Messenger voice calling on Quest. For a while now it’s been possible to send text-based messages through Messenger on Quest, but now you’ll be able do voice calls too.
Meta says this feature will be rolling out “over the coming weeks,” so if you don’t see it right away, fear not.
– – — – –
As with prior updates, v35 will roll out slowly to Quest and Quest 2 users, likely over the course of a week or more, but you can check for an update manually to see if it’s available to you. Here’s how:
How to Update Quest and Quest 2
In your headset, bring up the Quest menu by pressing the Oculus button on your right controller. Click on the clock to access Quick Settings.
At the top right of Quick Settings, click the Settings button (gear icon).
On the left of the Settings section select ‘About’ at the bottom of the list
Look next to the ‘Software Update’ label to see if a new version is available
Check the ‘Version’ label to see which version is currently installed
If v35 isn’t available but you think it should be by now, you can also try restarting your headset and repeating the above steps.
Varjo’s XR-3 headset has perhaps the best passthrough view of any MR headset on the market thanks to color cameras that offer a fairly high resolution and a wide field-of-view. But rather than just using the passthrough view for AR (bringing virtual objects into the real world) Varjo has developed a new tool to do the reverse (bringing real objects into the virtual world).
At AWE 2021 this week I got my first glimpse at ‘Varjo Lab Tools’, a soon-to-be released software suite that will work with the company’s XR-3 mixed reality headset. The tool allows users to trace arbitrary shapes that then become windows into the real world, while the rest of the view remains virtual.
This makes it possible to bring parts of the real world into virtual reality. For instance, as the company shows, you can trace the outline of a physical steering wheel peripheral so that you can see the real wheel inside of the virtual world.
The Varjo Lab Tools software supports three core masking modes: static, depth, and marker-based. With the static mode you can draw a passthrough window that will essentially be tracked to your head. With the depth mode you can have the system automatically pull in anything that’s within a certain distant to the headset (judged by the lidar sensor). You can imagine this being used to automatically show your real hands holding something when you raise them up in front of you, without pulling in the background of your room.
In the marker-based mode you can create a window that’s tracked to a simple fiducial marker and moves with the marker. This would be great for something like the steering wheel example above, or allowing your real keyboard to persistently appear in the virtual world. The markers can be easily printed and mounted anywhere… and you can track up to 1,000 of them.
At AWE 2021 I got to try the system for myself, and I was impressed with the user-friendliness of the tool. To draw a marker-tracked mask, for instance, you just use a VR controller to select the marker you want to track, and then trace out the mask with a simple point-and-click system. For example, I used the tool to mask out an area around a gamepad that had one of the markers attached to it. Then when I jumped into VR I could see a window around the gamepad, allowing me to look down and see my real hands on the gamepad and easily reference the position of my fingers with regard to the buttons.
And what’s particularly cool is that all of this passthrough magic is happening completely independent of the VR application, which means the passthrough windows that you draw can function inside of any app without a special integration by the developer.
It’s a very cool system that makes passthrough fundamentally more useful, but it isn’t yet flawless. One of the most obvious limitations right now is the update rate of the masked area. While the view through the passthrough window (ie: the view of the real world) updates at a perfectly fast rate, the shape of the window itself (when attached to a marker) only seems to update a few times per second. So if you’re moving the marker even slightly fast you’ll see the passthrough window lag behind as it tries to keep up.
I’m not sure what the limiting factor is on the update rate for the passthrough window, but I take it this is something that will be improved in the future. A much faster update rate would make real-world objects appear much more seamlessly within the virtual world.
– – — – –
Varjo says that the passthrough masking capabilities can also be used with the chroma key functionality it has introduced previously, making the XR-3’s passthrough system very functional and flexible compared to anything else on the market today. Indeed, just as the impressive resolution of Varjo’s flagship headsets has long felt like a glimpse of the eventual future of consumer VR headsets, we’d also hope to see this kind of advanced passthrough functionality come to consumer headsets, eventually.
For now the capability is only on Varjo’s high-end enterprise XR-3 headset (as it’s the only one with all the passthrough hardware necessary to make it all happen). The company says the Varjo Lab Tools software, which will enable these features, will launch alongside the next major update to its core ‘Varjo Base’ software.
