In place of GDC this year, which was postponed due to the coronavirus, Facebook is announcing some of its news via the decidedly more sterile environment of the Oculus blog, starting today and going until Thursday, March 19th. Today, the company unveiled its next major update to the Oculus Quest system experience.
Facebook says in a blog post that its next Quest update will make VR “more flexible, social, and easy to use than ever before,” touting a redesigned Universal Menu, new immersive overlays, and multi-window support for 2D apps. The updates are slated to launch as Experimental Features on Quest later this month.
The redesigned universal menu is supposed to organize information more clearly, something Facebook says will let you navigate to commonly-used system apps like Explore, Store, Browser, and TV with more ease.
Settings such as brightness and volume are also included here, which all seems like a nice quality of life tweak. Much like a mobile OS, the universal menu redesign will bring recently and frequently used apps to the forefront.
Image courtesy Facebook
To boot, the new immersive overlays make it so you don’t need to return to Home, as you will be able to bring the Universal Menu up while in-game. Tools such as friend requests, casting, and livestreaming is also supposed to be easier and quicker to find while you’re in a VR app.
Facebook says the new overlays will first be available in WebXR apps launched from Oculus Browser, and will later come to “more apps on the Oculus Platform as developers add support.”
Image courtesy Facebook
Something Quest users have been waiting for: multiple window support for Oculus Browser. A new control box is said to let you open, close, and rearrange windows. Support for more 2D apps like Chats and Store is coming at a later date.
Image courtesy Facebook
Outside of the experimental user-focused updates, Facebook is also about to make it easier for developers to manage and create DLC and in-app purchases for their games. Facebook says the update, which is also slated to arrive sometime this month, will make it this ‘add-on content’ more searchable and purchasable directly from the application page for consumers.
The latest 8.0 update to Oculus Browser adds experimental support for the Oculus Quest’s hand tracking feature. That means you can use your hands in WebXR apps, which are VR experiences that run natively on a browser page, without the need to download anything.
The feature won’t work in all WebXR apps natively and is not true hand tracking support just yet — it only enables hand tracking to work as an emulated controller. The update arrives just as Facebook adds enhanced support for hand tracking on Quest.
While the experimental feature is not listed in the Oculus Browser 8.0 release notes, a member of the Oculus WebVR/XR team, Artem Bolgar, tweeted about the addition earlier today. As noted by Artem, the support works by emulating controllers and doesn’t yet support tracking a full hand model like you would use in Oculus Home yet.
In the example app linked by Bolgar, the feature tracks your hands to move the in-game controller models in 6DoF. You can then point at objects using the cursor and pinch to make a selection, which will change the color of the cubes. Although you can see this displayed in the video below, it doesn’t show the pinching motion of my fingers, as there’s no in-game representation of the action (as the hand tracking is simply emulating controllers).
The new feature is not enabled by default and needs to be enabled first. You can enable it by navigating to chrome://flags in Oculus Browser and turning on the WebVR hand tracking option. There are a couple of different input methods, but the one we used in the video and the one that worked best is “hands and pointers.”
While the feature is only very experimental for now, and clearly not a proper implementation of full controller-free hand tracking in WebXR applications, it is the first step towards full support. This would open up a wealth of possibility, such as using your own hands to shoot webs as Spider-Man in a WebXR app that runs entirely through the Oculus Browser.
Recently, Mozilla unveiled ‘Hello WebXR’, a demo designed to show that WebXR is a viable alternative to store apps. We tried it out, and found it certainly makes a convincing case.
The Mozilla Foundation is a not-for-profit organization which seeks to advance the open web. It develops and maintains the open source Firefox web browser, including the mobile VR version Firefox Reality (available on Oculus Quest, Oculus Go, Vive Focus, Lenovo Mirage Solo, and almost all other standalone headsets).
WebXR is an open standard application programming interface (API) which enables 3D webpages to display on VR headsets and be interacted with by VR controllers. WebXR experiences are rendered with WebGL, usually with an engine like three.js.
Impressions Of This Demo
Hello WebXR contains a large hall as well as three portal doors which bring you to other mini-experiences. Locomotion is handled by a simple teleport system in which you point your handed controller and press the trigger.
The experience uses controller models rather than hands, and the controllers are generic rather than tailored to the headset you’re using. Mozilla mentions that they plan to use WebXR Input profiles in the future, which may solve this problem.
The first thing that stood out, and thus the first thing I tried, was the xylophone in the center. Using the triggers each stick can be picked up to play the keys. The interactions feel smooth and solid, other than the lack of haptics.
Also in the center of this hall are spheres which, when selected, immerse you in a 360 degree photo, similar to the user experience of Valve’s The Lab. Each loads instantly, demonstrating how WebXR can be easily used to distribute this kind of content.
But what about bringing the regular web into the VR web? Mozilla demonstrates this with a “Twitter Feed” screen, which shows Tweets that use the #helloWebXR hashtag. This is where WebXR shines, developers can more easily integrate the wealth of APIs and frameworks that already exist for the web ecosystem.
Current VR headsets have limited resolution, and the lack of variable focus means you don’t want to get extremely close to things. So how can VR be used to show detailed paintings or other works of art? Mozilla’s solution is a magnifying glass, and it feels entirely intuitive and natural to use.
One of the three rooms you can teleport to is the “dark room”, which lets you hear positional sounds (crucial for VR audio). This is probably powered by Resonance Audio. The positioning was relatively precise, but not quite as pinpoint as I’m used to with Oculus Audio or Steam Audio.
Another of the rooms was incredibly impressive — a classical sculpture captured using photogrammetry. Even on the mobile headset, this essentially looked photorealistic. This was not something I expected from a web page, especially not one that loaded instantly.
The experience also performs excellently on the Oculus Quest standalone headset. Quest uses a smartphone processor with limited power, so it is notable to see it handle all of this so smoothly.
When we tried WebVR in the past, even on PC, performance was mixed. Hello WebXR seemed to maintain a constant 72 FPS, with the exception of the graffiti wall, where the framerate was significantly lower.
The Potential Of A VR Web
Oculus Quest is a semi-closed platform, with a heavily curated store. Facebook will only allow apps that meet its strict standards on quality and value to be distributed through its store.
Even on the more open stores for PC VR content still requires a basic approval process, and sometimes fees (Steam charges $100, for example). Developers are reliant on each store to distribute to a large number of users. While executables can be distributed through the web, this is a fairly major security risk.
WebXR makes the argument that just like websites, no central authority (other than government in case of law violation) should control VR content. And because browsers keep web code in a sandbox, there shouldn’t be any security risks.
The SideQuest project aims to offer an alternative route to distribution on Oculus Quest, but this requires connecting your headset to a PC and apps still have to be downloaded and installed. WebXR enables content to be delivered near instantaneously directly from the headset from a simple web URL.
A problem that arises, however, is monetization. This is more necessary on the web than through a store since each developer would be hosting their own app. While web hosting and bandwidth has never been cheaper than today, it is still a recurring cost. And of course, developers may build WebXR apps as deserving of a price as store apps.
The Oculus Store within Quest allows the user to enter a simple PIN code to use the payment method on their account. Would users really grab their debit card and awkwardly enter their details while the headset is half on their head? Probably not. But the open standard Payment Request API (which Facebook is contributing to) seeks to solve this kind of problem on the web, so in the future we might see payments passed through to the Oculus Browser.
How To Try It
You can access Hello WebXR on Quest via Oculus Browser — just Google Search ‘Hello WebXR’ inside it.
Mozilla launched a new WebXR app called ‘Hello WebXR’, which is compatible with most headsets that use web browsers such as Oculus Browser on the Quest or Google Chrome on PC VR headsets. The app acts an an introductionary experience for those who are new to VR, showcasing the different types of content and interactions available on the platform.
The experience will work on any WebXR compatible browser on a headset, including Mozilla’s own VR browser Firefox Reality. Other browsers, such as the Oculus Browser on Oculus Quest or Google Chrome on Oculus Rift, also support WebXR and should work with the Mozilla Hello WebXR site.
The main hall environment is where you start Hello WebXR, and features a bunch of different objects to interact with. You can play the xylophone, look at some paintings, look at a Twitter feed displaying any tweets using #hellowebxr, and even spray some graffiti on the walls. The main hall also has some globes that will transport you to 360 degree panoramic scenes.
Three doors will teleport you from the main hall to other locations, where you can experience positional audio, look at some classic sculptures and walk around the top floor of a skyscraper.
Mozilla stated in a blog post that they expect the experience to grow over time, and develop it into “a sandbox that we could use to prototype new experiences and interactions.” To try out Hello WebXR for yourself, just head to this page on your WebXR-compatible headset.
Starting this week with the launch of hand tracking, Oculus Quest owners no longer need controllers to operate the standalone VR headset.
UPDATE Dec 10: v12 is now rolling out with this feature available!
Facebook calls the feature “Hand Tracking” and it uses the on-board cameras to identify the movements of your fingers and hand without held controllers. It will be available in the ‘Experimental’ section of the Settings in update v12, which is rolling out this week.
This is a firmware update, no new hardware is required. In fact, less hardware is required. For the first time, Quest owners will be able to demonstrate some basic functionality of the standalone headset to friends and family for the holidays in 2019 without Oculus Touch controllers.
The company first announced this feature at Oculus Connect 6 in September, stating that it would arrive in ‘2020’. But in a holiday season surprise they’ve managed to ship it early — for system apps at least.
This isn’t the first headset to support this feature. HTC’s PC-based Vive Cosmos VR headset also supports optical finger tracking. Doing this on a standalone headset, however, is significantly more difficult, and reports of the Cosmos finger tracking quality have not been great.
Quest System Apps At First
The only apps which will support Hand Tracking this week are a handful of system apps. So you’ll have to wait a bit longer for some of your favorite games and experiences to support the feature.
The main Oculus menu, including the Store and Library
Oculus Browser (the built in web browser)
Oculus TV (for both 2D content and 360 videos)
This means you can use Quest as an immersive media viewer or portable cinema without picking up or bringing along the Touch controllers.
We don’t know when Facebook will launch Elixir, an introductory experience used to show off the feature to media at Oculus Connect 6.
SDK Available Next Week
Next week, Facebook plans to release the software development kit (SDK) for Hand Tracking, so that any Quest developer can support the feature in their app or game.
Oculus Avatars is targeted to support finger tracking in 2020, which will allow developers to add hands matching the user’s actual skin tone by default.
We don’t yet know which developers will support Hand Tracking, but we’d expect most social and media apps to do so. We’ll start a list of Quest apps with plans for hand tracking support, so please email tips@uploadvr.com if you’re a VR developer embracing the feature. Bigscreen’s CEO already hinted at support and we confirmed that Virtual Desktop from Guy Godin will be working on the feature when he gets access to the SDK.
Facebook expects the first third party apps to start supporting the feature in 2020.
Keep in mind that it is likely only some games will be adding Hand Tracking support. Many games require buttons, analog sticks, triggers, or haptic feedback. Additionally, based on our experience at Oculus Connect 6, the tracking quality likely wouldn’t be suitable for advanced two hand interactions.
A Long Researched, Challenging Feature
The first indication of Facebook’s interest in finger tracking for VR was revealed in late 2014. The company acquired startup Nimble VR, composed of four veterans of hand tracking technology. But upon this acquisition, Oculus was clear that this technology “may not even be used in the CV2 or CV3”.
That’s likely because Nimble’s tech wasn’t just software, but a dedicated depth sensor. Depth sensors deliver excellent finger tracking such as that found in the new HoloLens 2 AR headset. But these dedicated sensors can add cost and weight to headsets when reducing both is key to broader adoption of the technology. Delivering high quality finger tracking on regular cameras, as is the case on Quest, is a different level of challenge. Its implementation on Quest using the same hardware employed for other kinds of tracking is likely a competitive advantage for Facebook and something likely to be more broadly adopted in future products from the company.
In 2016 at Oculus Connect 3, Chief Scientist Michael Abrash made a range of predictions about VR in the year 2021. He noted that finger tracking could be done perfectly with markers on gloves (and showed this), but not yet directly. However, he predicted that by 2021, it would be possible:
With the roll-out of this feature, then, Facebook will have essentially beaten Abash’s prediction by a year. In 2018 at Facebook’s annual F8 conference the company showed high quality gloveless, markerless finger tracking:
An Oculus representative explained on stage that this “breakthrough” was made possible by Facebook’s heavy investment in machine learning (“AI”) research. They use the kind of glove system described by Abrash in 2016 to train a convolutional neural network to recognize hand poses. This is likely how the feature was made possible.
“If I put my fists together my hands could disappear. If I turned my head to the side and then looked back at my hands it would take a second for them to pop back into existence at a spot where I expected them to be. If I moved my hands too fast they could disappear. If I touched my pointer fingers tip to tip their virtual representations still showed a slight gap between them before quickly correcting themselves to a more believable position. All of these I would characterize as momentary lapses during deliberate attempts to break the tracking rather than just enjoying the things for me to do in the fantasy demo world.“
Unless Facebook made dramatic improvements in the past three months, you won’t have the same tracking robustness you get with the Touch controllers. The controllers contain an accelerometer and a constellation of infrared LEDs designed specifically for tracking in a range of conditions.
But finger and hand tracking should still be good enough for social VR, media consumption, and some very casual games designed specifically with the feature’s limitations in mind. Facebook plans to “add new features and functionality to improve the experience of hand tracking on Quest in 2020.”
At Oculus Connect 6, Engineering Manager Robert Wang explained this current iteration “only scratched the surface of what we can do with hand tracking”. While he was careful not to give a roadmap, he did state that Facebook would “love to be able to” reliably track two handed interactions with regular cameras.
It will certainly be interesting to see what developers do with this feature, and where Facebook will take it in future.
Oculus Quest and Oculus Go are bringing a lot of new buyers to VR headsets in 2019 and that means a lot of new people using the Oculus Browser to surf the Web.
For those unfamiliar, the Oculus Browser lets you visit any normal web page while also including support for virtual worlds built with WebVR. The software includes most standard browsing features, including private browsing mode. We contacted Facebook recently to get explanations about what the company stores related to browsing activity.
We received some interesting answers over email and are including them below. If you have any questions or additional information — please share in the comments or email tips@uploadvr.com.
We’ll plan to update this post over time if we receive substantial updates.
What data, if any, does the Oculus Browser transmit back to Facebook? Does it transmit browser history?
The Oculus Browser does not share information with the Facebook family of apps unless you choose to log into Facebook via FB login (this would be the same information that Facebook would receive if you logged in to Facebook while using another browser). That said, as with using any browser, web sites themselves may collect information about you. What data is collected, how it is used, and who it is shared with is up to the web site and their privacy policy. Some web sites may share information about you with Facebook through wikipedia.com) to assess how many unique users visit a specific website to help us prioritize what sites we focus on from a compatibility and performance standpoint. We disassociate this information from our users because we only care that a person visited a site, not that YOU in particular visited that site. We limit the storage of this data to 30 days.
An additional point to add – one of the features of the Oculus Browser includes curated WebVR experiences that we recommend on the ‘New Tab” page of the browser. In this case, we measure when someone interacts with an experience we recommend to help us improve the recommendations and offer more relevant pages.
Has this ever happened in the past — whether a person has logged into Facebook or not?
As explained above, this is always how our Browser behavior has worked.
Does private browsing mode have any effect on this functionality?
When you are in private browsing mode, we don’t log data about the domain names users visit. And a quick reminder that we don’t show the new tab page in private browsing mode.
As part of the continued stream of announcements for the Samsung Gear VR taking place at the Samsung Unpacked event, New York, today, Oculus VR has announced the launch of a brand new web browser for the mobile virtual reality (VR) device. Available now, the Oculus Browser is available for all commercial editions of the Samsung Gear VR.
The Oculus Browser is a fully native app that renders video, images, and text on the Samsung Gear VR, as well as letting you search the web directly from Home. Using the virtual keyboard users can navigate search engines, websites and social media channels just like you would on a desktop. The browser brings 2D and most 360 degree video content right into Oculus Home.
There’s plenty more details on Samsung and Oculus VR’s initiatives to come from today’s Samsung Unpacked event, and VRFocus will keep you updated with all the latest details.
