Oculus Insight – Augmented & Virtual Reality Confabulation

December 23, 2019

Oculus Quest & Rift S Controller Tracking Patched To Work Near Christmas Trees

The controller tracking of the Oculus Quest and Rift S needed to be patched to work properly near Christmas trees and other holiday lights.

Oculus Touch controllers are built with a constellation of infrared LEDs under the plastic of the tracking ring. These lights are tracked by the cameras on the headset in order to determine the position of the controller.

Holiday lights like those on Christmas trees can look a lot like these LEDs to the cameras. This means the algorithm has more sources of light in each frame to analyze, and sometimes it can’t tell the difference between the controller LEDs and the irrelevant LEDs at all. This could make the controller tracking work poorly, showing the wrong position for the controller.

The solution works because the headset tracking algorithm already remembers static landmarks seen by the cameras in the room — that’s how it works without external sensors. By keeping a track of these landmarks the system can reject blobs of light which stay in the same position and don’t move.

This process on its own, however, is not enough to eliminate all the issues. So Facebook also trained a neural network to detect and filter out blobs of lights that are too small or too large to be a controller LED given its last position.

You can read a full technical explanation of the solution in Facebook’s blog post.

This isn’t the first time Facebook improved the controller tracking on Quest and Rift S. Both headsets use the Oculus Insight tracking system and launched on the same day. At launch, the controllers wouldn’t track when brought too close to the headset and tracking could break when one was placed in front of the other. This made games like shooters difficult to play until a patch was released one month after launch which fixed these issues.

The post Oculus Quest & Rift S Controller Tracking Patched To Work Near Christmas Trees appeared first on UploadVR.

September 4, 2019

Facebook: We Don’t Collect Or Store Quest Camera Or Guardian Data

Facebook says its Oculus Quest and Rift S headsets store room mapping data locally and do not send that data to their servers.

The explanation comes after attempts to clarify how the system functions for consumers and how it was developed and tested internally with employees. Facebook’s two key VR products of 2019 — Oculus Quest and Rift S — both rely on the camera-based Oculus Insight system to function.

According to Facebook, Insight is a “visual-inertial SLAM” system which uses “image data from cameras in the headset to generate a 3D map of the world pinpointing landmarks” combined with inertial measurement units in both controllers and the headset. This is done by the Oculus Quest’s Snapdragon processor, or by your computer on Rift S. The camera imagery is not stored, but a 3D cloud of points identifying the placement of unique static features of the environment is saved locally on your device for any room where you have a Guardian boundary set up. This allows the system to recognize what room you’re in and load the Guardian boundary you previously created. Facebook tells us that this data doesn’t leave the local device.

Oculus Quest Touch Controllers — Facebook’s Oculus Quest with the Touch Controllers. The standalone VR system includes everything needed to visit virtual worlds for $400.

We first reported in October last year that a white light on top of the Oculus Quest is “hard-wired to the sensor power rail.” That means whenever the camera sensors are supplied with power the light should turn on. In late August this year, Facebook explained in a fair amount of detail what happens when that light turns on. In particular, they explained how measurements from the cameras fuse in real-time with inertial measurement units in both the controllers and headset.

Rift S Delete Guardian Boundary Insight — Oculus “clear boundary history” option on a PC.

“This information is only stored locally,” a Facebook representative wrote in an email. “Not sent back to us.”

Quest owners could test some of this themselves by using the headset with the Wi-Fi connection turned off — either on the headset itself or at your local networking box. You won’t be able to download games or play online but the system will still let you move around in VR just fine.

Quest Delete Guardian Boundary Insight — Oculus “clear boundary history” option on Oculus Quest.

Here’s Facebook’s video explaining the Oculus Insight system:

Facebook employs the terminology “3D map of your environment” to describe this collection of measurements “which helps locate your headset and controllers in a known space so Quest/Rift S can work and keep you safe.”

“We don’t collect and store images or 3D maps of your environment on our servers today — images are not stored anywhere, and 3D maps are stored locally on the headset [for Quest] and on your local PC, where you have access to delete it [for Rift S],” a Facebook representative wrote in an email. “That said, we’ll notify consumers if this information is required for VR experiences we provide on Quest/Rift S in the future.”

“The only information we keep on our servers consists of performance metrics that don’t contain any recognizable detail about your environment,” an email states. “These metrics help us improve Oculus Insight.”

Facebook’s Employee Testing

An alarming post to Reddit sparked our most recent series of questions to Facebook.

The now-deleted post was allegedly from someone who worked at Facebook and said they debugged the Insight Tracking system during its employee-based internal testing phase. The post claimed that during the employee-driven process of testing the Guardian passthrough system there was a way to submit bug reports. These reports allowed workers back at Facebook headquarters to see screenshots, some of which included views into the bedrooms of the company’s executives. A Facebook representative wrote in an email that the “report is not a true representation of our internal testing practices.”

“For our internal hardware testing, employees volunteer to take home and test products and agree to share their experiences to help improve the product for consumers,” the representative wrote. “Employees are given the option to include a screenshot when they report bugs (to assist with debugging) and they can choose if they’d like to submit it for review.”

The representative added that employees were “made aware/understand that if they submit a bug in that passthrough mode, that an image of their surroundings will be included with the report. As we noted previously, this is for internal testing purposes only.”

If you try to take a screenshot at home with a consumer version of the Quest while in Guardian view — like when setting up a new play space — the cameras will be shown as black in the resulting screenshot.

Screenshot Passthrough Insight — The cameras are blocked when taking a screenshot with Oculus Quest at home.

So to reiterate, according to Facebook, there was a bug reporting system used only with employees which came with disclosures to let workers know that when they submit a report from the headset while in view of the surrounding environment, like during Guardian setup, the view of their physical rooms may be sent to Facebook’s servers.

“If an employee is in passthrough mode – i.e. Guardian Setup – for testing purposes, then yes, the visuals of what might be in view of the headsets (their desk, etc.) would be included in the screenshot that they can choose to include if they were to submit a bug report,” the representative wrote.

Critical, then, to Facebook’s position regarding the implications of that original post:

“People are made aware/understand that if they submit a bug in that passthrough mode, that an image of their surroundings will be included with the report,” Facebook explained. “As we noted previously, this is for internal testing purposes only. We strive to provide the same transparency to our employees that we provide to our consumers who use our headsets.”

I asked Facebook to send me a screenshot of the disclosure employees saw as I think there’s public interest in seeing it. Facebook said they can’t share that externally.

The post Facebook: We Don’t Collect Or Store Quest Camera Or Guardian Data appeared first on UploadVR.

August 22, 2019

Oculus Lifts the Lid on its Tracking Tech Oculus Insight

Oculus Quest might look like a nice compact virtual reality (VR) headset (in comparison to other VR devices) but it packs a lot under that hood. One of its core pieces of technology – which is also used by Oculus Rift S – is Oculus Insight, the inside-out tracking technology that helps keeps players safe without the need for cumbersome external sensor solutions. Today, the company has revealed the story behind Oculus Insight and how it gives players a full six-degrees-of-freedom.

Inside-out tracking was once considered the holy grail of VR freedom so that users weren’t confined to one play space, able to use a VR device pretty much wherever they were, their movements fully replicated in the virtual space whilst avoiding hazards in the real world.

Inside Facebook Reality Labs – previously Oculus Research – an engineering team was put together to tackle the problem. “We wanted to create a system that lets you move and explore a VR world just as naturally and easily as you would in real life,” says Anna Kozminski, Software Program Manager in a blog posting. “With inside-out tracking in the headset, VR becomes as easy as putting on headphones to listen to music.”

So at the core of Oculus Insight’s tracking solution is a technology called SLAM (simultaneous localization and mapping). This uses computer vision algorithms to combine multiple sensor data, fixing the position of an object – in this case a headset and controllers – within a digital map which is continually updated.

SLAM was first used in the early Oculus Santa Cruz prototype headset, unveiled during Oculus Connect 3 (OC3) in 2016. But the team needed an even more advanced version of SLAM if they were t achieve the level of precision and efficiency required for consumer VR gaming. To do this team drew upon Facebook’s previous SLAM work for mobile AR whilst experimenting with accurate OptiTrack motion-capture cameras to fine-tune the system’s computer vision algorithms.

The team then tested Oculus Insight with OptiTrack camera’s in a number of settings, exhaustively making sure the system worked in whatever condition thrown at it. And that’s why today, you can go out and by an Oculus Quest, charge it up and start playing VR in your living room, bedroom, hallway, pretty much anywhere with enough space (some people have even tried the headset outside). The solution may also help to provide better augmented reality (AR) experiences in the future.

August 22, 2019

Oculus Insight: Facebook Details Quest’s Inside Out Tracking System

Facebook provided background on the development of the inside out positional tracking technology which enables both Oculus Quest and Rift S to operate without any external cameras.

A pair of blog posts published today by Facebook explain how a team spread across the company’s VR development labs in Zurich, Menlo Park, and Seattle built the technology.

Facebook Optitrack Oculus Quest Home Testing — Facebook employees installed OptiTrack cameras in their own homes to test out VR tracking in a variety of conditions which could be used as the basis to improve Oculus Quest’s tracking system.

According to Facebook, the company used OptiTrack cameras and “by comparing the measurements recorded with the OptiTrack cameras with the data from Oculus Insight, the engineers were able to fine-tune the system’s computer vision algorithms so they would be accurate within a millimeter.” Employees tested out the cameras in their own work spaces and homes to recreate a variety of conditions in which a headset like Quest might be used.

Here’s how Facebook described the process:

“The OptiTrack systems would track the illuminators placed on participants’ HMDs and controllers and throughout each testing environment. This allowed us to compute the exact ground-truth 3D position of the Quest and Rift S users and then compare those measurements to where Oculus Insight’s positional tracking algorithm thought they were. We then tuned that algorithm based on the potential discrepancies in motion-capture and positional data, improving the system by testing in hundreds of environments that featured different lighting, decorations, and room sizes, all of which can impact the accuracy of Oculus Insight.”

“In addition to using these physical testing environments, we also developed automated systems that replayed thousands of hours of recorded video data and flagged any changes in the system performance while viewing a given video sequence. And because Quest uses a mobile chipset, we built a model that simulates the performance of mobile devices while running on a general server computer, such as the machines in Facebook data centers. This enabled us to conduct large-scale replays with results that were representative of Quest’s actual performance, improving Insight’s algorithms within the constraints of the HMD it would have to operate on.”

Facebook says that to get this same Simultaneous Localization And Mapping (SLAM) technology to work inside slimmer AR glasses, they’ll have to figure out how to reduce latency further while “cutting power consumption down to as little as 2 percent of what’s needed for SLAM on an HMD.”

Here’s a video detailing how the technology works:

The post Oculus Insight: Facebook Details Quest’s Inside Out Tracking System appeared first on UploadVR.

July 24, 2019

Oculus Rift S & Quest Receive Major Tracking Improvements

While Oculus’ inside-out tracking solution for Quest and Rift S has in many ways been a success, there’s always room for improvement and additional tweaks. The company has now rolled out several updates to enhance the overall controller tracking experience on both devices, as well as improving Oculus Quest’s Guardian system.

Oculus Rift S

First of all, the new Oculus Touch controllers for Oculus Quest and Oculus Rift S have had their tracking algorithms fine-tuned to update their accuracy, such as when hands are close to the headset or angled away from it. So players should see smoother tracking and fewer errors when using either device, great for throwing a few punches in Creed: Rise to Glory, reloading weapons in Gun Club VR or simply picking stuff up in Job Simulator.

And just for Oculus Quest, the Guardian system which keeps players safe in VR – hopefully no walking into walls – is being made even more accurate and flexible. The updated system is now better at automatically recognising multiple play areas in different rooms, so Oculus Quest can quickly and easily swap between locations. Or for those with one massive room with multiple areas to enjoy VR, there’s now support for several Guardian setups in the same room. Meaning there’s no fighting between siblings who may want their own little slice of VR heaven.

In addition to all of that, the Guardian settings page now includes options to adjust the floor height and alter playspace boundaries without having to start from scratch.

Oculus Rift S Lifestyle 1

Back on Oculus Rift S, there’s a new setting on Dash so you can turn on the stereo-correct passthrough feature, Passthrough+, for as long as you want. Addressing the issue some users have had with intermittent displays, Oculus says on its blog: “We’ve been working hard to address them with each software release since launch, and we’re close to resolving the occasional screen flashes that have been reported. Look for an update later this week, and thanks in advance for your patience.”

Plus, from today Oculus Home’s Gateways feature will let you link directly to your favourite videogames, experiences, and friend’s homes. As Oculus releases further updates, VRFocus will let you know.

May 1, 2019May 21, 2019

Oculus Quest – The Review: Everything You Need to Know

When the Oculus Rift arrived in 2016 it marked a turning point in the videogame industry, offering a new entertainment medium that opened the door to exciting virtual worlds that could be physically interacted with. But there were limitations and as we know it’s not been quite as big a success as virtual reality (VR) advocates had hoped. To break into that mainstream appeal Oculus released the Oculus Go last year, and for 2019 it’s the turn of Oculus Quest, upgrading the standalone experience even further with a device which does just about everything right.

First Impressions

Straight out the box the Oculus Quest has the look and feel of a proper high-end piece of kit – and it should for $399/£399. There’s a nice mix of fabric and plastic so the headset doesn’t come across as one big nasty lump of black plastic. The same does for the rest of the presentation, from the box to the controllers and the setup, it’s all as smooth and fluid as you’d hope for this price bracket – which is important when it comes to new VR customers (more on that later).

The Headset

Oculus wants the Quest to have a much broader appeal than anything it has created previously, helping drop those barriers that have been holding the industry back. One could say that strapping a computer to your face is the main obstacle but we’ll ignore that as demoing a decent experience can help to overcome initial hurdles.

The headsets’ two main features are certainly its trump cards, an all-in-one system and inside-out tracking. Straight away it’s easy to tell Oculus Quest is easy to use. With no PC to deal with and no sensors to arrange around a play space all that’s required is a clutter-free area to enjoy some VR gaming in; around 2m x 2m is fine for roomscale. There’s also a stationary option for those times where you just want to sit or stand, in one spot.

Oculus Quest can’t be set up without the accompanying app but rest assured its pain-free – especially if you already have an Oculus device. Once connected over WiFi you then have to set up the Guardian system to help avoid walking into walls which has been improved over the previous iteration on Oculus Rift. Rather than having to actually walk around your play area, the Oculus Quest’s cameras provide a pass-through black and white image so you can use a pointer to mark the area with the Oculus Touch controllers.

Oculus Quest Lifestyle 4

Those lenses also provide the highly important inside-out tracking called Oculus Insight. This was one of the most important features to get right, and it’s safe to say Oculus has done a very good job. While you do have to be careful that greasy fingers don’t touch the lenses and impede their performance – which is easy to do when taking the headset on and off – they do manage to track a very wide area, and at a decent speed. And this is full 6 degrees of freedom (6DoF) tracking which makes the Oculus Go’s 3DoF look antiquated.

If you’ve ever used an Oculus Rift then you will find Oculus Quest to be a little front heavy as it does contain all the computing and battery power. That being said, get all the straps correct and comfortable for your head and the extra weight (it’s 571g) isn’t uncomfortable, even on longer play sessions. Anyway, there’s also the small matter of it being battery operated. Actual battery duration is slightly imprecise as it depends on the operation but for gameplay sessions it lasted for just over two hours with Oculus claiming 3 hours for general media content, films and such. The USB-C charging lead is fairly long but what’s the point in using a wireless headset cabled. Don’t worry, as a full charge doesn’t take too long, from 5% it took 1 hour 45 minutes to get to 100%.

Now it’s time for the good stuff, the visuals. A VR experience is only as good as the screen and lenses, and these have seen a decent boost. Featuring an OLED panel with a resolution of 1440×1600 per eye (Oculus Rift had 1080×1200 per eye for comparison), the screendoor effect is greatly reduced (not eliminated). The lens clarity is superb for a headset at this price point, and the manual IPD adjustment makes fine tuning the visuals a doddle.

From the improved Oculus Go menu system to the videogames themselves, the colours are crisp and vibrant, with titles like Beat Saber and Dance Central suitably popping out from the screen. Even running at 72Hz on the Qualcomm Snapdragon 835 processor there are no horrendous latency issues that would make VR players nauseated in any way.

Visuals are only one part of the whole VR experience, audio plays a vital role in creating an emotional response. Oculus has used the same sound-pipe technology found in Oculus Go for the Quest, channelling audio down the arms of the headstrap. This works perfectly well, even managing to provide positional audio so you can hear where shots are fired from or where your teammates are located. The neat solution does have the drawback that any outside noise can enter your ear, which can be distracting and immersion breaking. A decent set of headphones are always recommended with a 3.5mm jack located on either side of the device, so long as you don’t mind the cable.

The Controllers

Oculus Quest might be good, but it’s nothing without some decent input, and the redesigned controllers help to seal the deal. Lighter than the previous model, the most obvious difference is the band which now goes over, rather than under the controller so it can be tracked.

They feel just as easy to use as before, with the wrist strap always a must for the more frantic videogames. The button and stick layout are the same, and you can perform the same hand and finger gestures as before.

The only real issue that I found was with the battery compartment lid. This was a much bigger cover than the previous model making it easy to replace the singular AA battery. However, on a couple of the more energetic titles like Creed: Rise to Glory and Sports Scramble, the cover did begin to open. The design still has the two magnets inside to hold the cover, but they do seem weaker, making the compartment easier to open than before.

The Games

Oculus has revealed the 50+ titles for the launch of Oculus Quest on 21^st May. for the review a decent sample of experiences were made available (some in preview form, others whole), including Journey of the Gods, Thumper, Racket Fury, Beat Saber, Dance Central, Ballista, Apex Construct, Wander, Bait!, Rush, Dead and Buried II, Virtual Virtual Reality, Space Pirate Trainer and several more.

Before trying out the new ones like Ballista and Journey of the Gods, a port test had to be done; whether titles like Apex Construct and Creed: Rise to Glory would work on the portable system. It safe to say that they do, for the most part. Where you might think it’s the visual side that would struggle it isn’t, actually it’s the tracking. Yes, there’s a drop in the visual quality, but nowhere near what you’d expect, with all the studios doing a fine job of squeezing these videogames onto Oculus Quest.

They do however test the tracking to its max. For example in Apex Construct the bow would glitch across the screen when my hands were too low (near my waist), coming back into place when raised. The tracking also struggled with the speed of Creed: Rise to Glory. As a boxing sim, the controllers were right in front of the headset, between all the sensors. Even so, it couldn’t quite keep up with a flurry of punches. But it’s unclear whether it’s Oculus Insight’s fault or the software’s. In comparison, Beat Saber which is known for its fast gameplay didn’t miss a beat (pun intended). There was no issue whatsoever when playing a song on Expert (Expert+ is just too fast for me), with the tracking working beautifully.

The Verdict

So does this mean Oculus Quest is a game changer when it comes to VR? Well yes and no. The all-in-one design is ideally suited to welcoming new players into the fold. Being able to take a device that can display experiences that are almost Rift quality anywhere might just tip the scale in Oculus’ favour. However, for those who already own an Oculus Rift, Oculus Quest doesn’t quite make for a suitable purchase. Yes, there are a growing number of titles with cross-buy support so you’ve got an instant portable library, but you’ll miss that outside-in tracking.

March 2, 2019

Oculus Quest ‘Significantly Faster’ Than Oculus Go, 6DoF Tracking ‘Doesn’t Affect’ Performance

Oculus Director of Ecosystem Chris Pruett revealed in a forum AMA yesterday that Oculus Quest will be “significantly faster” than Oculus Go:

Quest is significantly faster than Oculus Go from both a CPU and GPU perspective. Part of this is just the raw performance of the chipset itself, but a lot of it has to do with the effort we’ve put into the design of the headset and the core rendering architecture. Tracking isn’t in contention with and doesn’t affect the performance of your application.

Oculus Quest is essentially a VR games console. Like Go it is standalone with all the compute onboard, but unlike Go it has room scale tracking and Touch controllers.

Whereas Go features a Snapdragon 821 chipset, Quest uses the newer and faster Snapdragon 835. It’s around 30% more powerful, or can achieve the same performance with around 40% less energy.

It’s All About The Cooling

But the chip itself likely isn’t the main reason for the “significant” performance improvement. Like Pruett’s comments suggest, it’s also the design of the headset.

A key element often overlooked in computing hardware performance is cooling. The limiting factor in smartphones for example is not the chip’s raw capability, but the fact that when running at full speed it will very quickly exceed its maximum temperature. To avoid hardware damage the chip must then downclock itself until it cools down more, or even shut down. This is called thermal throttling.

Oculus Go features a heatpipe, but not a fan

Thermal throttling is arguably the core flaw of smartphone VR. It can limit graphically intense experiences to a matter of minutes. To overcome this, Go used a heatpipe and the entire front is metal, allowing it to act as a heatsink.

This cooling system allowed Facebook to overclock the Snapdragon 821, and to sustain that performance for hours. The result was that Oculus Go performs “significantly better” than a Galaxy S7 using the same chip.

Quest goes even further with cooling by adding an active cooling fan. This has rarely ever been done with ARM processors. The new Apple TV and the HTC Vive Focus are the only instances on the consumer market we know of.

With the active cooling system, Quest should be able to have higher clockspeeds than smartphones or Oculus Go. Everything still needs to be rendered for each eye but the higher clock speed should provide more complex and detailed virtual worlds compared with Go. Of course, Quest will still not come close to the power of a PC.

Hardware Accelerated Tracking

Interestingly, Pruett confirmed that the 6DoF headset and controller tracking “doesn’t affect” performance. Developers seemingly won’t have to worry about it when optimizing their game.

This is because tracking is not done on the CPU, but rather on the Hexagon DSP- digital signal processor. DSPs are dedicated programmable chips designed specifically for sensor and image processing. Smartphones mostly use the DSP to enhance the photos taken by their cameras- Quest uses it for VR tracking.

Tagged with: facebook, insight, oculus, Oculus Go, oculus insight, Oculus Quest

The post Oculus Quest ‘Significantly Faster’ Than Oculus Go, 6DoF Tracking ‘Doesn’t Affect’ Performance appeared first on UploadVR.

September 26, 2018

Oculus Quest Features Multi-Room Guardian and Arena-Scale Tracking Using Oculus Insight

While the announcement of Oculus Quest was certainly eye catching, with the standalone headset set to launch in Spring 2019 for $399 USD, it was the tech making it all possible that made it all the more impressive. That technology was Oculus Insight, the inside-out tracking solution which sets it apart for Oculus Rift and Oculus Go.

Oculus Insight

Oculus Insight uses four ultra wide-angle external sensors and computer vision algorithms to track and map the area in real time whilst also tracking the Oculus Touch controllers. The technology looks for edges, corners, and any other distinct feature in the environment to generate a point map of the users location, creating an estimate every millisecond of where their head is.

And thanks to the way Oculus Insight works it means that Oculus Quest has ‘arena-scale’ tracking, allowing for gameplay in much larger environments than at home. During Oculus Connect 5 this week this function is being put to the test in a 4,000 sq ft arena with guests able to play Dead & Buried Arena.

All this tracking also works for co-location multiplayer, as Oculus explains: “Oculus Insight can build and store a “spatial map” of any environment. It can retrieve this map and use camera data to “see” where it’s located. In the current demo, we created a master map of the entire space and made the map accessible to multiple devices in the same room over the network. That way, the devices know where they are in relation to one another, allowing them to co-locate each other.

The system also employs Oculus Rift’s Guardian tech, which stops users walking into walls, or punching furniture, bringing up a virtual wall to notify them. Naturally this has been taken further to offer a ‘Multi-Room Guardian’ which remembers the layout of rooms for quick and easy use when taking round a friends then coming back home. What Oculus hasn’t quite detailed is how the scanning works, whether its completely automatic or if its manually operated.

How well this all works is another matter. With four sensors the controller tracking should be better than rivals Vive Focus and Lenovo Mirage Solo which only use two forward facing cameras. When VRFocus finds out we’ll let you know.