Facebook’s Prototype Photoreal Avatars No Longer Require Face Tracking

Facebook Reality Labs’ prototype photorealistic avatars can now work without face tracking cameras, bringing the technology’s potential deployment closer than ever.

Facebook, which owns the Oculus brand of VR products (which, as of yesterday, fall under the Facebook Reality Labs label), first showed off work on ‘Codec Avatars’ back in March 2019. Powered by machine learning, the avatars are generated using a specialized capture rig with 132 cameras. Once generated, they can be driven by a prototype VR headset with three cameras; facing the left eye, right eye, and mouth.

Even if codec avatars can in future be generated with widely accessible hardware, no consumer VR headset today has the necessary cameras facing the mouth and upper face. Adding these cameras to headsets would increase cost- and be dead weight in offline experiences.

That’s why this latest incarnation of Codec Avatars does away with the need for dedicated face cameras. The new neural network fuses a headset’s eye-tracking data with the microphone audio feed to infer the likely facial expression.

Unlike facial expression cameras, eye tracking could be useful for much more than avatars. It could enable prioritizing resolution based on gaze (known as foveated rendering) as well as precise optical calibration, and even variable focus optics with realistic blur. An Oculus headset sporting eye-tracking seems like a question of when, not if.

So does this approach really work? Can a neural network really infer facial expression from only eye-tracking directions and microphone audio? Based on the video examples provided- it looks like the answer is yes.

The researchers say the network can even pick up the audio cues for subtle actions like wetting your lips with your tongue. It’s noted that picking up such cues would require the headset to have a high-quality microphone, though.

There is of course a major catch. Training the model requires a multi-camera 3D capture setup with 45 minutes of unique data for each test user. When first shown in 2018, Codec Avatars were described by Facebook as “years away”. While this new research makes the hardware needed to drive avatars more practical, it still doesn’t solve the core issues of generating the avatar in the first place.

If such problems can be solved, the technology could have tremendous implications. For most, telepresence today is limited to grids of webcams on a 2D monitor. The ability to see photorealistic representations of others in true scale, fully tracked from real motion, with the ability to make eye contact, could fundamentally change the need for face to face interaction.

The post Facebook’s Prototype Photoreal Avatars No Longer Require Face Tracking appeared first on UploadVR.

Facebook’s VR/AR Division Renamed Facebook Reality Labs, Oculus Branding Remains

Facebook has a new name for its entire VR and AR team – Facebook Reality Labs.

Okay, a sort of new name.

Previously, Facebook Reality Labs referred to the R&D division of the company’s VR and AR work, which itself started out as Oculus Research. As of today, though, that refers to every element of the social networking giant’s efforts in these fields, from consumer Oculus headsets to its SparkAR smartphone offerings and beyond. The original Reality Labs will continue on under Chief Scientist Michael Abrash as FRL Research.

A Facebook spokesperson confirmed that the company will still retain and use the Oculus branding, so expect future hardware like the heavily-rumored new Oculus Quest to still carry that label. Whats more, the company confirmed there have been no leadership or executive changes made in the transition.

In a blog post announcing the news — along with a date for the newly-named Facebook Connect developer event — Facebook VP of VR and AR Andrew Bosworth reasoned that the company had previously “lacked a unified brand identity from which to tell the story of how we’re building the future of VR and AR.”

“This is our seventh year hosting our annual AR/VR conference, and over the years, Connect has grown to include so much more than Oculus, with research updates and product news from Spark AR and more,” Bosworth wrote. “Moving forward, our annual AR/VR event will be called Facebook Connect to better reflect its broader scope, and we look forward to sharing even more news that represents the work happening across the entire Facebook Reality Labs team.”

Facebook Doubles Down On Facebook

The news comes a week after Facebook announced that, from October 2020, all first-time sign-ins to Oculus headsets will require a Facebook account. Existing Oculus accounts will be supported up until 2023, when users will lose access to unspecified features.

Facebook Connect takes place on September 16, and the company says it will share more on its new name and plans for the future during the show. We’ll be bringing you extensive coverage over the course of the event.

The post Facebook’s VR/AR Division Renamed Facebook Reality Labs, Oculus Branding Remains appeared first on UploadVR.

Facebook Researchers Develop Bleeding-edge Facial Reconstruction Tech So You Can Make Goofy Faces in VR

Facebook Reality Labs, the company’s R&D division, has been leading the charge on making virtual reality avatars realistic enough to cross the dreaded ‘uncanney valley’. New research from the group aims to support novel facial expressions so that your friends will accurately see your silly faces VR.

Most avatars used in virtual reality today are more cartoon than human, largely as a way to avoid the ‘uncanny valley’ problem—where more ‘realistic’ avatars become increasingly visually off-putting as they get near, but not near enough, to how a human actually looks and moves.

The Predecessor: Codec Avatars

The ‘Codec Avatar’ project at Facebook Reality Labs aims to cross the uncanny valley by using a combination of machine learning and computer vision to create hyper-realistic representations of users. By training the system to understand what a person’s face looks like and then tasking it with recreating that look based on inputs from cameras inside of a VR headset, the project has demonstrated some truly impressive results.

Recreating typical facial poses with enough accuracy to be convincing is already a challenge, but then there’s a myriad of edge-cases to deal with, any of which can throw the whole system off and dive the avatar right back into the uncanny valley.

The big challenge, Facebook researchers say, is that it’s “impractical to have a uniform sample of all possible [facial] expressions” because there’s simply so many different ways that one can contort their face. Ultimately this means there’s a gap in the system’s example data, leaving it confused when it sees something new.

The Successor: Modular Codec Avatars

Image courtesy Facebook Reality Labs

Researchers Hang Chu, Shugao Ma, Fernando De la Torre, Sanja Fidler, and Yaser Sheikh from the University of Toronto, Vector Institute, and Facebook Reality Labs, propose a solution in a newly published research paper titled Expressive Telepresence via Modular Codec Avatars.

While the original Codec Avatar system looks to match an entire facial expression from its dataset to the input that it sees, the Modular Codec Avatar system divides the task by individual facial features—like each eye and the mouth—allowing it to synthesize the most accurate pose by fusing the best match from several different poses in its knowledge.

In Modular Codec Avatars, a modular encoder first extracts information inside each single headset-mounted camera view. This is followed by a modular synthesizer that estimates a full face expression along with its blending weights from the information extracted within the same modular branch. Finally, multiple estimated 3D faces are aggregated from different modules and blended together to form the final face output.

The goal is to improve the range of expressions that can be accurately represented without needing to feed the system more training data. You could say that the Modular Codec Avatar system is designed to be better at making inferences about what a face should look like compared to the original Codec Avatar system which relied more on direct comparison.

The Challenge of Representing Goofy Faces

One of the major benefits of this approach is improving the system’s ability to recreate novel facial expressions which it wasn’t trained against in the first place—like when people intentionally contort their faces in ways which are funny specifically because people don’t normally make such faces. The researchers called out this particular benefit in their paper, saying that “making funny expressions is part of social interaction. The Modular Codec Avatar model can naturally better facilitate this task due to stronger expressiveness.”

They tested this by making ‘artificial’ funny faces by randomly shuffling face features from completely different poses (ie: left eye from {pose A}, right eye from {pose B}, and mouth from {pose C}) and looked to see if the system could produce realistic results given the unexpectedly dissimilar feature input.

Image courtesy Facebook Reality Labs

“It can be seen [in the figure above] that Modular Codec Avatars produce natural flexible expressions, even though such expressions have never been seen holistically in the training set,” the researchers say.

As the ultimate challenge for this aspect of the system, I’d love to see its attempt at recreating the incredible facial contortions of Jim Carrey.

Eye Amplification

Beyond making funny faces, the researchers found that the Modular Codec Avatar system can also improve facial realism by negating the difference in eye-pose that is inherent with wearing a headset.

In practical VR telepresence, we observe users often do not open their eyes to the full natural extend. This maybe due to muscle pressure from the headset wearing, and display light sources near the eyes. We introduce an eye amplification control knob to address this issue.

This allows the system to subtly modify the eyes to be closer to how they would actually look if the user wasn’t wearing a headset.

Image courtesy Facebook Reality Labs

– – – – –

While the idea of recreating faces by fusing together features from disparate pieces of example data isn’t itself entirely new, the researchers say that “instead of using linear or shallow features on the 3D mesh [like prior methods], our modules take place in latent spaces learned by deep neural networks. This enables capturing of complex non-linear effects, and producing facial animation with a new level of realism.”

The approach is also an effort to make this kind of avatar representation a bit more practical. The training data necessary to achieve good results with Codec Avatars requires first capturing the real user’s face across many complex facial poses. Modular Codec Avatars achieve similar results with greater expressiveness on less training data.

SEE ALSO
Facebook Reality Labs Says Varifocal Optics Are "almost ready for primetime," Details HDR Research

It’ll still be a while before anyone without access to a face-scanning lightstage will be able to be represented so accurately in VR, but with continued progress it seems plausible that one day users could capture their own face model quickly and easily through a smartphone app and then upload it as the basis for an avatar which crosses the uncanny valley.

The post Facebook Researchers Develop Bleeding-edge Facial Reconstruction Tech So You Can Make Goofy Faces in VR appeared first on Road to VR.

Facebook’s Display Research Lead: Varifocal Half-Dome 3 ‘Almost Ready For Prime Time’

Facebook’s Director of Display Systems Research Douglas Lanman revealed the Half-Dome 3 prototype shown late last year is further along the development cycle than some other research projects.

It’s something that is not quite a publication. It’s beyond that. And this thing is more like level five. It’s almost ready for primetime

Stated at an SPIE talk given earlier in the year, Lanman was referencing NASA’s Technology Readiness Levels (TRLs), adapted for tech products:

Lanman works at Facebook Reality Labs, the company’s VR/AR research division. He explained that a “real-deal scientist” works at TR1, while his team usually works between TR2 and TR4.

Startups, Lanman elaborated, typically start at TR6 or TR7, whereas shipping a consumer product at scale involves TR7, TR8, and TR9.

“At some point, you walk into an electronics store and you see level nine. And you start to wonder, am I just going to do this cycle of level 2 through 4 forever? How do I actually change the world rather than just have good ideas?”

Lanman explained that what makes him proud of Half-Dome 3 in particular is that it’s at “Level 5 or beyond“. More specifically, it’s “almost ready for prime time“.

https://www.youtube.com/watch?v=YWA4gVibKJE

All current VR headsets, outside of lab prototypes, have fixed focus lenses. Your brain gets a different image for each eye, but the images are permanently focused at the same distance (usually a few meters away).

This makes VR feel less real since you’re missing one of the real world’s depth cues. It also introduces the Vergence-Accommodation conflict, a leading cause of eye strain in head mounted displays.

The original Half-Dome prototype, shown off at F8 in May 2018, aimed to deliver near-silent variable focus (varifocal) experience. It also increased the field of view significantly (from around 100 degrees to 140 degrees wide) while maintaining roughly the same form factor.

Half-Dome 1 used physical actuators to move the position of each lens relative to the displays. A product that relies on constant movement like that for hours on end seems like a reliability nightmare waiting to happen, which is where Half-Dome 3 comes in.

Half-Dome 1, Half-Dome 2, and Half-Dome 3

Half-Dome 3 was first shown by Facebook’s Chief Scientist Michael Abrash (Lanman’s boss) in October at Oculus Connect 6. The field of view is “20% wider than Quest”, but no longer 140 degrees.

But Half-Dome 3 brought the technology to a considerably smaller design, with no moving parts. This is achieved by using a stack of liquid crystal lens layers. Applying a voltage to each lens layer changes its focal length, so each unique combination of on and off results in a different plane of focus.

Half Dome 3

At another talk given this year, Lanman expressed just how important he feels variable focus and depth cues are for virtual reality. Given this goal, the number of varifocal prototypes we’ve seen, and Half-Dome 3’s compact form factor and lack of moving parts, could varifocal technology now be out of the research labs- on the beginning of the path to productization?

To be clear, there’s no indication the recent ‘Oculus Quest 2’ leaks have any such technology.

The post Facebook’s Display Research Lead: Varifocal Half-Dome 3 ‘Almost Ready For Prime Time’ appeared first on UploadVR.

Facebook Reality Labs Says Varifocal Optics Are “almost ready for primetime,” Details HDR Research

Facebook Reality Labs, the company’s R&D department, previously revealed its ‘Half Dome’ prototype headsets which demonstrated functional varifocal optics small enough for a consumer VR headset. At a conference earlier this year, the Lab’s Director of Display Systems Research said the latest system is “almost ready for primetime,” and also detailed the Lab’s research into HDR (high-dynamic range) and pupil-steering displays for XR headsets.

Technological Readiness

Douglas Lanman, Director of Display Systems Research at Facebook Reality Labs, gave a keynote presentation at the SPIE AR VR MR 2020 conference earlier this year. In his presentation, which was recently posted online, Lanman introduced a scale of ‘technological readiness’:

  1. Basic Research – Basic principles observed
  2. Technology Formulation – Technology concept and application formulated
  3. Initial Validation – Experimental proof of concept
  4. Small Scale Prototype – Technology validated in lab
  5. Large Scale Prototype – Technology initially validated in intended environment
  6. Prototype System – Technology robustly demonstrated in intended environment
  7. Demonstration System – System prototype demonstrated in operational environment
  8. First of a Kind Commercial System – System complete and qualified
  9. Generally Available Commercial System – Actual system proven in operational environment

While the scale was originally used by NASA, Lanman likened it to the journey that research (level one) takes all the way through widespread availability of a product (level nine).

Lanman explained that the work of researchers tends to focus on levels 2 through 4, at which point the research is published and the researchers move onto another project, but rarely see their work reach the higher levels on the scale.

The Display Systems Research team at Facebook Reality Labs is unique, Lanman said, because the group has the capacity to work between levels 1 to 6, taking research all the way from “first principles” through to polished prototypes; much closer to a finished product than researchers typically see their work carried.

“So what’s really unique about this Display Systems Research team is that we’re not quite a startup, we’re not quite a major company, and we’re not quite academics. We really play from the absolute fundamental vision science through very polished prototypes—more polished than you’d see from most startups—to try to do one thing, which is [have a genuine impact on future products].”

Half Dome “almost ready for primetime”

Image courtesy Oculus

The team created a series of prototypes dubbed ‘Half Dome’, which employ varifocal displays that allow the headset to correctly support both vergence and accommodation together—something no consumer VR headset does to date.

Half Dome 3 is the latest of the prototypes which Facebook Reality Labs has spoken about publicly. Instead of relying on a mechanically-driven varifocal display like the prior prototypes, Half Dome 3 implemented a static varifocal display which uses a series of liquid crystal lenses that allow the headset’s optics to change between 64 discrete focal planes. Half Dome 3 also employs ‘folded optics’ which significantly reduce the size of the display module.

Size comparison: Half Dome 3 static varifocal display module with folded optics (left), Half Dome mechanical varifocal display module (right) | Image courtesy Oculus

The first Half Dome prototype was revealed back in 2018. At the time, Oculus said that customers shouldn’t “expect to see these technologies in a product anytime soon.” A year later the Half Dome 3 prototype was revealed, but Oculus was still tight lipped about whether or not the tech would find its way into a headset.

While it’s still not clear how close Oculus is to productizing a varifocal or folded optic display, Lanman ranked both Half Dome and Half Dome 3 on the technology readiness scale that he introduced earlier in this talk.

He placed Half Dome, the mechanically-driven varifocal headset, at level 6 (Prototype System), and Half Dome 3, the static varifocal headset with folded optics, at level 5 (Large Scale Prototype). “It’s almost ready for primetime,” he said of Half Dome 3.

Continue on Page 2: A Step Toward “creating the world’s first HDR headset” »

The post Facebook Reality Labs Says Varifocal Optics Are “almost ready for primetime,” Details HDR Research appeared first on Road to VR.

Facebook Reality Labs Shows Method for Expanding Field of View of Holographic Displays

Researchers from Facebook’s R&D department, Facebook Reality Labs, and the University of California, Berkeley have published new research which demonstrates a method for expanding the field-of-view of holographic displays.

In the paper, titled High Resolution Étendue Expansion for Holographic Displays, researchers Grace Kuo, Laura Waller, Ren Ng, and Andrew Maimone explain that when it comes to holographic displays there’s an intrinsic inverse link between a display’s field-of-view and its eye-box (the eye-box is the area in which the image from a display can be seen). If you want a larger eye-box, you get a smaller field-of-view. And if you want a larger field of view, you get a smaller eye-box.

If the eye-box is too small, even the movement from the rotation of your eye would make the image invisible because your pupil would leave the eye-box when looking any direction but forward. A large eye-box is necessary not only to keep the image visible during eye movement, but also to compensate for subtle differences in headset fit from one session to the next.

The researchers explain that a traditional holographic display with a 120° horizontal field-of-view would have an eye-box of just 1.05mm—far too small for practical use in a headset. On the other hand, a holographic display with a 10mm eye-box would have a horizontal field-of-view of just 12.7°.

If you want to satisfy both a 120° field-of-view and a 10mm eye-box, the researchers say, you’d need a holographic display with a resolution of 32,500 × 32,500. That’s not only impractical because such a display doesn’t exist, but even if it did, rendering that many pixels for real-time applications would be impossible with today’s hardware.

So, the researchers propose a different solution, which is decouple the link between field-of-view and eye-box in a holographic display. The method proposes the use of a scattering element placed in front of the display which scatters the light to expand its cone of propagation (also known as étendue). Doing so allows the field-of-view and eye-box characteristics to be adjusted independently.

But there’s a problem of crouse. If you put a scattering element in front of a display, how do you form a coherent image from the scattered light? The researchers have developed an algorithm which pre-compensates for the scattering element, such that the ‘scattered’ light actually forms a proper image after being scattered.

At a high level, it’s very similar to the approach that existing headsets use to handle color separation (chromatic aberration) as light passes through the lenses—rendered frames pre-separate colors so that the lens ends up bending the colors back into the correct place.

Here the orange box represents the field of view of a normal holographic display while the full frame shows the expanded field of view | Image courtesy Facebook Reality Labs

The researchers used optical simulations to hone their algorithm and then built a benchtop prototype of their proposed pipeline to experimentally demonstrate the method for expanding the field of view of a holographic display.

Although the researchers believe their work “demonstrates progress toward more practical holographic displays,” they also say that there is “additional work to be done to achieve a full-color display with high resolution, complete focal depth cues, and a sunglasses-like form factor.”

Toward the end of the paper they identify miniaturization, compute time, and perceptual effects among the challenges needed to be addressed by further research.

The paper also hints at potential future projects for the team, which may be to attempt to combine this method with prior work from one of the paper’s researchers, Andrew Maimone.

“The prototype presented in this work is intended as a proof-of-concept; the final design is ideally a wearable display with a sunglasses-like form factor. Starting with the design presented by Maimone et al. [2017], which had promising form factor and FoV but very limited eyebox, we propose integrating our scattering mask into the holographic optical element that acts as an image combiner.”

Image courtesy Facebook Reality Labs

If you read our article last month on Facebook’s holographic folded optics, you may be wondering how these projects differ.

The holographic folded optics project makes use of a holographic lens to focus light, but not a holographic display to generate the image in the first place. That project also employs folded optics to significantly reduce the size of such a display.

On the other hand, the research outlined in this article deals with making actual holographic displays more practical by showing that a large field-of-view and large eye-box are not mutually exclusive in a holographic display.

The post Facebook Reality Labs Shows Method for Expanding Field of View of Holographic Displays appeared first on Road to VR.

Watch Facebook’s Display Research Lead Outline Future VR Headsets

Video from a talk given by the director of Display Systems Research at Facebook Reality Labs offers an eye-opening overview of the future virtual reality headset optical designs.

Video of Douglas Lanman’s talk was posted this week, nearly half a year after he delivered it at the Electronic Imaging symposium in January in San Francisco. That places the talk a few months after the head of Facebook’s VR and AR research, Michael Abrash, delivered an annual update at Oculus Connect, and right before in-person events worldwide were cancelled due to the worsening COVID-19 pandemic.

Lanman previously worked at NVIDIA where he developed ideas related to near-eye light fields. Now with more than six years at Facebook he occasionally delivers talks that encourage other researchers to take employment at FRL while divulging a range of advances to VR visuals being explored.

Much of this research has been explained before, including Half Dome varifocal prototypes which are aimed at offering greater visual comfort in VR by mitigating the vergence-accommodation conflict plaguing practically all current consumer designs. With this talk, however, Lanman seemed more convinced really good eye tracking is going to be an important part of future advances.

If you don’t have time for the full presentation, you can watch a clipped down version here here:

Still, the one hour and 10 minute talk offers a relatively concise and scientifically accurate overview of the cutting edge of VR headset optical research. We recommend giving it a watch if you’d like to see the kinds of ideas, and challenges, one of the best-funded VR research groups on the planet are grappling with to improve future headset designs.

The post Watch Facebook’s Display Research Lead Outline Future VR Headsets appeared first on UploadVR.

‘Neural Supersampling’ Could Give Future Oculus Quests Console-Quality Graphics

A new neural network developed by Facebook’s VR/AR research division could enable console-quality graphics on future standalone headsets.

This ‘Neural Supersampling’ algorithm can take a low resolution rendered frame and upscale it 16x. That means, for example, a future headset could theoretically drive dual 2K panels by only rendering 540×540 per eye – without even requiring eye tracking.

Being able to render at lower resolution means more GPU power is free to run detailed shaders and advanced effects, which could bridge the gap from mobile to console VR. To be clear, this can’t turn a mobile chip into a PlayStation 5, but it should bridge the gap somewhat.

“AI upscaling” algorithms have become popular in the last few years, with some websites even letting users upload any image on their PC or phone to be upscaled. Given enough training data, they can produce a significantly more detailed output than traditional upscaling. While just a few years ago “Zoom and Enhance” was used to mock those falsely believing computers could do this, machine learning has made it reality. The algorithm is technically only “hallucinating” what it expects the missing detail should look like, but in many cases there is little practical difference.

Facebook claims its neural network is state of the art, outperforming all other similar algorithms- the reason it’s able to achieve 16x upscaling. What makes this possible is the inherent knowledge of the depth of each object in the scene- it would not be anywhere near as effective with flat images.

In the provided example images, Facebook’s algorithm seems to have reached the point where it can reconstruct even fine details like line or mesh patterns.

Back in March, Facebook published a somewhat similar paper. It also described the idea of freeing up GPU power by using neural upsampling. But that wasn’t actually what the paper was about. The researchers’ direct goal was to figure a “framework” for running machine learning algorithms in real time within the current rendering pipeline (with low latency), which they achieved. Combining that framework with this neural network could make this technology practical.

“As AR/VR displays reach toward higher resolutions, faster frame rates, and enhanced photorealism, neural supersampling methods may be key for reproducing sharp details by inferring them from scene data, rather than directly rendering them. This work points toward a future for high-resolution VR that isn’t just about the displays, but also the algorithms required to practically drive them,” a blog post by Lei Xiao explains.

For now, this is all just research and you can read the paper here. What’s stopping this from being a software update for your Oculus Quest tomorrow? The neural network itself takes time to process. The current version runs at 40 frames per second at Quest resolution on the $3000 NVIDIA Titan V.

But in machine learning, optimization comes second- and happens to an extreme degree. Just three years ago, the algorithm Google Assistant uses to speak realistically also required a $3000 GPU. Today it runs locally on several smartphones.

The researchers “believe the method can be significantly faster with further network optimization, hardware acceleration and professional grade engineering”. Hardware acceleration for machine learning tasks is available on the Snapdragon chips- Qualcomm claims its XR2 has 11x the ML performance as Quest.

If optimization and models built for mobile system-on-a-chip neural processing units don’t work out, another approach is a custom chip designed for the task. This approach was taken on the $50 Nest Mini speaker (the cheapest device with local Google Assistant). Facebook is reportedly working with Samsung on custom chips for AR glasses, but there’s no indication of the same happening for VR- at least not yet.

Facebook classes this kind of approach “neural rendering”. Just like neural photography helped bridged the gap between smartphone and digital single lens reflex cameras, Facebook hopes it can one day push a little more power out of mobile chips than anyone might have expected.

The post ‘Neural Supersampling’ Could Give Future Oculus Quests Console-Quality Graphics appeared first on UploadVR.

Facebook’s Prototype Photoreal Avatars Now Have Realistic Eyes

Researchers at Facebook figured out how to add natural looking eyes to their photorealistic avatar research.

Facebook, which owns the Oculus brand of VR products, first showed off this ‘Codec Avatars’ project back in March 2019. The avatars are generated using a specialized capture rig with 132 cameras. Once generated, they can be driven by a prototype VR headset with three cameras; facing the left eye, right eye, and mouth. All of this is achieved with machine learning.

While the graphics and face tracking of these avatars are impressive, the eyes tended to have an uncanny feeling, with odd distortions and gaze directions that didn’t make sense.

In a paper titled ‘The Eyes Have It: An Integrated Eye and Face Model for Photorealistic
Facial Animation
‘, the researchers present a solution to this problem. Part of the previous pipeline involved a “style transfer” neural network. If you’ve used a smartphone AR filter that makes the world look like a painting, you already know what that is.

But while this transfer was previously done at the image stage, it’s now done on the resulting texture itself. The eye direction is explicitly taken from the eye tracking system, rather than being estimated by the algorithm.

The result, based on the images and videos Facebook provided, is significantly more natural looking eyes. The researchers claim eye contact is critical to achieving social presence, and their system can now handle this- a feature you won’t get in a Zoom call.

Our goal is to build a system to enable virtual telepresence, using photorealistic avatars, at scale, with a level of fidelity sufficient to achieve eye-contact

Don’t get too excited just yet- this kind of technology won’t be on your head any time soon. When presenting codec avatars, Facebook warned the technology was still “years away” for consumer products.

When it can be realized, however, such a technology has tremendous potential. For most, telepresence today is still limited to grids of webcams on a 2D monitor. The ability to see photorealistic representations of others in true scale, fully tracked from real motion, with the ability to make eye contact, could fundamentally change the need for face to face interaction.

The post Facebook’s Prototype Photoreal Avatars Now Have Realistic Eyes appeared first on UploadVR.

Facebook’s Future VR Headsets Could Feature Holographic Optics

Facebook Holographic-optics

One of the big hindrances to widespread virtual reality (VR) adoption is the fact that headsets are bulky devices, so a lot of people simply don’t want them on their face. Companies like Facebook are spending enormous amounts trying to improve the form factor of headsets and recently the tech giant unveiled a new research project which uses holographic optics to create a ‘VR Glasses’ device.

Facebook Holographic-optics

Current VR technology uses small LCD or OLED displays alongside lenses to focus the light into your eyes. While this is a proven method, this does require them to be a certain distance away from each other to work, enabling the optics to actually fold the light properly. The knock-on effect is that a VR headset has to be deep to fit all of this inside.

Researchers Andrew Maimone and Junren Wang from Facebook Reality Labs (FRL) will be presenting their new research at SIGGRAPH’s virtual conference this August, a system which uses holographic optics to make a device far thinner and lighter than current models, aiming for that coveted sunglasses-like VR hardware.

Just a proof-of-concept research device at the moment, it uses polarization-based optical folding to mimic that conventional distance but in a form factor that’s less than 9mm in depth. At the same time, the team claim that the field of view (FoV) is comparable to existing VR devices.

Facebook Holographic-optics

This is achieved by using flat films as optics and laser illumination. “Holographic optics compel the use of laser light sources, which are more difficult to integrate but provide a much richer set of colours than the LEDs common in nearly all of today’s VR headsets,” FRL notes in a blog post. Presently the research device outputs in monochrome (as seen in the above-left image) but the team do have a larger full-colour benchtop prototype working (right image). The goal now is to bring full colour to the smaller unit.

Obviously this is still an early research project so there are plenty of other variables to solve such as a power source and processing, would these be on-board or in a separate device like the Nreal Light? Ideally, it would be an all-in-one form factor yet those products are still years away.

VRFocus is still waiting to see if anything comes from Michael Abrash’s Half Dome prototypes plus there’s the smaller Oculus Quest Facebook is reportedly working on. For further updates on Facebook’s VR research, keep reading VRFocus.