Meta reportedly canceled a near-term successor to Quest Pro, with the next model now “way out in the future”.
Quest Pro has received mixed at-best reviews, with criticism falling on its high price, underwhelming resolution, hefty weight, outdated processor, and lack of automatic room sensing for headline mixed reality features.
The Verge’s Alex Heath reports Meta CTO Andrew Bosworth told staff a planned second generation of Quest Pro is canned. Instead, Meta’s VP of VR Mark Rabkin reportedly told staff the company plans an advanced headset “way out in the future” featuring Codec Avatars, the company’s long-running research project to achieve photoreal spatial telepresence via face tracking sensors on VR headsets.
The Codec Avatars project was first revealed in 2019. Last year it’s lead Yaser Sheikh said that when the project started it was “ten miracles away”, and he now believed it was “five miracles away”.
“We want to make it higher resolution for work use and really nail work, text and things like that,” Rabkin reportedly said of the “way out” headset.
In November, a South Korean news outlet reported Meta executives met with Samsung and LG’s display divisions about supplying OLED and MicroLED microdisplays for future VR and AR devices. Microdisplays have significantly higher pixel densities and thus can enable higher resolution in headsets more compact than today’s.
Last year The Information reported the near-term Quest Pro successor had been slated for 2024, suggesting this “way out in the future” headset won’t arrive until 2025 at the absolute earliest.
UPDATE March 2: Heath confirmed the cancelation of the near-term Quest Pro successor in his weekly Command Line newsletter. This article has been updated to reflect this.
The capture rig used to generate Codec Avatars until now
Previously, generating an individual Codec Avatar required a specialized capture rig called MUGSY with 171 high resolution cameras. But Meta’s latest research gets rid of this requirement, generating an avatar with a scan from a smartphone with a front facing depth sensor, such as any iPhone with FaceID. You first pan the phone around your neutral face, then again while copying a series of 65 facial expressions.
This scanning process takes 3 and a half minutes on average, the researchers claim – though actually generating the avatar (in full detail) then takes six hours on a machine with four high end GPUs. If deployed in a product, this step would likely happen on cloud GPUs, not the user’s device.
So how is it possible for what once required more than 100 cameras to now require only a phone? The trick is in the use of a Universal Prior Model “hypernetwork” – a neural network which generates the weights for another neural network – in this case the person-specific Codec Avatar. The researchers trained this UPM hypernetwork by scanning in the faces of 255 diverse individuals using an advanced capture rig, similar to MUGSY but with 90 cameras.
While other researchers have already demonstrated avatar generation from a smartphone scan, Meta claims the quality of its result is state of the art. However, the current system cannot handle glasses or long hair, and is limited to the head, not the rest of the body.
Of course, Meta still has a long way to go to reach this kind of fidelity in shipping products. Meta Avatars today have a basic cartoony art style. Their realism has actually decreased over time, likely to better suit larger groups with complex environments in apps like Horizon Worlds on Quest 2’s mobile processor. Codec Avatars may, however, end up as a separate option, rather than a direct update to the cartoon avatars of today. In his interview with Lex Fridman, CEO Mark Zuckerberg described a future where you might use an “expressionist” avatar in casual games and a “realistic” avatar in work meetings.
In AprilYaser Sheikh, who leads the Codec Avatars team, said it’s impossible to predict how far away it is from actually shipping. He did say that when the project started it was “ten miracles away” and he now believes it’s “five miracles away”.
Meta researchers built a custom chip to make photoreal avatars possible on standalone headsets.
Facebook first showed off work on ‘Codec Avatars’ back in March 2019. Powered by multiple neural networks, the avatars are generated using a specialized capture rig with 171 cameras. Once generated, they can be driven in real time by a prototype VR headset with five cameras; two internal viewing each eye and three external viewing the lower face. Since then the researchers have showed off several evolutions of the system, such as more realistic eyes, a version only requiring eye tracking and microphone input, and most recently a 2.0 version that approaches complete realism.
But driving photorealistic avatars on the mobile chipsets used in standalone headsets like Quest is no easy feat. While Meta’s newest & most efficient model can only render (decode) five avatars at 50 frames per second on a Quest 2, this doesn’t include the other side of the equation – converting (encoding) the eye tracking and face tracking inputs to a codec avatar face state.
In a paper published for the 2022 IEEE Custom Integrated Circuits Conference, Meta researchers present a prototype of a neural network accelerator chip “that achieves energy efficient performance for running eye gaze extraction of the Codec Avatar model”.
Transferring data between components on mobile devices requires significant energy. The custom chip is located near the eye tracking camera inputs on the main board to minimize this, and handles converting these camera inputs into a Codec Avatar state. The researchers point out this could also have benefits for privacy and security, as the general purpose processors wouldn’t even have access to the raw camera data.
To be clear, this chip is just an experiment. It only handles one small part of the Codec Avatars workload, it’s not a general solution for making the system viable on near-term standalone headsets. The regular chipset (eg. Snapdragon XR2) is still handling all other tasks.
What this demonstrates is the viability of using custom silicon to solve problems like avatars. The idea could theoretically in future be dramatically expanded to handle more VR-specific tasks.
Meta’s researchers are approaching complete realism with Codec Avatars 2.0, their prototype VR avatars using advanced machine learning techniques.
Facebook first showed off work on ‘Codec Avatars’ back in March 2019. Powered by multiple neural networks, the avatars are generated using a specialized capture rig with 171 cameras. Once generated, they can be driven in real time by a prototype VR headset with four cameras; two internal viewing each eye and three external viewing the lower face. Since then the researchers have showed off several evolutions of the system, such as more realistic eyes and a version only requiring eye tracking and microphone input.
At MIT’s Virtual Beings & Being Virtual workshop in April Yaser Sheikh, who leads the Codec Avatars team, showed a video of the latest version of the project, described as “Codec Avatars 2.0”:
“I would say a grand challenge of the next decade is to see if we can enable remote interactions that are indistinguishable from in person interactions“, Sheikh remarked.
In a paper published last year, Sheikh and his colleagues claim their newest models are smaller and more efficient than their past research, with the neural network now computing only the pixels visible to the headset. With this advancement, a Quest 2 headset is apparently able to render five avatars in realtime in the same (likely empty) scene.
Still, the company seems to have a long way to go to reach this kind of fidelity in shipping products. Meta Avatars today have a basic cartoony art style. Their realism has actually decreased over time, likely to better suit larger groups with complex environments in apps like Horizon Worlds on Quest 2’s mobile processor.
Codec Avatars may however end up as a separate option, rather than a direct update to the cartoon avatars of today. In his interview with Lex Fridman, CEO Mark Zuckerberg described a future where you might use an “expressionist” avatar in casual games and a “realistic” avatar in work meetings.
During the workshop Sheikh noted that it’s impossible to predict how far away Codec Avatars is from actually shipping. He did however say that when the project started it was “ten miracles away”, and he now believes it’s “five miracles away”.
Researchers at Meta Reality Labs have created a prototype VR headset with a custom-built accelerator chip specially designed to handle AI processing to make it possible to render the company’s photorealistic Codec Avatars on a standalone headset.
Long before the company changed its name, Meta has been working on its Codec Avatars project which aims to make nearly photorealistic avatars in VR a reality. Using a combination of on-device sensors—like eye-tracking and mouth-tracking—and AI processing, the system animates a detailed recreation of the user in a realistic way, in real-time.
Or at least that’s how it works when you’ve got high-end PC hardware.
Early versions of the company’s Codec Avatars research were backed by the power of an NVIDIA Titan X GPU, which monstrously dwarfs the power available in something like Meta’s latest Quest 2 headset.
But the company has moved on to figuring out how to make Codec Avatars possible on low-powered standalone headsets, as evidenced by a paper published alongside last month’s 2022 IEEE CICC conference. In the paper, Meta reveals it created a custom chip built with a 7nm process to function as an accelerator specifically for Codec Avatars.
Specially Made
Image courtesy Meta Reality Labs
According to the researchers, the chip is far from off the shelf. The group designed it with an essential part of the Codec Avatars processing pipeline in mind—specifically, analyzing the incoming eye-tracking images and generating the data needed for the Codec Avatars model. The chip’s footprint is a mere 1.6mm²
“The test-chip, fabricated in 7nm technology node, features a Neural Network (NN) accelerator consisting of a 1024 Multiply-Accumulate (MAC) array, 2MB on-chip SRAM, and a 32bit RISC-V CPU,” the researchers write.
In turn, they also rebuilt the part of the Codec Avatars AI model to take advantage of the chip’s specific architecture.
“By re-architecting the Convolutional [neural network] based eye gaze extraction model and tailoring it for the hardware, the entire model fits on the chip to mitigate system-level energy and latency cost of off-chip memory accesses,” the Reality Labs researchers write. “By efficiently accelerating the convolution operation at the circuit-level, the presented prototype [chip] achieves 30 frames per second performance with low-power consumption at low form factors.”
The prototype headset is based on Quest 2 | Image courtesy Meta Reality Labs
By accelerating an intensive part of the Codec Avatars workload, the chip not only speeds up the process, but it also reduces the power and heat required. It’s able to do this more efficiently than a general-purpose CPU thanks to the custom design of the chip which then informed the rearchitected software design of the eye-tracking component of Codec Avatars.
But the headset’s general purpose CPU (in this case, Quest 2’s Snapdragon XR2 chip) doesn’t get to take the day off. While the custom chip handles part of the Codec Avatars encoding process, the XR2 manages the decoding process and rendering the actual visuals of the avatar.
Image courtesy Meta Reality Labs
The work must have been quite multidisciplinary, as the paper credits 12 researchers, all from Meta’s Reality Labs: H. Ekin Sumbul, Tony F. Wu, Yuecheng Li, Syed Shakib Sarwar, William Koven, Eli Murphy-Trotzky, Xingxing Cai, Elnaz Ansari, Daniel H. Morris, Huichu Liu, Doyun Kim, and Edith Beigne.
It’s impressive that Meta’s Codec Avatars can run on a standalone headset, even if a specialty chip is required. But one thing we don’t know is how well the visual rendering of the avatars is handled. The underlying scans of the users are highly detailed and may be too complex to render on Quest 2 in full. It’s not clear how much the ‘photorealistic’ part of the Codec Avatars is preserved in this instance, even if all the underlying pieces are there to drive the animations.
– – — – –
The research represents a practical application of the new compute architecture that Reality Lab’s Chief Scientist, Michael Abrash, recently described as a necessary next step for making the sci-fi vision of XR a reality. He says that moving away from highly centralized processing to more distributed processing is critical for the power and performance demands of such headsets.
One can imagine a range of XR-specific functions that could benefit from chips specially designed to accelerate them. Spatial audio, for instance, is desirable in XR across the board for added immersion, but realistic sound simulation is computationally expensive (not to mention power hungry!). Positional-tracking and hand-tracking are a critical part of any XR experience—yet another place where designing the hardware and algorithms together could yield substantial benefits in speed and power.
Fascinated by the cutting edge of XR science? Check out our archives for more breakdowns of interesting research.
In a post on the Facebook blog, VP of Artificial Intelligence Jerome Pesenti outlines that the system would not only be turned off in the coming weeks, but all associated data would also be deleted. This will “result in the deletion of more than a billion people’s individual facial recognition templates. ”
The system is currently used on Facebook to identify and recognize the faces of Facebook users who appear in photos on the social media network, uploaded by themselves or others. This had various uses, including suggested/automatic tagging of people in photos and an automatic alt-text generator, which used AI to form a description for the visually-impaired and identify if their Facebook friends appeared in a photo.
The decision to remove and delete the system is part of “a company-wide move away from this kind of broad identification, and toward narrower forms of personal authentication.” Pesenti says that while there are helpful applications for facial recognition systems, they also “need to be weighed against growing concerns about the use of this technology as a whole.”
While the broad system used across the Facebook platform is being shut down, Meta’s director’s still believe that there will remain “a narrow set of use cases” where facial recognition technology will be appropriate. One of these could be identity verification using an on-device facial recognition system, or potentially for driving some form of Meta’s in-development photorealisitc ‘Codec Avatars’.
Talking about on-device recognition, Pesenti had this to say:
This method of on-device facial recognition, requiring no communication of face data with an external server, is most commonly deployed today in the systems used to unlock smartphones.
We believe this has the potential to enable positive use cases in the future that maintain privacy, control and transparency, and it’s an approach we’ll continue to explore as we consider how our future computing platforms and devices can best serve people’s needs.
This is a very important decision. We must continue having a public discussion about the societal implications of this technology. https://t.co/qs6Xj7TaG9
This slew of recent policy changes likely points to Meta eventually releasing a VR headset that can be unlocked with on-device facial verification and no requirement to connect or link your Facebook profile.
Meta VP of VR/AR and incoming 2022 CTO Andrew Bosworth tweeted (embedded above) that the decision to move away from broad facial recognition on Facebook is “very important” and encouraged continued discussion about the implications of emerging technologies.
He said Facebook is already working on its next VR headsets:
“Because of how hardware gets developed, you kind of need to know what your next three products are going to look like all at the same time. It’s not like software where we’re changing it every couple of weeks. We have product teams spun up now working on the next few generations of virtual reality and what Quest 3 and 4 are gonna look like.“
Zuckerberg revealed one of his personal most-wanted features in upcoming Oculus headsets; eye & face tracking to drive avatars in social experiences:
“One of the things I’m really excited about for future versions is getting eye tracking and face tracking in, because if you’re really excited about social presence you want to make sure the device has all the sensors to really kind of animate realistic avatars so you can communicate well.“
Later in the interview he spoke about some of the difficulties in making this a reality:
“On the VR side, I think the biggest things that we’re very focused on now are: how do you pack more sensors, to create a better social experience, into the device? To run each sensor requires more CPU power and that generates more heat and creates all these different issues.
When I think about where you’re at with VR today, there’s some pretty great games & different experiences butI’d love to get to the point where you have realistic avatars of yourself, and where you can make real authentic eye contact with someone and have real expressions that get reflected on your avatar.
So what do you need for that? Well you need to be able to do eye tracking to do eye contact. You need to have some sense of face tracking or sense of what’s going on with the person’s expressions in order to be able to have those emotions come through naturally.“
VR headsets today with eye tracking or face tracking are geared toward enterprise buyers. Pico’s Neo 2 Eye is the first standalone headset with eye tracking, priced at $900 with a focus on enterprise use cases. HP’s PC-tethered Reverb G2 Omnicept Edition will have eye tracking & face tracking, but is also focused on enterprise use cases with pricing not yet revealed.
Facebook’s Avatar Research
Facebook first showed off its long term research on photoreal ‘Codec Avatars’ back in March 2019.
Powered by machine learning, the avatars are generated using a specialized capture rig with 132 cameras. Once generated, they can be animated in real time by a prototype VR headset with eye tracking & face tracking cameras.
When first presenting codec avatars, Facebook warned the technology was still “years away” for consumer products – shipping this kind of photoreal avatar will require a number of breakthroughs.
Facebook’s Avatars Today
Oculus Quest today has a built-in basic avatar system, Oculus Avatars. It’s used in a few apps like PokerStars VR and Tribe XR DJ, but not much else. Platforms like Bigscreen, VRChat, and Rec Room use their own separate avatar systems.
Current headsets don’t have eye or face tracking, but Oculus Avatars uses a neural network to simulate lip movement, and developers can set up priority-ranked gaze targets to simulate eye movement.
Facebook Avatars seems to take a step backward in artistic realism, but adds a complete torso and simulated arms.
In today’s interview Zuckerberg announced Facebook Avatars will ship this year, saying it will “get more and more realistic over time”, suggesting the company intends to take an iterative approach. Zuckerberg also commented on Facebook’s low cost hardware strategy.
Facebook Reality Labs’ prototype photorealistic avatars can now work without face tracking cameras, bringing the technology’s potential deployment closer than ever.
Facebook, which owns the Oculus brand of VR products (which, as of yesterday, fall under the Facebook Reality Labs label), first showed off work on ‘Codec Avatars’ back in March 2019. Powered by machine learning, the avatars are generated using a specialized capture rig with 132 cameras. Once generated, they can be driven by a prototype VR headset with three cameras; facing the left eye, right eye, and mouth.
Even if codec avatars can in future be generated with widely accessible hardware, no consumer VR headset today has the necessary cameras facing the mouth and upper face. Adding these cameras to headsets would increase cost- and be dead weight in offline experiences.
That’s why this latest incarnation of Codec Avatars does away with the need for dedicated face cameras. The new neural network fuses a headset’s eye-tracking data with the microphone audio feed to infer the likely facial expression.
Unlike facial expression cameras, eye tracking could be useful for much more than avatars. It could enable prioritizing resolution based on gaze (known as foveated rendering) as well as precise optical calibration, and even variable focus optics with realistic blur. An Oculus headset sporting eye-tracking seems like a question of when, not if.
So does this approach really work? Can a neural network really infer facial expression from only eye-tracking directions and microphone audio? Based on the video examples provided- it looks like the answer is yes.
The researchers say the network can even pick up the audio cues for subtle actions like wetting your lips with your tongue. It’s noted that picking up such cues would require the headset to have a high-quality microphone, though.
There is of course a major catch. Training the model requires a multi-camera 3D capture setup with 45 minutes of unique data for each test user. When first shown in 2018, Codec Avatars were described by Facebook as “years away”. While this new research makes the hardware needed to drive avatars more practical, it still doesn’t solve the core issues of generating the avatar in the first place.
If such problems can be solved, the technology could have tremendous implications. For most, telepresence today is limited to grids of webcams on a 2D monitor. The ability to see photorealistic representations of others in true scale, fully tracked from real motion, with the ability to make eye contact, could fundamentally change the need for face to face interaction.
Facebook Reality Labs, the company’s R&D division, has been leading the charge on making virtual reality avatars realistic enough to cross the dreaded ‘uncanney valley’. New research from the group aims to support novel facial expressions so that your friends will accurately see your silly faces VR.
Most avatars used in virtual reality today are more cartoon than human, largely as a way to avoid the ‘uncanny valley’ problem—where more ‘realistic’ avatars become increasingly visually off-putting as they get near, but not near enough, to how a human actually looks and moves.
The Predecessor: Codec Avatars
The ‘Codec Avatar’ project at Facebook Reality Labs aims to cross the uncanny valley by using a combination of machine learning and computer vision to create hyper-realistic representations of users. By training the system to understand what a person’s face looks like and then tasking it with recreating that look based on inputs from cameras inside of a VR headset, the project has demonstrated some truly impressive results.
Recreating typical facial poses with enough accuracy to be convincing is already a challenge, but then there’s a myriad of edge-cases to deal with, any of which can throw the whole system off and dive the avatar right back into the uncanny valley.
The big challenge, Facebook researchers say, is that it’s “impractical to have a uniform sample of all possible [facial] expressions” because there’s simply so many different ways that one can contort their face. Ultimately this means there’s a gap in the system’s example data, leaving it confused when it sees something new.
The Successor: Modular Codec Avatars
Image courtesy Facebook Reality Labs
Researchers Hang Chu, Shugao Ma, Fernando De la Torre, Sanja Fidler, and Yaser Sheikh from the University of Toronto, Vector Institute, and Facebook Reality Labs, propose a solution in a newly published research paper titled Expressive Telepresence via Modular Codec Avatars.
While the original Codec Avatar system looks to match an entire facial expression from its dataset to the input that it sees, the Modular Codec Avatar system divides the task by individual facial features—like each eye and the mouth—allowing it to synthesize the most accurate pose by fusing the best match from several different poses in its knowledge.
In Modular Codec Avatars, a modular encoder first extracts information inside each single headset-mounted camera view. This is followed by a modular synthesizer that estimates a full face expression along with its blending weights from the information extracted within the same modular branch. Finally, multiple estimated 3D faces are aggregated from different modules and blended together to form the final face output.
The goal is to improve the range of expressions that can be accurately represented without needing to feed the system more training data. You could say that the Modular Codec Avatar system is designed to be better at making inferences about what a face should look like compared to the original Codec Avatar system which relied more on direct comparison.
The Challenge of Representing Goofy Faces
One of the major benefits of this approach is improving the system’s ability to recreate novel facial expressions which it wasn’t trained against in the first place—like when people intentionally contort their faces in ways which are funny specifically because people don’t normally make such faces. The researchers called out this particular benefit in their paper, saying that “making funny expressions is part of social interaction. The Modular Codec Avatar model can naturally better facilitate this task due to stronger expressiveness.”
They tested this by making ‘artificial’ funny faces by randomly shuffling face features from completely different poses (ie: left eye from {pose A}, right eye from {pose B}, and mouth from {pose C}) and looked to see if the system could produce realistic results given the unexpectedly dissimilar feature input.
Image courtesy Facebook Reality Labs
“It can be seen [in the figure above] that Modular Codec Avatars produce natural flexible expressions, even though such expressions have never been seen holistically in the training set,” the researchers say.
Beyond making funny faces, the researchers found that the Modular Codec Avatar system can also improve facial realism by negating the difference in eye-pose that is inherent with wearing a headset.
In practical VR telepresence, we observe users often do not open their eyes to the full natural extend. This maybe due to muscle pressure from the headset wearing, and display light sources near the eyes. We introduce an eye amplification control knob to address this issue.
This allows the system to subtly modify the eyes to be closer to how they would actually look if the user wasn’t wearing a headset.
Image courtesy Facebook Reality Labs
– – – – –
While the idea of recreating faces by fusing together features from disparate pieces of example data isn’t itself entirely new, the researchers say that “instead of using linear or shallow features on the 3D mesh [like prior methods], our modules take place in latent spaces learned by deep neural networks. This enables capturing of complex non-linear effects, and producing facial animation with a new level of realism.”
The approach is also an effort to make this kind of avatar representation a bit more practical. The training data necessary to achieve good results with Codec Avatars requires first capturing the real user’s face across many complex facial poses. Modular Codec Avatars achieve similar results with greater expressiveness on less training data.
It’ll still be a while before anyone without access to a face-scanning lightstage will be able to be represented so accurately in VR, but with continued progress it seems plausible that one day users could capture their own face model quickly and easily through a smartphone app and then upload it as the basis for an avatar which crosses the uncanny valley.
There was a lot to take in from this week’s Facebook Developers Conference (F8) 2019, not just the official rollout of pre-orders and launch dates for Oculus Quest and Oculus Rift S. The company showcased some of the work it was doing across a range of challenges it was trying to tackle, one of them being social interaction in virtual reality (VR). During the second day’s keynote, Facebook unveiled its latest Oculus Avatars prototype which aims to bring someones entire body into VR.
The work is being done by Facebook Reality Labs (FRL) with the goal to be able to fully represent a user from head to toe in VR, making social interactions even more seamless and natural.
FRL’s Ronald Mallet began by discussing a project the team revealed a couple of months ago, Codec Avatars, which was about making avatars as photorealistic as possible. While certainly a massive leap from the current avatars available today, a massive amount of equipment was required to scan a user’s features, plus it only worked from the shoulders up.
Mallet noted that while plenty of information can be glean from someones facial expressions, the same can work for the entire body; where full motion can tell you about emotion, agreement, trust and empathy. The needed to create a body model that represented the human anatomy but which needed to be fully adaptive to match any individual automatically.
To build the model FRL started from the inside out, designing an anatomically correct skeleton with proper articulations, from the way the shoulders roll to joint movement. Then they matched the skeleton to a users body structure, which accurately followed the team members movements, all from a single sensor – Mallet didn’t specify which sensor if it was a normal Oculus one.
Of course, we can’t all be skeletons in VR so muscles were added to the model, helping to predict how they move and contract. Finally, skin and clothes were added as faithfully as possible with a physics-based model for cloth deformations, with the final prototype seen in the above image.
There’s still a way to go before this tech will be consumer ready, it does look very promising. For now, Oculus Rift users will have to be happy with the Expressive Avatars update. As further details are released, VRFocus will let you know.
