Qualcomm is today announcing a new reference design VR headset based on its XR2 chipset that’s said to be 5G-capable.
Qualcomm regularly debuts new reference designs to show what’s possible with its latest processors and this new VR headset design is its most advanced yet. The company’s processors are at the core of both the Oculus Go and Oculus Quest VR headsets, though both systems from Facebook use relatively older and more affordable processors from Qualcomm. The Snapdragon 820 series from Qualcomm was first announced in 2015 and Oculus Go uses the 821. 2019’s Oculus Quest uses the Snapdragon 835 which was announced in 2017. The XR2 platform Qualcomm is promoting in 2020 is a VR and AR-focused variant of the 865 which was first announced at the end of 2019.
The latest XR2 Qualcomm reference design is said to be 5G-capable
This latest processor and reference design is said to be capable of split rendering over a 5G wireless connection — that is, some processing is done on the headset itself and some is done at a cell tower where power-hungry processors could theoretically draw photorealistic visuals for the headset. In theory, this would allow a relatively lightweight wireless VR headset to offer the kinds of impressive visuals which are currently delivered by an expensive high-powered PC over a wired connection. Qualcomm announced support for split rendering via a wireless connection to a nearby PC last year and the Pico Neo 2 is said to support this functionality.
It may be years still before this type of rendering pipeline is broadly available to consumers. Hugo Swart is in charge of the VR and AR efforts at Qualcomm and said in a conference call with journalists it is likely we’ll see trials of 5G for VR and AR uses this year and initial deployments sometime in 2021. Cellular network technology upgrades usually take place over many years and only some cities get support first. Likewise, Facebook and other companies building VR and AR headsets try to keep their hardware costs to a minimum to access the largest set of price-sensitive buyers. That usually means using the cheapest (sometimes older) components whenever possible.
Still, the latest reference design meshes together a series of technologies, some of which might make it into more headsets more quickly than 5G. The device uses NDI’s electromagnetic controller tracking, for instance, which we saw demonstrated at CES on the Pico Neo 2. This type of tech still tracks the controllers even if something — like your body — is between the headset and the controllers.
Here’s a video showing how that technology works.
In addition to 5G, Qualcomm says its reference design supports up to seven cameras. Two of them are internal for eye tracking and four are external, two each for showing passthrough views of the environment and two for head tracking. Device manufacturers could also also add face-tracking via another inward-facing camera. Qualcomm says the device also includes an infrared emitter to track hand movements and head movements with the same cameras and “2Kx2K per eye dual panel LCD support,” though the XR2 is supposed to be able to support even higher resolution panels as well.
During CES 2020 last month VRFocus saw the potential of augmented reality (AR) collaboration demoing Spatial’s software solution. Today, the company has revealed a new effort in partnership with several companies including AR headset manufacturer Nreal to accelerate mass-market adoption of 5G-optimized AR collaboration tools.
5G looks set to become the technology of 2020 as compatible devices begin to flood the market. Hence why more and more companies are seeking to employ 5G across a range of use cases, AR collaboration being one of them.
Spatial and Nreal have teamed up with Qualcomm Technologies, LG Uplus, KDDI, and Deutsche Telekom to help make this a reality as soon as possible. Each company will work together to share technical requirements and best practices, with Spatial’s device-agnostic collaboration software, coupled with Nreal’s latest headset ‘Nreal Light’ offering the first 5G combination. Spatial also plans to make its platform available on consumer 5G devices later this year.
“Consumers will be able to work or interact with anyone, anywhere as if sitting next to each other,” said Co-Founder and CEO Anand Agarawala in a statement. “Spatial is already in use and being actively explored by a significant portion of the Fortune 1000. A large part of that interest is driven by improved access to hardware and, as 5G networks and mass-market headsets like Nreal Light become commonplace, we’re throwing jet fuel into that fire. This combination of hardware, chipset, and carrier giants is a perfect storm for AR just like what drove the mobile revolution in the early 2000s with smartphones, high-speed mobile data, and app stores.”
“The 5G networks will bring out the change in the way people communicate. The 3G networks enabled voice call to video call. The 4G networks make that video call popular to the public. The 5G networks will promise the communication in virtual reality with virtual 3D avatar which makes a video call even more realistic,” said Daewon Song, the head of LG Uplus Future Device Unit. “That is why LG Uplus is working with Spatial for telepresence solution, to prove the validity and potential of Spatial’s Telepresence on Nreal AR glasses.
Spatial’s platform is already available on headsets like Microsoft HoloLens and Magic Leap 1 in a limited capacity as customers need to signup to unlock the application. The Nreal Light developer kit is available to pre-order now for $1,199.00, weighing in at 88g whilst sporting a 1080P display with a 52° field of view (FoV). As further details of the project are released, VRFocus will let you know.
AT&T will be at the DreamHack Open to show what a 5G network can do by streaming footage of the CS:GO tournament live. Three 5G-enabled smartphones will be on hand to capture unique perspectives.
Tis the season of 5G, the next-gen mobile data technology which promises to boost bandwidth and reduce latency. With 5G phones and networks slowly starting to become a reality, carriers would have you believe that the tech is going to radically change the world, and then some. But what does 5G actually mean for VR & AR?
As the cellular industry’s hottest new technology (and hopeful revenue driver), every stakeholder from cell phone makers to chip manufacturers is playing up 5G as the start of a revolutionary shift—and what better way to do so than to associate it with buzz-worthy technologies?
You can Google “5G” along with any other hot technology in the same query and you can find someone telling you that somehow and at some point 5G is going to revolutionize that thing. I’m not even joking. Let’s take a quick survey:
The list goes on and on, and it of course includes 5G + VR and 5G + AR, with terrible articles like this one that fundamentally butcher the relationship between the technologies with sentences like these:
AR and VR depend on decreased latency to run efficiently. The user experience will be amazing with a reduced lag time. With a better UX comes more opportunity. Telemedicine, virtual training environments for business and more will be the result.
In an effort to combat this hand-waving nonsense, let’s talk about what 5G could actually mean for VR and AR. The easy way to do that is to first understand what 5G actually brings to the table.
What Does 5G Mean For You
Quite simply: 5G promises greater bandwidth and lower latency than current mobile data connections offer. It also has the potential to make such high performance networks more accessible to more people.
Exactly how fast and what latency 5G will provide to individual end users won’t be clear until the technology is actually rolled out at scale, but mobile industry marketing would have you believe that 1Gbps download bandwidth and 1ms latency is possible.
To quickly put that into perspective, the average mobile and home connection speed in the US was 33Mbps and 112Mbps, respectively, in January 2019, according to internet metrics company Ookla. As for latency, 35ms for mobile and 5-15ms for home connections are common. Roughly speaking then, 5G could offer 30x and 9x the bandwidth of mobile and home connections, respectively, with 35x and 10x faster latency.
But what in the realm of VR and AR is actually enabled or enhanced with more bandwidth and less latency?
Immersive Video Streaming
Image courtesy NextVR
When it comes to streaming non-interactive 360 or 180 video content to VR and AR devices, one major misconception is that 5G’s low latency will be a huge benefit. In fact, the opposite is much more likely to be true—it’s 5G’s big bandwidth which opens the door to significantly improved quality which would not only improve the immersive video experience of today, but also offer a pathway for further growth as VR and AR displays become higher resolution.
The reason latency hardly matters for non-interactive streaming video content is that headtracking on VR and AR headsets is not coupled to the streaming latency. In most incarnations of 180 and 360 immersive streaming technology, the visual adjustments from the movement of the user’s head are processed locally on the device at a high rate, rather than sending the motion to some remote server and waiting to receiving an updated frame which corrects for head motion.
In fact, on today’s VR headsets you could be in the middle of watching a 360 video and then completely lose connection and the only impact would be that the video would freeze—moving your head would still respond just fine because the scene’s movement with regard to your head is processed right on the device.
There’s some nuance to this: some more sophisticated VR and AR streaming uses ‘view-dependent’ technology which optimizes the quality of the footage specifically in the area that you’re looking (in an effort to minimize the bandwidth bottleneck). High latency in this case could mean that if you turn your head quickly to a new part of the scene, you might notice that the video isn’t as sharp until the higher quality section of the video pops into view. In this case lower latency will make those transitions smoother, but it’s more of bonus than a radical shift in the experience.
When it comes to bandwidth however, 5G could significantly improve 180 and 360 video streaming quality. The most immersive footage—360 3D or full volumetric video—is also the most bandwidth intensive and is used sparingly today in a large part because most users don’t have sufficient connections to stream this kind of content in high quality.
Cloud-rendered VR and AR Gaming
Image courtesy Microsoft
When it comes to interactive VR content, 5G could make cloud-rendered VR and AR gaming possible, but this use-case is still largely hypothetical.
The pitch goes: users will own a simple inexpensive headset that streams high-end PC VR or AR graphics right from the cloud with no PC needed! It sounds amazing, but you aren’t going to see this happen until well after normal non-VR game streaming matures because VR demands significantly higher resolutions and lower latency than non-VR game streaming (which is still struggling to meet the performance needs of competitive gamers).
While latency isn’t really that important in the case of non-interactive video streaming, here it is crucial because what’s being rendered needs to change in real-time based on the actions of the user. So if 5G can truly offer up, let’s say realistically, 4ms or less latency, it really could unlock the cloud-rendered use-case of VR and AR.
But that’s a big if. Not because sub-4ms latency isn’t possible, but because you need that level of latency not just to your nearest connection in the network, but specifically to the location where your frames are being rendered.
If you run a network speed test right now, the test will very likely connect to a server that’s very close to you. Your latency is the round-trip time that it takes to get information to that server and back. If you test again and select a server that’s a few states over, you’ll find something very different. I just did this exercise and saw 10ms of latency to a server in my city and 35ms to a server that’s three states over.
This is to demonstrate that network latency increases as a function of your physical distance to the servers you are pinging. If you want to render VR and AR content in the cloud at low latency, the user needs to be physically close to the location where the frames are being rendered. This is much more difficult to achieve at scale because the frames of an interactive application have to be rendered on the fly.
When it comes to non-interactive video streaming, it’s relatively trivial to distribute the pre-rendered video files to servers across the globe ahead of time, and then serve them over the network as needed. Real-time rendering of VR content requires high-end GPU hardware in each data center from which cloud rendering would happen; each properly equipped data center can only deliver those frames with low enough latency to users who are within a certain physical distance.
That is to say that a single data center in the middle of the continental US, for instance, would have too much latency by the time it reaches the country’s coasts for a viable cloud-rendered experience experience. A distributed cluster of capable data centers (also known as ‘edge computing’) is the key to making this viable; 5G helps by expanding the ‘coverage’ of capable data centers by reducing latency, while high bandwidth makes greater quality imagery possible in this use-case (which will be increasingly important as the resolution of VR and AR devices increase).
Though it may take both 5G and edge computing to make this all possible, the good news is that cloud-rendered VR and AR services can piggyback off the same infrastructure that’s being deployed for non-VR game streaming services like Google’s Stadia and Microsoft’s xCloud.
On this episode of Digital Trends Live, we discuss the biggest trending stories in tech, including announcements from Disney’s D23 convention, a leaked PlayStation 5 design, and more.
2025 is the year that mobile carriers say 5G goes mainstream. How will the technology change, revolutionize, or make obsolete the things we do today? We asked futurists to give us an idea of our future 5G connected world.
Though faster speeds and increase capacity are often heralded as benefits to Wi-Fi 6, the promise for longer battery life, lower latency, and better security could be equally important to PC users.
Known worldwide for its contributions to the MPEG format — the compression technology used in MP3 audio files and MP4 videos — Germany’s Fraunhofer has recently turned its attention to the next frontier in media: virtual reality. After unveiling affordable VR headset microdisplay hardware last year, the company is now showing off next-generation video compression software using the new MPEG-OMAF standard, the first VR specification enabling 360-degree videos to stream over 5G networks.
Based in “significant” part upon Fraunhofer video compression technologies, MPEG-OMAF breaks wraparound videos into grids of tiles encoded at multiple resolutions. The explanatory image above uses red tiles to indicate areas that are being streamed at low resolution, versus normally colored tiles that are being streamed at high resolution.
Unlike traditional videos, which stream from servers at one user-selected resolution, these VR videos dynamically use high-resolution tiles where the viewer is currently looking, and low-resolution tiles for parts that are out of sight. As the user’s head position changes, the headset or display device requests a different mix of streamed tiles optimized for the user’s current focus area.
This trick enables the entire 360-degree video to continue streaming while devoting maximum detail to whatever the user is viewing. It parallels the recent use of foveated rendering to maximize real-time 3D graphics for VR users, guaranteeing that head-moving viewers will always be able to see something through their peripheral vision, even if it’s lower in fidelity.
International cellular standards organization 3GPP has adopted the MPEG-OMAF standard for 5G VR streaming, so it will likely underpin plenty of 360-degree virtual reality video streams — just like MP3 and MP4 defined prior generations of digital audio and video. Current 360-degree VR videos streamed over 4G and even Wi-Fi networks tend to suffer from low overall resolutions, across-the-board compression artifacts, and high latency, all of which the new standard and higher bandwidth networks could eliminate.
Fraunhofer is demonstrating the new technology using a combination of JavaScript, Apple’s Safari web browser, the WebGL API for rendering, and HEVC video support; a technical video is available here. Source code for the JavaScript player and instructions on creating standards-compliant content are available now on GitHub.
Faster speeds and more bandwidth are some of the many promises that 5G can deliver, but for gamers, the most important thing is low latency. To achieve low latency, carriers like AT&T and Verizon are exploring hybrid models for game streaming and mixed reality experiences.
With 5G networks rolling out, the telecom industry and its partners are trying to identify use cases that will justify the trillions of dollars they are spending on this infrastructure. MWC Barcelona 19 this week left little doubt what many believe to be the answer: virtual reality and cloud gaming.
Across the sprawling showroom floors and the flood of press releases, it was hard to find anyone who wasn’t eager to talk up the potential for these overlapping services. While 5G promises to enable all sorts of disruptive uses, like smart cities and connected factories, VR and cloud gaming are increasingly seen as the most promising ways to convince consumers to pay for new services that will get them on the 5G bandwagon.
By the same measure, 5G offers a shot at redemption for two technologies that have seen their own cycles of hype and disappointment as sales, technology, and experiences repeatedly failed to meet expectations. Industry insiders are betting that 5G is the missing ingredient that will allow them to finally win over consumers.
“This is a key use case for 5G for telecoms to reach the residential markets,” said Javier Polo, CEO of PlayGiga, a VR and cloud gaming startup based in Madrid.
Certainly hype around cloud gaming has been building over the past year. The Research and Reports firm projects the global cloud gaming market will grow to $6.226 billion by 2026, up from $765 million in 2017. The reasons point to a kind of two-step evolutionary process that telecom firms envision for the adoption of 5G.
Today’s landline broadband connections are growing so quickly that streaming quality has improved dramatically, making multi-player gaming, in particular, more reasonable. Polo notes that this offers immediate payoffs in global markets where large percentages of users don’t have powerful enough PCs or consoles to play the latest games, numbers that spike in developing markets. Streaming eliminates that issue, while also offering subscription-based payments that provide access to a huge catalog of games.
While systems like PlayGiga work with 4G networks on mobile devices, the latency completely disappears with 5G. On top of that, 5G will allow VR and AR headsets to be untethered, allowing users to access content and connections from anywhere. (PlayGiga demonstrated its service at the Vodafone booth at MWC, and the two companies have a research partnership.)
But PlayGiga is already powering cloud gaming services for carriers in Italy, Argentina, and Chile and is talking to carriers in the U.S. These carriers charge customers a monthly subscription fee, of which PlayGiga takes a cut.
The startup also offers a $110 retail version in Spain that comes with a set-top box for TVs, a game controller, and a six-month subscription. In a demo at PlayGiga’s MWC booth, Polo tapped the service on a smartphone and launched into Rise of the Tomb Raider in about 30 seconds.
“We can stream to any device,” he said. “It’s just a question of the market being ready.”
MWC saw a flood of related announcements. Vuzixunveiled its M400 enterprise smart glasses, which use Qualcomm’s Snapdragon XR1 platform. Qualcomm was particularly bullish on cloud gaming and VR. The company also demonstrated XR viewers (Extended Reality) that could be paired with high-end handsets from partners such as Xiaomi, Vivo, OnePlus, LG, BlackShark, and Asus.
Around the show, VR and cloud gaming were ubiquitous, with carriers, chip makers, and partners eager to dazzle attendees and make 5G seem more concrete.
Ericsson, meanwhile, was playing up its partnership with Blade to deliver cloud gaming.
Huawei didn’t have an announcement, but wanted to show off its own advances.
Nokia is also pushing VR and cloud gaming as it tries to convince carriers to accelerate purchases of network gear for 5G.
The multi-player version of the Spider-Man: Far From Home game released last year has been turned into a streaming game for 5G that drew big crowds at Nokia’s booth.
And PlayGiga was center stage in Vodafone’s booth.
Qualcomm’s bets on cloud gaming and VR were prominent.
With Intel also making a bid for this market.
For now, cloud gaming seems to be a safe bet for carriers, a way to get users to pay for one more subscription service. Whether that will truly translate and drive greater adoption of 5G services remains a bigger gamble, even as carriers move ahead in hopes that they can entice consumers to come along.
“The 5G networks will bring out the change in the way people communicate. The 3G networks enabled voice call to video call. The 4G networks make that video call popular to the public. The 5G networks will promise the communication in virtual reality with virtual 3D avatar which makes a video call even more realistic,” said Daewon Song, the head of LG Uplus Future Device Unit. “That is why LG Uplus is working with Spatial for telepresence solution, to prove the validity and potential of Spatial’s Telepresence on Nreal AR glasses.
