Eye tracking experts SensoMotoric Instruments (SMI) have collaborated with Valve to bring their technology to OpenVR. Eye tracking-enabled HTC Vive units are being shown at GDC 2017, as part of an R&D effort to integrate eye-tracking support into Valve’s OpenVR API.
At this week’s Game Developers Conference, along with LG’s ‘next generation’ VR headset prototype, Valve are also showing some upgraded HTC Vive headsets at their booth, fitted with eye tracking technology by SensoMotoric Instruments (SMI). The German computer vision provider is a world leader in eye tracking technology, and has produced kits for the Oculus DK2, Gear VR and HTC Vive HMDs.
A peek inside one of SMI’s Modded Gear VR units, from E3 2016.
Eye tracking is a major technology hurdle that must be conquered in order to take VR to the next level of visual quality. Performance requirements for VR rendering exponentially increase with display resolution and field of view, so eye-tracked foveated rendering is essential – a technique that removes unnecessary fidelity from parts of the image that miss the detailed part of the retina (the fovea). While SMI demonstrated 250Hz eye tracking and foveated rendering over a year ago, Michael Abrash described it as “not a solved problem at all” at his keynote at Oculus Connect in October.
SMI are showcasing their latest work with ARM at their own booth, to highlight the benefits of foveated rendering on mobile VR devices, and Valve are giving demos of OpenVR eye-tracking features using modified Vive headsets. Integrating eye-tracking into the OpenVR API is an indication that Valve want to move forward swiftly with the technology, although it remains unclear whether the next round of consumer headsets will include eye tracking. Speaking to Tom’s Hardware before the event, Valve developer Yasser Malaika said “Eye tracking opens up several interesting possibilities to both VR developers and customers. Our collaboration with SMI on R&D, as well as on SMI’s efforts to make eye tracking enabled Vive units available to the larger VR community, have been critical to our growing understanding of how HMDs with integrated eye tracking will positively impact the future of VR.”
Eye-tracking is arguably one of the most important pieces of VR’s future, and SensoMotoric Instruments (SMI) and Valve are working together to bring the tech closer to reality.
As reported by Tom’s Hardware, SMI partnered with Valve to integrate its eye-tracking tech into OpenVR SDK and API, which will allow other companies to implement support into their VR software. Not only that, but the pair have also successfully integrated SMI’s tech into select HTC Vive units.
Those units have been shipped out to research partners who are currently busy toiling away at getting the most out of VR eye-tracking. In fact, this tech is apparently behind the mixed reality face-scanning tech that Google demonstrated last week, bringing a user’s full face into these videos. It’s not clear if Valve or HTC currently has any plans to offer any sort add-on device for consumers to bring eye-tracking to Vive, nor if a future iteration of the device includes it, though the latter is a safe bet, no matter how far off it is.
On the surface, eye-tracking might only appear to be useful as another means of input for VR experiences, but its importance actually runs far deeper than that. Firstly, the tech is vital for foveated rendering, a term that refers to a VR experience only fully rendering its graphics directly in the center of where a user’s eyes are looking. This hugely efficient process means apps won’t have to render a full screen at all times, and could feasibly lower the barrier to entry for VR hardware.
That’s something that eye-tracking headset FOVE is looking into, as announced this week, and will no doubt become an important feature of many devices in the future.
Furthermore, eye-tracking will one day be essential to recreating virtual avatars of ourselves for social VR experiences. If our virtual bodies are to become truly indistinguishable from our real ones, then incredibly accurate eye-tracking will be required, the kind not seen by any current device.
We’re hoping to go eyes-on with this solution at GDC this week, so stay tuned.
Visuals and the rendering of them is one of a few significant obstacles when it comes to the future of VR headsets. Mobile headsets are handicapped largely due to the dependence on the capabilities of the smartphones being used, as noted by the NFL’s media director in a previous report. The more major headsets have their issues as well and FOVE has partnered with AMD to demo a new style of rendering that could influence the trajectory of the VR industry as a whole.
FOVE developed their own VR headset, which was funded with over $400k in contributions via Kickstarter. They staked the claim that the FOVE 0 is the very first VR headset capable of eye-tracking and they use a “foveated rendering” technique that produces full-resolution imagery where a person is looking and spends fewer resources on images in the periphery. At GDC, FOVE partnered with AMD and digital entertainment studio Frima to show AMD Multires Rendering running on Radeon hardware for the first time. VR can tax graphics processors in a big way but this technique is FOVE’s attempt at helping the next generation of VR with efficient processing, realistic imagery, and affordability.
The result of foveating is something that closely mirrors the way human eyes function. In the press release for the announcement, AMD’s Director of VR Darryl Sartain says FOVE gives them the crucial ability to render high-resolution where it matters. The demo shows off a baseline GPU improvement of 30% which is more than enough to be impactful, but the team occasionally say improvements of 100% which nearly doubled the framerate. The CTO of FOZE, Lochlainn Wilson, says they’re still learning about foveating “but these early improvements are very real and very exciting”.
Maintaining a high framerate is crucial for VR and being able to sustain it with less processing power will pay major dividends across the entirety of the VR industry. Stay tuned to UploadVR for more GDC coverage this week.
In der nächsten Woche findet wieder die GDC in San Francisco statt und Virtual Reality Nerds sollten Augen und Ohren offen halten, denn es wird sicherlich wieder einige spannende News und Entwicklungen geben. Tobii ist ein Unternehmen, welches bereits seit einigen Jahren Eye Tracking Produkte entwickelt, die aber bisher für den Einsatz mit einem herkömmlichen Monitor gedacht waren. Auf der GDC wird das Unternehmen aber auch ein Produkt für das Eye Tracking in Virtual Reality Brillen zeigen und die Demo mit einer HTC Vive vorführen.
Tobii zeigt HTC Vive mit Eye Tracking auf der GDC
Die Demo auf der GDC soll aber nicht nur zeigen, dass Tobii ein System für das Eye Tracking anbieten kann, sondern sie soll auch die Wichtigkeit eines solchen Systems für Social VR Erfahrungen deutlich machen. Gleichzeitig zeigt das kurze Video aber auch, dass die Bewegung der Augen den Avatar noch nicht menschlich genug machen. Trotz der Augenbewegungen wirkt der Avatar leblos und unspekatakulär.
This short but ingenious video snippet from eye-tracking specialists Tobii was built to tease a social VR demo they’ve been working on for GDC next week. It aptly highlights the subtle yet striking enhancements that eye tracking may being to social VR applications.
We wrote just today that eye-tracking used inside virtual reality headsets, used to detect the gaze of the user, could play an extremely important part in making more realistic VR scenes at higher resolutions possible through the use of foveated rendering, but eye-tracking has important implications for a more human aspect of virtual reality.
Social VR has already proven itself as a potent example of the power that VR presence-inducing capabilities bring to online social interaction. Facebook recently showcased an impressive demonstration of how effective chatting with other humans inside a virtual space can be and how important that might be in the social media giant’s future offerings. However, despite the clever approximations of avatar mouth movement and hand gestures captured with Oculus Touch motion controllers, the dead eyes of the participants digital doppelgangers left something to be desired.
Swedish company Tobii, who has specialised in eye tracking since its inception in 2001, have produced hardware for gaze detection hardware in various guises for some time now, and has recently announced plans to bring its tech to virtual reality, via a $50M funding round.
What you see (embedded above) is the company’s brief but intriguing video, built for posting to the HTC Vive subreddit /r/vive, to tease a new demo the company seem to be planning to show at next weeks Game Developer Conference in San Francisco. The demo is of a virtual avatar, standing in front of two virtual mirrors, demonstrating how much more relatable the character shown in the right hand mirror (showing eye tracking enabled) than the left (sans eye tracking). This, combined with emulated mouth movements alongside head tracking and hand motions via VR controllers, really will elevate the social VR experience.
ARM Holdings and Sensormotoric Instruments (SMI) are teaming up at GDC to showcase the potential of powerful mobile GPUs, eye-tracking and foveated rendering technologies in a new made-for-VR demo that will debut at GDC next week.
ARM, one of the world’s leaders in microelectronic design and the company behind the enormously successful Cortex mobile CPUs and Mali GPU, is partnering with eye tracking specialist SMI at San Francisco’s Game Developer Conference this year to demonstrate how mobile rendering when partnered with techniques like foveated rendering can deliver high fidelity VR experiences.
ARM claims that its Mali line of mobile GPUs shipped over 750 million units in 2015 with its latest G71 series being pitched as sporting the grunt required to meet the high demands VR places on rendering hardware. The G71 has been rumoured to feature in Samsung’s latest smartphone, the Galaxy S8, details of which we may learn more at next week’s Mobile World Congress event in Barcelona.
At GDC however, SMI and ARM will join forces to demonstrate what current generation hardware can achieve when foveated rendering – a technique that optimises the level of level of detail in a VR scene rendered based on a user’s gaze, freeing up clock cycles for rendering more detail or a smoother frame rate (or both).
The new demo, on display at SMI’s booth (1924) at GDC next week, will be shown on a Samsung Galaxy S7 equipped Gear VR with SMI’s retrofitted eye-tracking technology. According to a press release from SMI, the new demo will “teleport the user into the heart of a smartphone, exploring the inside – from chip to camera to speakers – from an entirely new perspective, guided by a friendly robot companion.”
“Eye tracking technology will bring yet another level of sharpness and detail to untethered VR worlds,” said Pablo Fraile, director of ecosystems, mobile compute, ARM. “Our demonstration of SMI mobile eye tracking technology on ARM-based devices highlights how foveated rendering will increase the efficiency of mobile VR experiences without compromising frame rates or visual quality.”
SMI have been extremely aggressive in positioning themselves as the front runner in eye tracking technology for the VR space. Although gaze detection data and input didn’t make its way into this generation of consumer VR headsets, it’s expected that the technology holds the key to allowing systems to cope with the demands of higher resolution, next gen headsets. We’ve been impressed in the past with SMI’s demos, spending time with both an SMI modded Oculus Rift DK2 and a Samsung Gear VR. The company claims to have delivered over 10 eye tracking solutions in the VR space to date.
As for ARM, its messaging on virtual reality has been less forthright, but with the announcement of the Mali G71 and Cortex A73 products at Computex last year, this changed. The new Mali GPU ARM claims “was developed expressly to meet the needs of new industry advancements such as the Vulkan cross-platform graphics API from Khronos as well as the ever growing demand for a smooth mobile VR experience.” And if the rumours of its inclusion in Samsung’s flagship Galaxy smartphones are true, it will have ample opportunity to prove its mettle powering Gear VR applications.
Road to VR will be at GDC 2017 next week bringing you news on this and everything else VR at the show.
Last September we reported on the fact that Qualcomm was launching their own VR development kit with the ability to deliver standalone VR. What made the VR 820 so compelling was that it had 6-DoF tracking as well as integrated compute (Snapdragon 820) which was on par with all the latest flagship phones. It even had support for eye tracking, which we now know was through a partnership with none other than SMI. However, there was one thing that was missing, hand tracking. In fact, Intel was already demoing hand tracking this year at CES with their Project Alloy prototype.
Anyone that has used mobile VR knows that controllers are nice, but unless you can ‘see’ your hands and interact with your surroundings with your hands, the immersion is lost. HTC and Valve do this with their Vive controllers that are super low latency and extremely accurate and Oculus does this with their touch controllers and their extremely natural ergonomics. When it comes to mobile, in many cases you’re either stuck with a Bluetooth gamepad on Samsung or a controller like the Daydream controller which simply put isn’t good enough. Thankfully, the team at Leap Motion have been working tirelessly to deliver hand tracking and late last year launched their much more compact hand tracking solution specifically aimed at mobile form factors.
Now that their technology has been miniaturized, it can be integrated into platforms. One such platform that’s launching at MWC and GDC (since both shows are happening simultaneously), is Qualcomm’s new Snapdragon 835 VR development kit. This new Snapdragon 835 VR development kit features a 2560×1440 AMOLED display, 6DoF tracking, eye tracking, foveated rendering and many other performance and power saving features. This system is essentially an upgrade over the Snapdragon 820 developer kit that Qualcomm launched at IFA 2016. The real improvements are increased performance, power savings and support for Leap Motion. While we don’t quite yet know the performance of the Snapdragon 835, the expectations are that it will be quite a bit faster on the GPU than the Snapdragon 820, which is a blessing for VR. The Snapdragon 835 VRDK is expected to be available in Q2 through the Qualcomm Developer Network. This device is really designed to help developers optimize their apps for the Snapdragon 835 HMDs that are due out in the second half of this year.
In addition to announcing the partnership and support of Leap Motion and a new VR development kit based on Snapdragon 835, Qualcomm is also announcing an HMD accelerator program. This program is specifically aimed at accelerating the time to market for HMD manufacturers, which has been an issue for some companies. The program is designed to help HMD manufacturers reduce their engineering costs and time to market so that they can seed the market with these HMDs faster. Part of this program utilizes the newly announced Snapdragon 835 VR HMD and will connect OEMs with ODMs like Thundercomm or Goertek, the two leading HMD ODMs in the world. The program is designed to help OEMs modify the reference Snapdragon 835 VR HMD and enable pre-optimized features like SMI’s eye-tracking and Leap Motion’s hand tracking.
These three announcements are very closely intertwined and show where mobile VR and more specifically standalone VR is going. Mobile VR itself will still benefit from the advances that result from these new developments, however standalone VR is currently the focus of this platform. The interesting thing about the mobile industry and players like Qualcomm is that they can iterate so much more quickly than their PC counterparts that we are seeing mobile HMD feature sets leapfrog PC. The fact that the Snapdragon 835 VR platform will support both eye tracking and hand tracking is huge because both of those are natural interfaces. Combining hand tracking, eye tracking and voice recognition into a single device means that a user can naturally interface with their VR HMD without ever needing to touch anything. Ultimately, hands free VR is the holy grail and I think that Qualcomm has brought us one step closer to that reality.
Disclosure: My firm, Moor Insights & Strategy, like all research and analyst firms, provides or has provided research, analysis, advising, and/or consulting to many high-tech companies in the industry, including Google, Intel, Qualcomm and Samsung cited or related to this article. I do not hold any equity positions with any companies cited in this column.
Tobii's eye tracking technology allows you to control games with nothing more than your gaze. Here are the games that currently support it, as well as the features you can expect to use in them.
Sword Art Online (SAO), the VR-centric manga and anime series, will soon be gracing the eyeballs of Fove 0 owners in a new VR experience featuring the series’ female protagonist, Asuna. The SAO experience was created to welcome new users and teach them how to use Fove 0, the first commercially available VR headset with integrated eye-tracking.
According to Anime News Network, Fove 0 users will be able to meet the series’ female protagonist Asuna, who acts as a sort of concierge to new users. The experience leverages the headset’s eye-tracking capabilities to give Asuna the ability to recognize when you’re looking at her—but more importantly where your eye drifts when she speaks. Looking at her right in the eye will elicit a smile, but ignore her and she’ll get upset. Haruka Tomatsu, Asuna’s Japanese-language voice actor, says in a video presenting the headset (Japanese only) that Asuna will even get angry if you ogle her for too long.
The experience was created in honor of the upcoming Sword Art Online The Movie: Ordinal Scale (2017) film, and will be first made available in Japan and Korea starting January 31 with availability ending July 31, 2017. If you’re in either Japan or Korea, you can download the experience here or visit. There’s still no word on when or if the SAO ‘welcome experience’ will be made available to Western audiences.
Fove, the product of a successful Kickstarter campaign and hailed as the first of the ‘next generation of VR headsets’, launched Fove 0 worldwide late last year at $599. The San Francisco-based company has since secured over $11m across 3 funding rounds with investors including Samsung Ventures, Colopl VR Fund, and Foxconn Technology Group.
Sporting a single WQHD OLED 2560×1440 display (1280×1440 per eye), 70Hz refresh rate, and about a 100° field-of-view (FOV), Fove 0 is a modest offering specs-wise in comparison to current PC VR headsets like the Rift and Vive, which both have higher refresh rates, slightly higher FOV and dual displays for better interpupillary distance (IPD) fit, but makes up for it by packing an accurate and reliable infrared eye-tracking system that not only lets you use your eyes as an input device, but introduces a bevy of possibilities when it comes to making VR seem more real to users. Road to VR Executive Editor maintains a later prototype of the Fove 0 “serves as a solid proof of concept of what eye-tracking can add to virtual reality, and they’ve so far got an impressive headset to boot.” Check out the full hands-on article here.
Fove 0 is compatible with Valve’s OpenVR API, giving it basic access to a swath of SteamVR-compatible content. The company has since released their SDK which allows developers to integrate Fove support for projects built-in Unity, Unreal Engine, and CryEngine.
Aktuell ist noch nicht klar, in welchen Ländern die Erfahrung erscheinen wird und welcher Umfang uns erwartet. FOVE verwendet die Erfahrung aber direkt, um die eigene VR Brille zu bewerben und kündigt das Spiel auf der eigenen Webseite für Japan und Korea an. Laut Manga Tokyo soll die Erfahrung Meet Asuna FOVE VR Home heißen und den neusten Film bewerben, welcher im nächsten Monat in Japan startet.
Asuna wird also ein Bestandteil der VR Erfahrung sein und ihr könnt mit Asuna interagieren. Hierbei soll auch die Eye-Tracking Technologie von FOVE eine Rolle spielen. Wenn ihr Asuna Beachtung schenkt, dann wird sie sich freuen. Wenn ihr sie ignoriert, dann soll sie wütend werden. Die Sword Art Online VR Erfahrung für die FOVE 0 Brille wird also vermutlich nur eine kleinere Erfahrung und wir sind noch weit entfernt von einem kompletten Sword Art Online VR Ableger.
Die Anwendung soll am 31. Januar 2017 veröffentlicht werden und wird bis zum Juli verfügbar sein.
Das FOVE Team beginnt derzeit mit der Auslieferung der FOVE 0 Brille und Vorbesteller sollten in naher Zukunft ihre neue VR Brille erhalten. Aktuell wird das FOVE 0 für 599 US-Dollar angeboten und die Versandzeit beträgt 2-4 Wochen. Leider wird das FOVE 0 ohne ein System für das Positional Tracking ausgeliefert und deshalb wird das Headset wohl nur für Menschen interessant sein, die mit dem Eye-Tracking experimentieren wollen. Demnächst sollte unser FOVE 0 eintreffen und dann nehmen wir das Produkt für euch unter die Lupe.
