deepfocus – Augmented & Virtual Reality Confabulation

December 20, 2018

DeepFocus: Facebook veröffentlicht KI-Renderalgorithmus als Open-Source-Software

Facebook veröffentlicht seinen KI-Renderalgorithmus DeepFocus als Open Source, um freien Zugang zur Software zu ermöglichen. Das Renderverfahren ist in der Lage, natürlich wirkende Unschärfe in Echtzeit zu erzeugen und dadurch ein deutlich realistischeres visuelles Bild der VR zu erzeugen.

DeepFocus – Facebooks KI-Rendering ab sofort als Open-Source-Software erhältlich

Bereits auf der diesjährigen Facebook-Entwicklerkonferenz F8 präsentierte das Unternehmen die Arbeit an einem eine neuen Prototyp namens Half Dome, welcher es erlaubt die verbauten Bildschirme mechanisch vor- und zurückzubewegen. Die Varifocal Displays sollen in den VR-Brillen der Zukunft verbaut werden, um verschiedene Fokusebenen innerhalb der VR zu erzeugen. Damit würden sie nach dem Vorbild des menschlichen Auges beispielsweise nahe Objekte scharf stellen, während weit entfernte Objekte in einer natürlichen Unschärfe dargestellt werden und umgekehrt.

Je nach Fokuspunkt und Blickrichtung soll also auch das Bild in Echtzeit angepasst werden, was für ein deutlich realistischeres und somit immersives VR-Erlebnis sorgt. Zudem schont es das menschliche Auge und sorgt damit für mehr Komfort innerhalb der virtuellen Umgebung.

Mit herkömmlichen Renderverfahren ist dies, softwareseitig, allerdings äußert schwierig umzusetzen, da diese nicht in der Lage sind, die nötigen Ergebnisse in komplexen Szenen schnell genug zu verarbeiten oder es zu Fehlern in der Darstellung kommt. Hier kommt das KI-Renderverfahren DeepFocus ins Spiel, das dank Deep Learning mit einem neuronalen Netzwerk bestehend aus 196.000 Bildern aus einem Szenenzufallsgenerator trainiert wurde. Dadurch ist es in der Lage, die 3D-Objekte innerhalb der komplexen Szenen genau zu erkennen, zu unterteilen und den Unschärfeeffekt entsprechend anzupassen.

Der Algorithmus erlaubt es also den künstlichen Unschärfeeffekt so realistisch in Echtzeit darzustellen, dass er den Nutzern gar nicht auffällt. Dies war ebenso das ursprüngliche Ziel der Forschergruppe, wie Marina Zannoli, eine der Entwicklerinnen, im Blog beschreibt:

“Unser Endziel ist es, visuelle Erlebnisse zu liefern, die sich nicht mehr von der Realität unterscheiden.”

Dennoch ist noch viel Entwicklungsarbeit nötig, um die Software auch zukünftig im Endkonsumentenmarkt einsetzen zu können. So ist derzeit noch ein extrem leistungsfähiges Setup mit vier Grafikkarten nötig, um DeepFocus mit dem Half-Dome-Prototyp einzusetzen, wie Forscher Salah Nouri, beschreibt:

“Wir konnten DeepFocus und Half Dome mit einem Setup mit vier Grafikkarten demonstrieren – ein signifikant leistungsfähigeres Setup als das, was Verbrauchern derzeit zur Verfügung steht […]. Dabei mussten wir sehr vorsichtig sein, wenn es darum ging, die Arbeit zwischen den vier GPUs zu parallelisieren, sodass die Speichertransfers zwischen ihnen so pipeliniert werden, dass sie keine zusätzliche Latenz erzeugen und praktisch keine Rechenleistung verursachen.”

Facebook veröffentlicht nun sein KI-Renderverfahren mitsamt dazugehörigem Code und den Daten aus dem neuronalen Training als Open-Source-Software und stellt sie damit frei für Devs zur Verfügung. Weitere Informationen sowie das veröffentlichte Paper auf der diesjährigen Entwicklerkonferenz SIGGRAPH Asia findet ihr hier.

(Quellen: Oculus Blog | Upload VR | Videos: Oculus YouTube)

Der Beitrag DeepFocus: Facebook veröffentlicht KI-Renderalgorithmus als Open-Source-Software zuerst gesehen auf VR∙Nerds. VR·Nerds am Werk!

December 20, 2018

Facebook Open-Sources its AI Platform DeepFocus to Improve VR Visual Rendering

Facebook Reality Labs (FRL) has been experimenting with a number of ways to improve the realism of virtual reality (VR), through both hardware and software means. During the Facebook Developers Conference (F8) 2018 in May the company unveiled Half Dome, a prototype headset with a varifocal mechanism. Now the lab has revealed DeepFocus, an AI-powered platform designed to render blur in real time and at various focal distances.

What Deep Focus and Half Dome are both trying to achieve is something our eyes do naturally, a defocus effect. As the gif above demonstrates, when our eyes look at objects at different distances whatever they’re not focused on is blurred and out of focus. While this may seem simple, trying to replicate the effect in VR isn’t exactly easy, but its creation has a whole bunch of use cases for the technology.

The first is the goal of truly realistic experiences inside VR. “Our end goal is to deliver visual experiences that are indistinguishable from reality,” says Marina Zannoli, a vision scientist at FRL via the Oculus Blog. “Our eyes are like tiny cameras: When they focus on a given object, the parts of the scene that are at a different depth look blurry. Those blurry regions help our visual system make sense of the three-dimensional structure of the world, and help us decide where to focus our eyes next. While varifocal VR headsets can deliver a crisp image anywhere the viewer looks, DeepFocus allows us to render the rest of the scene just the way it looks in the real world: naturally blurry.”

Another important aspect of DeepFocus is the comfort. The more natural VR looks and feels, the easier it is to use. “This is about all-day immersion,” says Douglas Lanman, FRL’s Director of Display Systems Research. “Whether you’re playing a video game for hours or looking at a boring spreadsheet, eye strain, visual fatigue and just having a beautiful image you’re willing to spend your day with, all of that matters.”

While FRL is currently using DeepFocus with Half Dome the software has been designed to be platform agnostic, which is why the DeepFocus team is open-sourcing the work and data set for engineers developing new VR systems, vision scientists, and other researchers studying perception. As further updates from FRL are released, VRFocus will let you know.

December 19, 2018

Facebook Open-sources DeepFocus Algorithm for More Realistic Varifocal VR Rendering

DeepFocus is Facebook’s AI-driven renderer that’s said to produce natural looking blur in real-time, something that’s poised to go hand-in-hand with the varifocal displays of tomorrow. Today, Facebook announced that DeepFocus is going open source; while the company’s wide field of view (FOV) prototype ‘Half Dome’ may be proprietary, their deep learning tool will be “hardware agnostic.”

When you hold up your hand in front of you, your eyes naturally converge and accommodate, bringing your hand into focus. The experience of this isn’t the same in the VR headsets of today however since the light is coming from a fixed source, sending your eyes into overdrive to resolve near-field images. This is where varifocal displays and eye tracking comes in, as the once fixed focal length becomes variable to match your eyes depending on where you’re looking at any given moment.

Essentially it will let you focus on objects regardless of their distance from you, making the overall experience more comfortable and immersive. But the missing piece of the puzzle here is the ability for the headset to also replicate natural-looking defocus blur too, something that happens when you focus on your hand and the background goes fuzzy. Enter DeepFocus.

In a research paper presented at SIGGRAPH Asia 2018, the company says DeepFocus is inspired by “increasing evidence of the important role retinal defocus blur plays in driving accommodative responses, as well as the perception of depth and physical realism.”

Unlike more traditional AI systems used for deep learning-based image analysis, DeepFocus is said to processes visuals while maintaining the ultrasharp image resolutions necessary for high-quality VR.

That means not only will things look more realistic in varifocal VR headsets and even AR headsets with light-field displays, they’ll also mitigate eyestrain associated with vergence-accommodation conflict.

“This network is demonstrated to accurately synthesize defocus blur, focal stacks, multilayer decompositions, and multiview imagery using only commonly available RGB-D images, enabling real-time, near-correct depictions of retinal blur with a broad set of accommodation-supporting HMDs,” Facebook researchers say.

Facebook will be publishing both the source and neural net training data today “for engineers developing new VR systems, vision scientists, and other researchers studying perception,” the company says in a blog post.

Introducing DeepFocus at Oculus Connect 5, Facebook Reality Lab’s chief scientist Michael Abrash said that while Half Dome and DeepFocus are essentially “just the start for optics and displays, which is the poster child for how progress is accelerating.”

The post Facebook Open-sources DeepFocus Algorithm for More Realistic Varifocal VR Rendering appeared first on Road to VR.

October 16, 2018

Oculus Chief Scientist Dives Deep Into the Near Future of AR & VR

In his latest presentation at Oculus Connect 5, Oculus Chief Scientist Michael Abrash took a fresh look at the five-year VR technology predictions he made at OC3 in 2016. He believes his often-referenced key predictions are “pretty much on track,” albeit delayed by about a year, and that he “underestimated in some areas.”

Optics & Displays

Revisiting each area of technology in turn, Abrash began by discussing optics and displays. His predictions for headset capabilities in the year 2021 were 4K × 4K resolution per-eye, a 140 degree field of view, and variable depth of focus.

“This is an area where I clearly undershot,” he said, noting that Oculus’ own Half Dome prototype shown earlier this year had already met two of these specifications (140 degree FOV and variable focus), and that display panels matching the predicted resolution have already been shown publicly.

Abrash highlighted the rapidly progressing area of research around varifocal displays, saying that they had made “significant progress in solving the problem” with the AI-driven renderer DeepFocus that can achieve “natural, gaze-contingent blur in real time,” and that they would be publishing their findings in the coming months.

Beyond Half Dome, Abrash briefly mentioned two potential solutions for future optics: pancake lenses and waveguides. Like Fresels, the pancake lens isn’t a new innovation, but is “only now becoming truly practical.” By using polarization-based reflection to fold the optic path into a small space, Abrash says pancake lenses have the potential of reaching retinal resolution and a 200 degree field of view, but there would have to be a tradeoff between form-factor and field of view. Because of the way pancake lenses work “you can get either a very wide field of view or a compact headset […] but not both at the same time,” he said.

But waveguides—a technology being accelerated by AR research and development—theoretically have no resolution or field of view limitations, and are only a few millimetres thick, and could eventually result in an incredibly lightweight headset at any desired field of view and at retina resolution (but that is still many years away).

Foveated Rendering

Moving on to graphics, Abrash’s key prediction in 2016 was that foveated rendering would be a core technology within five years. He extended his prediction by a year (saying that he now expects it will happen within four years from now), and that the rendering approach will likely be enhanced by deep learning. He showed an image with 95% of the pixels removed, with the distribution of remaining pixels dissipating away from the point of focus. The rest of the image was reconstructed efficiently through a deep learning algorithm, and it was impressively similar to the original full resolution version, ostensibly close enough to fool your peripheral vision. Foveated rendering ties closely with eye tracking, the technology that Abrash thought was the most risky of his predictions in 2016. Today, he is much more confident that solid eye tracking will be achieved (it is already part of the way there in Half Dome), but this prediction was also extended by a year.

Spatial Audio

Spatial audio was the next topic, and Abrash conceded that his prediction of personalised Head-Related Transfer Functions (the unique geometry of each person’s ear which influences how they perceive the soundfield around them) becoming a standard part of the home VR setup within five years might also need to be extended, but he described how a recent demo experience convinced him that “audio Presence is a real thing.” Clearly the technology already works, but the personalised HRTF used for this demonstration involved a 30-minute ear scan followed by “a lengthy simulation,” so it’s not yet suitable for a consumer-grade product.

Controllers & Input

Regarding controllers, Abrash stood by his predictions of Touch-like controllers remaining the primary input device in the near future (alongside hand tracking). After running a short clip of one of Oculus’ haptic glove experiments, he adjusted his previous opinion that haptic feedback for hands wasn’t even on the distant horizon, saying that “we’ll have useful haptic hands in some form within ten years.”

Ergonomics & Form Factor

Abrash presented this sleek concept as plausible form-factor for an AR/VR headset once waveguide optics are mastered. | Image courtesy Oculus

On the subject of ergonomics, Abrash referred to the increasingly significant technology overlap between VR and AR research, noting that future VR headsets will not only be wireless, but could be made much lighter by using the two-part architecture already introduced on some AR devices, where heavy components such as the battery and compute hardware could be placed in a puck that goes in your pocket or on your waist. He said this companion device could also link wirelessly to the headset for complete freedom of motion.

Even still, optical limitations are largely the bottleneck keeping VR headsets from approaching a ski-goggle like design, but advances in pancake and waveguide optics could make for significantly more slender headsets.

Continued on Page 2 »

The post Oculus Chief Scientist Dives Deep Into the Near Future of AR & VR appeared first on Road to VR.