Google Trials ‘Starline’ Glasses-Free Light Field Display

Google’s research into light fields is bearing fruit with a glasses-free 3D display technology called “Project Starline” available at a few of its offices.

Google revealed the work as part of its annual developer conference this week. It is pitched as working like a “magic window” and relies on “custom-built hardware and highly specialized equipment” with advances in real-time compression, spatial audio, computer vision, and machine learning to provide a sense of being face to face with someone no matter the physical distance.

The image below posted by Google’s Vice President of AR and VR Clay Bavor offers a look at the substantial footprint for the system while it is used in one of Google’s offices.

project starline google

Google also posted a video showcasing the technology used for some person-to-person interactions said to provide “a sense of volume and depth that can be experienced without the need for additional glasses or headsets.” The company says it is planning to trial deployments with enterprise partners later this year.

We tested some early-stage glasses-free light field display technology in 2018 and it required years more development and enormous investments to improve brightness and cost enough to put it within reach of average consumers. In our 2018 demonstration from Light Field Lab, for instance, the 3D effect only worked if you kept your head in a very specific area relative to the display. Indeed, even with Google claiming key breakthroughs in its efforts to prove its glasses-free 3D display technology as a direction “technology can and should go”, the company cautions that only “some of these technical advancements” are likely to make it into its communication products.

Still, we’d love to go eyes-on with Project Starline at some point for a better sense of its use cases and the investment Google will need to spend to bring its advancements into wider use.

Google Launches Depth API for ARCore, Increasing Realism And Improving Occlusion

Google announced today that the Depth API is now available for ARCore 1.18 on Android and Unity. The Depth API is meant to improve occlusion and increase realism thanks to new interaction types.

The Depth API was first announced with a preview on the Google developers blog last year. The API allows a device to determine the depth of objects shown on the camera, according to how far or close by they are. In terms of AR, the API helps to significantly improve occlusion, which Google succintly describes as “the ability for digital objects to accurately appear in front of or behind real world objects.”

snapchat hotdog arcore

The example embedded above shows the dancing hotdog filter on Snapchat being accurately occluded by a lounge as the camera moves down. According to Google, another case where the API would be useful is in Five Nights at Freddy’s AR: Special Delivery, as occlusion is vital to the experience — characters can accurately hide behind objects and then provide a jump scare by moving out from behind the real-world object. Niantic showed something similar with Pokemon Go in the past as well.

However, Occlusion is not the only use for the Depth API — Google notes that developers have found many other uses as well, including implementing more realistic physics, better surface interactions, and environmental traversal. For example, the Google Creative Lab experiment ‘Lines of Play’ allows users to build AR domino arrangements that will accurately collide with furniture and walls in the room when the dominoes knocked over.

The Depth API will begin rolling out today. You can read more over on the Google developers blog.

The post Google Launches Depth API for ARCore, Increasing Realism And Improving Occlusion appeared first on UploadVR.

Retro-inspired Adventure ‘Pixel Ripped 1995’ to Launch Spring 2020

Pixel Ripped 1995, the sequel to the nostalgia-soaked VR game Pixel Ripped 1989 (2018), is now slated to launch this spring.

Created by São Paulo-based studio ARVORE, Pixel Ripped 1995 jumps six years forward into the history of gaming, leaving behind the 8-bit handhelds of the late ’80s and dipping its toes into the era of 16-bit and 32-bit consoles—all of course following the same trippy game-within-a-game style that Pixel Ripped 1989 pioneered.

Arvore says we should expect to find plenty of homages to ’90s games; to the studio, Pixel Ripped 1995 focuses on the historical transition from 2D to 3D gaming, as it includes action RPGs, brawlers, 2D and 3D platformers, space shooters and racing games.

The game is said to include six levels, which the studio says should individually “feel like an entire new game.”

Pixel Ripped 1995 is slated to support Oculus Quest, Oculus Rift, PSVR, and SteamVR headsets when it launches this spring. If you’re planning to play on a SteamVR headset, you can add it to your Steam wishlist in the meantime.

The post Retro-inspired Adventure ‘Pixel Ripped 1995’ to Launch Spring 2020 appeared first on Road to VR.

Google’s ARCore Is Getting Full Occlusion For More Real AR

Google’s ARCore is getting a new feature which will enable full occlusion of virtual objects in real scenes.

ARCore is Google’s Android augmented reality runtime and SDK. It provides positional tracking, surface detection, and lighting estimation so developers can easily create AR apps for high end Android phones.

But currently, virtual objects will always be shown on front, because ARCore has no good understanding of the depth of the real objects in the scene. Positional tracking works like on VR headsets- it tracks high contrast features in the scene, not the entire scene.

The ARCore Depth API estimates the depth of everything the camera sees. This allows for occlusion– virtual objects will now appear behind real objects if the real object is closer to the camera.

Occlusion is arguably as important to AR as positional tracking is to VR. Without it, the AR view will often “break the illusion” through depth conflicts.

Apple’s ARKit for iOS and iPadOS doesn’t have full depth occlusion yet. However, on the most recent powerful devices it does now have occlusion for human bodies, including hands. Whereas ARCore and ARKit have stayed roughly equivalent for now, it’s interesting to now see them diverge on different paths. This will make life harder for developers, but allow for specific innovation on each platform.

Real time understanding of scene depth could also be hugely beneficial for virtual reality. Current VR headsets keep you aware of your real surroundings by having you draw out your playspace during setup. If you come close to this boundary it will be shown. But this technology could allow this to be automatic and 3 dimensional – walking too close to your couch could make it appear in the headset.

oculus passthrough guardian

Facebook, which owns the Oculus brand, has shown off real time mobile depth mapping too, but hasn’t shipped it in a product yet.

ARCore’s Depth API isn’t publicly available and Google hasn’t given any release window, but developers can sign up for approval to try it out.

The post Google’s ARCore Is Getting Full Occlusion For More Real AR appeared first on UploadVR.

Google ARCore Depth API Now Available, Letting Devs Make AR More Realistic

ARCore, Google’s developer platform for building augmented reality experiences for mobile devices, just got an update that brings the company’s previously announced Depth API to Android and Unity developers. Depth API not only lets mobile devices create depth maps using a single RGB camera, but also aims to make the AR experience more natural, as virtual imagery is more realistically placed in the world.

Update (June 25th, 2020): Google today announced it’s making its Depth API for ARCore available to developers. A few studios have already integrated Depth API into their apps to create more convincing occlusion, such as Illumix’s Five Nights at Freddy’s AR: Special Delivery game, which lets enemies hide behind your real-world objects for more startling jump scares.

ARCore 1.18 for Android and Unity, including AR Foundation, is rolling out to what Google calls “hundreds of millions of compatible Android devices,” although there’s no clear list of which devices are supported just yet.

Original Article (December 9th, 2019): Shahram Izadi, Director of Research and Engineering at Google, says in a blog post the new Depth API now enables occlusion for mobile AR applications, and also the chance of creating more realistic physics and surface interactions.

To demonstrate, Google created a number of demos to shows off the full set of capabilities the new Depth API brings to ARCore. Keep an eye on the virtual objects as they’re accurately occluded by physical barriers.

“The ARCore Depth API allows developers to use our depth-from-motion algorithms to create a depth map using a single RGB camera,” Izadi says. “The depth map is created by taking multiple images from different angles and comparing them as you move your phone to estimate the distance to every pixel.”

Full-fledged AR headsets typically use multiple depth sensors to create depth maps like this, which Google says was created on device with a single sensors. Here, red indicates areas that closer, while blue is for farther areas:


“One important application for depth is occlusion: the ability for digital objects to accurately appear in front of or behind real world objects,” Izadi explains. “Occlusion helps digital objects feel as if they are actually in your space by blending them with the scene. We will begin making occlusion available in Scene Viewer, the developer tool that powers AR in Search, to an initial set of over 200 million ARCore-enabled Android devices today.”

Additionally, Izadi says Depth API does’t require specialized cameras and sensors, and that with the addition of time-of-flight (ToF) sensors to future mobile devices, ARCore’s depth mapping capabilities could eventually allow for virtual objects to occlude behind moving, physical objects.

The new Depth API follows Google’s release of its ‘Environmental HDR’ tool back at Google I/O in May, which brought more realistic lighting to AR objects and scenes, something which aims at enhancing immersion with more realistic reflections, shadows, and lighting.

Update (12:10): In a previous version of this article, it was claimed that Google was releasing Depth API today, however the company is only now putting out a form for developers interested in using the tool. You can sign up here.

The post Google ARCore Depth API Now Available, Letting Devs Make AR More Realistic appeared first on Road to VR.

Google ARCore Update Brings Changes To ‘Visual Processing In The Cloud‘

Google is updating its augmented reality cloud anchors system which takes camera data from your phone, processes parts of it on their servers, and produces a 3D map of the environment.

The technology allows for shared AR experiences where multiple camera-based gadgets can see the positions of one another. The change to the “Cloud Anchors API” is included in the latest version of Google’s augmented reality software ARCore, according to a Google blog post for developers published today.

”We’ve made some improvements to the Cloud Anchors API that make hosting and resolving anchors more efficient and robust. This is due to improved anchor creation and visual processing in the cloud. Now, when creating an anchor, more angles across larger areas in the scene can be captured for a more robust 3D feature map,” according to a post by Christina Tong, Product Manager, Augmented Reality at Google. “Once the map is created, the visual data used to create the map is deleted and only anchor IDs are shared with other devices to be resolved. Moreover, multiple anchors in the scene can now be resolved simultaneously, reducing the time needed to start a shared AR experience.”

I put a few pointed questions to Google representatives this morning for clarity on how exactly this functions. I asked for detail on what exactly “visual processing in the cloud” means and whether anything more than 3D pointcloud and location data is passed to Google servers. I also asked Google to specify how this API functioned differently in the past. Here’s the full response I received over email from a Google representative:

“When a Cloud Anchor is created, a user’s phone provides imagery from the rear-facing camera, along with data from the phone about movement through space. To recognize a Cloud Anchor, the phone provides imagery from the rear-facing camera,” according to Google. “Using the cloud (instead of the device) to do feature extraction allows us to reach a much higher bar of user experience across a wider variety of devices. By taking advantage of the computing power available in the cloud, we are able to extract feature points much more effectively. For example, we’re better able to recognize a Cloud Anchor even with environmental changes (lighting changes or objects moved around in the scene). All images are encrypted, automatically deleted, and are used only for powering shared or persistent AR experiences.”

For comparison, Apple is due to release its iOS 13 software soon and its iOS 12 documentation explains a method of producing a shared AR world map between local devices without sending data to a remote server.

Persistent Cloud Anchors

Google’s ARCore update also added “Augmented Faces”  support for Apple devices and the company says it is looking for developers to test “Persistent Cloud Anchors” with a form to fill out  expressing interest in “early access to ARCore’s newest updates.”

“We see this as enabling a ‘save button’ for AR, so that digital information overlaid on top of the real world can be experienced at anytime,” the Google blog post states. “Imagine working together on a redesign of your home throughout the year, leaving AR notes for your friends around an amusement park, or hiding AR objects at specific places around the world to be discovered by others.”

The post Google ARCore Update Brings Changes To ‘Visual Processing In The Cloud‘ appeared first on UploadVR.

Google ARCore Update Brings More Robust Cloud Anchors for Improved Multiuser AR

ARCore, Google’s developer platform for building augmented reality experiences, is getting an update today that aims to make shared AR experiences quicker and more reliable. Additionally, Google is also rolling out support for Augmented Faces on iOS, the company’s 3D face filter API.

Introduced last year, Google’s Cloud Anchors API essentially lets developers create a shared, cross-platform AR experience for Android and iOS, and then host the so-called anchors through Google’s Cloud services. Users can then add virtual objects to a scene, and share them with others so they view and interact simultaneously.

In today’s update, Google says it’s made improvements to the Cloud Anchors API that make hosting and resolving anchors more efficient and robust, something the company says is due to improved anchor creation and visual processing in the cloud.


Google AR team product manager Christina Tong says in a blog post that developers will now have access to more angles across larger areas in the scene, making for what she calls a “more robust 3D feature map.”

This, Tong explains, will allow for multiple anchors in the scene to be resolved simultaneously, which she says reduces the app’s startup time.

Tong says that once a map is created from your physical surroundings, the visual data used to create the map is deleted, leaving only anchor IDs to be shared with other devices.


In the future, Google is also looking to further develop Persistent Cloud Anchors, which would allow users to map and anchor content over both a larger area and an extended period of time, something Tong calls a “save button” for AR.

This prospective ‘AR save button’ would, according to Tong, be an important method of bridging the digital and physical worlds, as users may one day be able to leave anchors anywhere they need to, attaching things like notes, video links, and 3D objects.

Apps like Mark AR, a graffiti-art app developed by Sybo and iDreamSky, already uses Persistent Cloud Anchors to link user-made creations to real-world locations.

If you’re a developer, check out Google’s guide to creating Cloud Anchor-enabled apps here.

The post Google ARCore Update Brings More Robust Cloud Anchors for Improved Multiuser AR appeared first on Road to VR.

Google Maps’ ‘Live View’ AR Feature Available in Beta, Makes Getting Lost Harder

Google may seem to be losing interest in its virtual reality (VR) ventures such as Daydream View but on the augmented reality (AR) the company is still pressing forward with gusto. Having released an AR feature for Google Maps earlier this year to Google Maps Local Guides and Google Pixel users, the company has today begun a wider rollout of Live View.

Google Maps Live View

Currently still in beta, the feature to compatible iOS and Android devices which support ARKit and ARCore respectively. While the launch happens today, you may not be able to update your device just yet, as it may take Google several days or weeks to get to your region.

The whole purpose of the AR option is to make navigation using the highly popular Google Maps even easier and straight forward. All you need to do is tap on a location in the app, hit the Directions button then select the Walking option. After that, you should find the option to ‘Live View’ those directions towards the bottom of the screen. With Live View enabled you’ll get some gigantic handy arrows appear in the middle of the street (or wherever you are) telling you the right direction to head.

Obviously, this sort of feature isn’t supposed to make you continually hold your phone up and look like a lost kitten. You can simply bring it up when required to let you know you’ve gone the wrong way – or going the right way. It’s just one of a number of updates Google has added to the app, including being able to see all of your flight and hotel reservations in one place, or finding a nice restaurant and booking a reservation all without leaving the app.

Google Maps

While AR might be seen as the little brother to VR, it’s often thought of as having the greatest potential in the long run. Apart from apps like Google Maps, a lot of the AR content consumers are coming across at the moment are videogames such as Harry Potter: Wizards Unite and Minecraft Earth. VRFocus will continue its coverage of AR, reporting back with the latest updates.

PuzzlAR: World Tour is the First ARCore Experience Live in China

Western companies generally tend to struggle to break into the lucrative Chinese market for a number of reasons, which is why they’re inclined to find a home-grown partner to help facilitate the process. These first steps can be small and innocuous, such as Google’s ARCore arrival, helped by a videogame called PuzzlAR: World Tour.

PuzzlAR: World Tour image1

A 3D jigsaw puzzle videogame, PuzzlAR: World Tour first arrived for iOS in 2017 followed by Android devices in 2018. It uses landmarks such as the Statue of Liberty or the Taj Mahal and creates digital puzzles out of them. As is commonplace with most augmented reality (AR) titles you scan a flat surface to place the puzzle on, with all the pieces floating around you needing to be grabbed a put in place.

Developed by ONTOP Studios, its arrival in China is thanks to a collaboration with Chinese publisher NetEase (Nostos, Stay Silent), making the puzzle experience the very first ARCore compatible videogame available to the massive Chinese consumer market.

It’s not just mobile AR that ONTOP Studios has been interested in. PuzzlAR: World Tour also went live for the Magic Leap One headset last month, the first app supported by Magic Leap’s Independent Creator Program to do so. The program launched last year, seeing 31 companies chosen out of a pool of 6,500. Other successful applicants included Funktronic Labs (Starbear: TaxiCosmic Trip), Metanaut (Gadgeteer); Within, Felix & Paul Studios (Marshall from DetroitTraveling While Black) and Resolution Games (Angry Birds VR: Isle of Pigs).

“When I first discovered Magic Leap, I immediately knew this was a company with the vision and means to create technology that can make the future come to life, and I knew entertainment will soon change forever,” said ONTOP Studios’ Creative Director Nun Holhadela in a statement.

Mobile gaming is a massive market in China and one many AR developers like ONTOP Studios are keen to exploit, valved at $30.8 billion in 2018 and expected to rise to $41.5 billion in 2023 according to a report by Niko. Magic Leap doesn’t have quite the same mass-market appeal due to the cost of the device, but it’s encouraging more content on its platform, like the recently announced BBC Earth – Micro Kingdoms: Senses. As the AR market continues to develop, VRFocus will keep you updated.

Minecraft Earth Beta Coming To iOS This Month, Android ‘Soon’ After

Minecraft Earth, an all-new mobile AR version of the global sensation sandbox building and survival game, is coming to iOS later this month and Android “soon thereafter” according to the official blog.

We first heard about Minecraft Earth about two months ago in mid-May when developers Mojang released an announcement trailer and then last month debuted actual gameplay for the very first time. Now, they’ve got a brand new beta announcement video that goes over more details regarding the game’s mechanics and gameplay:

Basically you’re presented with an overworld map that looks strikingly similar to Pokemon Go, complete with a Minecraft-style block avatar complete with skins. It uses the actual world map to create the environment. You walk around and tap on items like animals and blocks to collect them. Naturally, they’re called “tappables” in Minecraft Earth.

Once you collect enough tappables, you level up and once you have enough resources you can build things that are placed into the real world from your phone screen. It’s described as a “living, breathing” Minecraft world. It includes multiplayer seamlessly integrated where people can help “or hinder” your creations. Then you can scale creations to life-size to explore and see in the world around you.

minecraft earth gameplay

I haven’t tried it yet, but honestly, it looks impressive.

The limited iOS beta for Minecraft Earth is due out within the next two weeks, which means sometime before July 26th. Only a “limited number of players in a few select cities” will be chosen before a wider release this summer.

You can sign up for a chance to be selected in the closed beta right here and check out the main website for more details as they’re available.

h/t: Engadget

The post Minecraft Earth Beta Coming To iOS This Month, Android ‘Soon’ After appeared first on UploadVR.