Apple Quietly Released One of The Most Impressive AR Room-mapping Tools

Apple has barely mentioned augmented or virtual reality in its big keynotes lately, however at WWDC 2022 earlier this month, the company quietly released probably one of the best 3D room-mapping tools for mobile AR yet.

Called RoomPlan, the ARKit Swift API uses the camera and LiDAR scanner on recent iPhones and iPads to create a 3D floor plan of a room, including key characteristics such as dimensions and types of furniture.

It’s not for consumers (yet) though. Apple says it’s aiming to appeal to professionals like architecture and interior designers for conceptual exploration and planning, as well as developers of real estate, e-commerce, or hospitality apps; developers can integrate RoomPlan directly into their AR-capable apps.

When it was released earlier this month, Jonathan Stephens, Chief Evangelist at spatial computing company EveryPoint, took RoomPlan for a test drive to see what it could do. The results are pretty surprising.

RoomPlan seems to be able to deal with a number of traditionally difficult situations, including the mirror seen above, but also messy spaces, open and closed doors, windows, and generally complex architecture. Still, Stephens’ house isn’t just a bunch of cube-shaped rooms, so there’s a few bits that just didn’t match up.

Vaulted ceilings, wall openings, multifloor areas like you might find in foyers were all a bit too difficult for RoomPlan to correctly digest. Although not perfect, it seems to at least autocorrect to some degree based on some assumptions of how things might best fit together.

RoomPlan isn’t just for app integrations though. Apple says it outputs in USD or USDZ file formats which include dimensions of each component recognized in the room, such as walls or cabinets, as well as the type of furniture detected.

If you’re looking to finetune the scan, dimensions and placement of each individual components can be adjusted when exported into various USDZ-compatible tools, such as Cinema 4D, Shapr3D, or AutoCAD, Apple says.

We’re still no closer to learning when the company plans to release its rumored mixed reality headset or its full-fledged AR glasses, however either AR or MR headset would need extremely robust space-mapping capabilities. Seeing Apple make these sorts of strides using its existent platforms certainly shows they’re on the right track.

If you haven’t been following along with the Apple rumor mill, check out some of the links below regarding the company’s mixed reality headset, codenamed N301:

What We (think we) Know About N301 Mixed Reality Headset

A special thanks to Hrafn Thorisson for pointing us to the news!

The post Apple Quietly Released One of The Most Impressive AR Room-mapping Tools appeared first on Road to VR.

HoloLens Chief Alex Kipman to Leave Microsoft Amid Misconduct Allegations

Alex Kipman is leaving Microsoft amid what an Insider report alleges to have stemmed from misconduct allegations leveled at the HoloLens co-creator.

The report maintains that Microsoft Cloud head Scott Guthrie is planning a reorganization of the departments, as Kipman is set to leave the company in the next two months as a part of transition process.

According to an email obtained by Insider, the company’s mixed reality hardware teams will join the Windows and Devices organization, which will be led by Panos Panay, whilst MR software teams will join the Experiences and Devices division under Jeff Teper.

The report details alleged actions by Kipman, including inappropriate behavior such as  “unwanted touching” of women employees and an instance wherein Kipman viewed an adult VR video in front of other employees.

“Managers warned employees not to leave women alone around Kipman,” the report maintains, according to three affected sources.

Insider says “[m]ore than 25 employees shared their experiences as part of a report that was compiled about Kipman.”

Military version of HoloLens (IVAS) | Image courtesy Microsoft

A former colleague told Insider that the pandemic was “[t]he best thing that happened, sadly,” as no one on the team had to interact with him personally.

Kipman hasn’t responded to any of these allegations. Microsoft also declined to confirm or deny the specific allegations against Kipman, however the company says it’s investigating every report and “for every claim found substantiated there is clear action taken.”

This follows a Business Insider report from earlier this year that cast doubt on a prospective HoloLens 3 amid an internal division that may have hobbled the company’s efforts to release its next AR headset as planned.

That earlier report maintained that progress on fulfilling its $22 billion US defense contract, which aims to put HoloLens in battlefield roles over the next 10 years, has been stymied by internal production issues.

An alleged internal rift stemming from competing designs, one of which would completely reposition HoloLens as a consumer AR device, were citied as reasons for the lack of progress on release of the next-gen device.

The post HoloLens Chief Alex Kipman to Leave Microsoft Amid Misconduct Allegations appeared first on Road to VR.

‘Asynchronous Reality’ Replays Events You Missed While In VR

Video game designers often make objects in the environment glow or stand out to let players know what’s interactable. Could the same idea be applied to the physical world to show you interesting moments from the past?

New research from the Sensing, Interaction & Perception Lab at ETH Zürich in Switzerland conceptualizes a work day with “Causality-preserving Asynchronous Reality“.  The idea shows how environments laden with depth sensors could allow collaborators to annotate objects in the physical environment, essentially leaving messages for their colleagues to pick up in the future. The effect shifts time like the answering machines of the last century but places messages in context of the physical environment in which they were made, much like the holographic recordings shown throughout the Horizon Zero Dawn games.

Researchers Andreas Rene Fender and Christian Holz explored the idea in a paper presented as part of the CHI human-computer interaction conference a few weeks ago in New Orleans. Holz was able to join our virtual studio to walk us through the research which he summarizes at the start of the video embedded below.

“We’re co-located but at different points in time, so it’s the same here but a different now,” explained Holz, assistant professor in computer science at ETH Zurich. “We can make sense of events as they happen in the shared environment.”


The work raises interesting questions around acclimatization to new forms of interpersonal communication that might be more common in the years to come. Will people really walk into a room and leave a message for someone who is so focused on their virtual environment they can’t be bothered? Employers everywhere are still establishing policies in response to employees asking to work remotely, with some companies like Apple, Google, and Meta working on headsets and services to move beyond video chat grids and power new paradigms in communication. Put another way, the workplace is changing and while “Asynchronous Reality” might not represent the exact way offices extend between physical locations in the future, this research still might give us an idea of what’s in store.

“We can take a piece of the office home and be co-located,” explained Holz. “I think technical feasibility is going to be there probably much sooner than the point at time at which we figured out what’s actually desirable.”

Niantic Launches Visual Positioning System For ‘Global Scale’ AR Experiences

Niantic‘s new Lightship Visual Positioning System (VPS) will facilitate interactions with ‘global scale’ persistent and synced AR content on mobile devices.

Niantic launched Lightship during its developer conference this week and you can see some footage in the video embedded above showing some phone-based AR apps using its new features starting from the 50:20 mark. The system is essentially a new type of map that developers can use for AR experiences, with the aim of providing location-based persistent content that’s synced up for all users.

Niantic is building the map from scanned visual data, which Niantic says will offer “centimeter-level” accuracy when pinpointing the location and orientation of users (or multiple users, in relation to each other) at a given location. The technology is similar to large-scale visual positioning systems in active development at Google and Snap.

While the promise of the system is to work globally, it’s not quite there just yet — as of launch yesterday, Niantic’s VPS system has around 30,000 public locations where VPS is available for developers to hook into. These locations are mainly spread across six key cities — San Francisco, London, Tokyo, Los Angeles, New York City and Seattle — and include “parks, paths, landmarks, local businesses and more.”

To expand the map, Niantic developed the Wayfarer app which allows developers to scan in new locations using their phones, available now in public beta. Niantic has also launched a surveyor program in the aforementioned six key launch cities to expedite the process.

“With only a single image frame from the end user’s camera, Lightship VPS swiftly and accurately determines a user’s precise, six-dimensional location,” according to a Niantic blog post.

Scaling VPS to a global level is a lofty goal for Niantic, but could improve mobile AR experiences which could seem to unlock far more interesting content with accurate maps pinning content to real world locations.

You can read more about Lightship VPS over on the Niantic blog.

Quest 2 Experimental Room Setup Adds Walls & Furniture To Mixed Reality

A new experimental room setup feature on Quest 2 allows owners to map out their walls, doors, windows, and furniture for a new class of mixed reality experience.

The new “Room Setup – Experimental” feature sees you mark out surfaces in your home to build out a basic outline of your walls and the objects within. Using the feature I was able to quickly make boxes for a table as well as an island in my kitchen. These two surfaces are on either side of my typical play space, and having them represented in VR made it so I could easily leave my Oculus Touch controllers on either surface without taking the headset off. That’s of course just the beginning as developers figure out creative ways to incorporate furniture and walls into their mixed reality apps.

The feature appeared in the settings of one of our Quest 2 headsets running v40 of the system. The new feature could be activated separate to the existing computer vision-based safety systems on Quest 2, Space Sense and Guardian boundaries, with the experimental feature built around a more robust “Scene Understanding” that was originally previewed last year during Meta’s Connect event.

“Bring the walls, furniture and objects from your room into VR so you can use apps that blend your real and virtual environments,” a dialog for the feature notes.

Late last year, developer Bob Berkebile built and released a free tool that enabled similar functionality, and some developers have been exploring these features in their Quest 2 apps on an individual basis as well. Notably, Berkebile lists on Linkedin he started at Meta in January of this year.

Earlier this month, Meta teased a new experience called The World Beyond coming to App Lab as a showpiece for the functionality. Meta said The World Beyond would launch with v40 of the software development kit for Oculus developers, but as of this writing v39 is still the latest version on the Oculus developer site.

I was able to test the new experimental room setup feature in the video below.


Later this year Meta is planning to sell a high-end standalone headset, currently known as Project Cambria, for significantly more than $800. It’ll feature a depth sensor and color passthrough views from higher resolution cameras that’ll likely make mixed reality experiences on that headset far more impressive.

Qualcomm: Latest XR2 Reference Design For AR Cuts The Cord

Qualcomm showed a new reference design for AR glasses with the XR2 chipset driving a wireless connection to a nearby phone, PC, or processing puck.

Qualcomm says the new “Wireless AR Smart Viewer” built by Goertek offers a diagonal field of view of just about 40 degrees. The new glasses design is the latest in a series of headsets from Qualcomm meant to make it easier for the company’s partners to make a product using each design as a reference.

A wired AR Smart Viewer design was announced last year powered by Qualcomm’s older and less expensive XR1 platform. XR2 is the newer, higher priced and more capable chip used to power Quest 2, Vive Focus 3, Pico Neo 3 Link. Qualcomm revealed a reference design for a VR headset based on XR2 back in 2020.

Qualcomm said several manufacturers are exploring the new wireless design which splits rendering and processing tasks between a compatible wireless device and the glasses themselves. Qualcomm claims there’s less than 3 milliseconds of latency between a smartphone and the glasses and the reference design carries a 90Hz refresh rate with 1920 x 1080 resolution per eye from a micro-OLED display.

Quest 2 is so inexpensive at $299 from Meta that the competition is essentially priced out of the consumer market for standalone VR while AR glasses designs offer such a slim field of view that their consumer appeal is fairly limited. That means few companies have taken advantage of these latest reference designs to build consumer products. The ThinkReality A3 smart glasses from Lenovo, for example, were developed in parallel to the XR1-based design and they only target business customers.

XR2 has been shipping in products for a couple years now and we asked Qualcomm’s Hugo Swart if he could offer a timeline for release of a second generation of the chipset. He declined to be specific but told journalists recently they’re “looking for the right time for the next leap in performance.” Later this year, Project Cambria from Meta is expected to bring a depth sensor to a high-end standalone product and we don’t yet know most of its specifications.

Google Previews ‘Prototype’ Glasses For Live AR Translation

Google’s keynote presentation at its annual developer’s conference closed out with a video showcasing a prototype live translation service on AR glasses.

The video shows Google product managers handing prototype glasses to research participants, “my mother speaks Mandarin and I speak English,” one of the participants explains, with the video showing “a simulated point of view” to bring across the concept of how the glasses could essentially enable real-time subtitles as a translation service next to the face, theoretically allowing people to maintain eye contact more while speaking.

While no details were revealed about the actual specifications of the glasses, the video continued a theme from the event of Google seeking to enhance or augment interactions in the physical world, in stark contrast from a few years ago when Google supported the development of virtual worlds with Daydream. The language Google executives used during the presentation also seemed to contrast with Meta’s current push toward the “metaverse.”

“We’ve been building augmented reality into many Google products, from Google Lens to multisearch, scene exploration, and Live and immersive views in Maps,” Alphabet and Google CEO Sundar Pichai wrote. “These AR capabilities are already useful on phones and the magic will really come alive when you can use them in the real world without the technology getting in the way.”

AR glasses face severe constraints in terms of battery consumption, heat dissipation, brightness, and field of view that seem to place the timelines for true consumer-oriented standalone AR glasses out into the future at least a couple years. Still, in a process that’s been building for a long time, we’re seeing technology giants begin to ready their existing services to power this coming augmented reality platform which Pichai called “the next frontier of computing.”

Gabe Newell Sold Jeri Ellsworth Key AR Tech While Firing Her

Tilt Five CEO Jeri Ellsworth is working with her team to ship their patented AR technology to Kickstarter backers as they build out from a focused vision starting as AR glasses for tabletop gaming.

About a decade ago, however, Ellsworth was developing some of the core of that technology while working at Valve with a small team of hardware engineers doing research on forward-facing ideas like AR and VR. The idea was to invent “novel user interactions that broaden the Steam user base” that also “bring the entire family together in the living room”, as Ellsworth describes on her LinkedIn page.

Then Ellsworth and her colleagues were fired, leaving the engineer to contemplate the value of her voice and ability to speak openly about her time at the company against the amount of money offered in a severance payment packaged with a non-disclosure agreement. Separate to this consideration, what would losing her job at Valve mean for the future of the technology she worked on there? Would she need to work on something else?

Ellsworth made a key decision the day she was fired. In a recent interview with UploadVR, here’s how she describes her memories of what happened during her last day at Valve:

“I feel very fortunate that I made a split second decision the day that Valve did their big layoff. I showed up at the office. I met someone in the elevator and they said, ‘did you hear what happened to Ed? They laid him off today’. And I’m like, ‘that’s my mechanical engineer, working on my project, how could they do that?’ So I stormed upstairs and then it was like a bomb had gone off in the middle of the room. Everyone’s just sitting around moping. I hadn’t even opened my email yet to see the HR request to come see them. And someone’s like, ‘you’re getting fired today.’ I’m like, what? How can that be? That was the strangest layoff I’ve ever been around…they just let us kind of hang out in the building for like eight hours and we were just assigned a time to go talk to HR.

Folks that knew they were getting laid off were like angry and sad, and there were just tons of emotions. I was later in the day when I was going to receive the bad news and so I go up and I was like prepared to chew someone’s ass out about it. I walk in the door and Gabe’s there in the room with a lawyer/HR/somebody, and I started off like super aggressive. I was like, well, ‘so this is it?’ You know? And then immediately broke down into tears and I was getting emotional. And I’m like ‘Gabe, you gave me this mission to bring the family together and I can’t believe you’re doing this to me…I was onto something amazing.’ And he said some things like ‘I’m always going to be a fan’, like, oh, okay. And got myself back under like emotional control. And as I was walking out the door, I think my back was even turned to him. And I was like, ‘you should just sell me the technology.’ And I turned around and he was like, ‘okay.’ And that was it.

He made the decision on the spot to let me take this optical technique out of Valve. It was pretty incredible. I could have just walked out the door and just moved on to whatever project after that and never thought back on it…I don’t think folks around Valve understood what we were really onto at the time, like how we could generate this light field and how comfortable and vivid and how it solved all these problems. It was just back in those days there was still this notion that somebody is going to just stumble onto a way to make this perfect AR system. And you won’t need to use anything like a game board and still people are still dreaming and hoping that they’ll stumble onto some way to make that happen. And it’s the laws of physics. It’s really difficult.

The agreement was for $100, Ellsworth said, plus the cost of lawyers to make it legal, and “we basically got everything in our lab dedicated to the retroreflective glasses which included the prototypes, optical components, software, computers, etc.” The agreement would essentially help launch development efforts at CastAR, a company the ex-Valve employees co-founded to continue developing their approach to AR.

“The biggest win was the legal documentation that gave us freedom to operate,” Ellsworth wrote in a direct message.

Years later, Ellsworth and other CastAR veterans would essentially need to buy it all again — now with actual patents backing their retroreflective optical technique — after CastAR went defunct. Now at Tilt Five, Ellsworth is determined to “take on the big players.”

“At castAR and TiltFive we made improvements to the original prototype designs and have about a dozen patents covering the current design,” Ellsworth wrote.

Tilt Five is working on drivers to enable multiple glasses to run from a single PC as well as driving the system from an iOS or Android smartphone, features which — if well supported among developers — could offer an approachable way for AR “to bring the entire family together in the living room.”

Niantic’s AR Pets Game Peridot Looks Like It Could Be Great For Glasses

Niantic is teasing a new AR pets game called Peridot with the tagline “always by your side” and gameplay built around exploring the world together with “creatures who feel so real, you’ll love every moment raising them from birth to adulthood.”

Niantic is in a build-up phase toward AR glasses alongside tech giants like Meta, Apple, and Google. The company is acquiring a range of startups to build out its Lightship platform for developers, hosting a developer event in late May and already teased AR glasses last year. Niantic’s breakout hit title Pokemon Go, however, is produced in partnership with The Pokemon Company and features AR interactions as something of an afterthought next to the premise “gotta catch ’em all.” Meanwhile, other licensed AR games from Niantic like Harry Potter: Wizards Unite and Catan: World Explorers have struggled so much they’ve shut down. Taken altogether, Niantic could use a hit title it owns entirely to function as both a testing ground for new ideas and a showcase for its strengths in augmented reality.

Enter Peridot, teased in the embedded video above, which looks set to combine aspects of Pokemon and Tamagotchi with the ability to breed creatures in collaboration with others “to expand the Peridot species.” A series of screenshots embedded below show some of the interactions in the game spending time with your “Dot”, as they’re called, as they join you in different contexts viewed through an AR camera view from your phone. Activities include “adventure together on daily walks”, “breed new generations of peridots”, and “uncover hidden treasures around the world.”

“Peridot is played in camera-based AR,” a fact sheet from Niantic explains. “And the result is that the creatures feel so real, players are immersed in a way that is only possible with Niantic’s Lightship ARDK technology. Dots are clever, and can recognize different real-world surfaces such as dirt, sand, water, grass, and foliage. When your Dot forages on one of these surfaces, it will obtain different kinds of foods accordingly – like kelp from the water or prickly beets from the sand.”

The game is in a soft launch testing phase with a sign-up page for alerts when it launches in various countries on Apple’s iOS and Google Play. While Niantic made no mention of glasses support in its announcement of the game, its use of Niantic’s Lightship development kit to power the experience likely means it’d be an obvious fit, with Niantic CEO John Hanke teasing last year that they’re working to “enable new kinds of devices that leverage our platform.”

Check out the gameplay screenshots below:

Tilt Five Aims For Mainstream AR With Android, iOS, And Multiple Glasses From One PC

Jeri Ellsworth sees Tilt Five paving the path to mainstream AR.

Tech giants like Apple, Google, Microsoft and Meta seem to be struggling to expand the field of view of lightweight AR glasses into something compelling you can wear anywhere and in wildly varying lighting conditions, but the timeline for release of this supposed AR eyewear for consumers always seems to be about half a decade away. Near-term, there are camera-equipped and audio-enabled glasses as well developer kits or high-end devices like HoloLens and Magic Leap that can be useful for training or certain specialized uses. Still, even high-end AR eyewear often fails to deliver a sense of compelling immersion because their augmentations are often severely limited to a central area of your vision.

And then there’s Tilt Five, priced starting around $359 for the glasses, wand and retroreflective board. Tilt Five’s system convincingly augments the view above and below the game board while letting you see your friends, family and physical environment naturally. It’s a constrained approach focused around tabletop gaming and “reinventing game night,” as Tilt Five’s website says.

“Our goal is to take on the big players. That’s the ultimate goal. I know we don’t think of this as some small thing. Technology changes over time. We’re going to get better. We’re going to innovate, we’re going to have amazing content on our system,” Ellsworth recently told me. “We’re going to be this center point in your living room that makes your real-world amazing and you’re not going to synthesize away what’s already amazing in your world. We started doing user testing and, play three hours of D&D on our system and see people laughing, sitting around the table being delighted — that’s, that’s very satisfying and I think probably more appealing to a broader audience.”

“It has a 110-degree field of view – by far larger than any other optical AR headset,” notes AR displays expert Karl Guttag in his write-up about Tilt Five. “What makes it ‘magical’ is that everything seems to work the way it should in a way I have not seen in any other AR device.”

That field of view is roughly on par with what’s achievable on most consumer VR headsets, including the $299 Quest 2. But while Quest 2 provides a black-and-white view of your physical surroundings with a computer vision-reconstructed AR passthrough view, that view is only meant for momentary checks of your physical space before returning to your fully simulated environment. Tilt Five, by contrast, is being tested with multi-hour tabletop gaming sessions. And while Quest 2 packs everything into the headset, Tilt Five still requires an external computing device to power its glasses. As of this writing, that’s still a PC. And with one PC per pair of glasses, the cost and complexity of an AR gaming night powered by Tilt Five stacks up considerably. A 3-pack of glasses, wands and a large-size game board was offered through the Kickstarter for $879.

Focus On The Future

The key right now to Ellsworth is focus — something that escaped its CastAR predecessor — and that means delivering to backers while setting the stage for Tilt Five’s next steps.

Delivery is still in progress and subject to ongoing international supply chain nightmares affecting every major manufacturer. “Barring the factory having to close because of COVID”, Ellsworth recently told us, they are on pace to fully deliver on their Kickstarter promises this summer (delivery was originally slated for summer 2020). The startup recently announced a deal to bring select titles from Asmodee Digital to the platform, with Asmodee being the publisher of well-known board games such as  CATAN, Ticket to Ride, and Pandemic, that suggests there could be some big games in store for Tilt Five. Plus, one of the first apps compatible with the glasses is Tabletopia, a free-to-play sandbox app with a wide range of board games accessible.

And those next steps? Those could be key toward broadening Tilt Five’s appeal.

“We’re starting internally sampling our driver that allows multiple headsets on one computer. And because we do reprojection in the headset we don’t have the same requirements as VR where you have to have insanely high frame rates because we upscale the images coming from the game engine up to 180 frames per second. So you always get a buttery smooth tracked image on the table even if you have multiple headsets on one device. And then our Android drivers are very close as well, so those should be rolling out fairly soon,” Ellsworth said. “iOS is a little bit later in the year. So it’ll give a lot of flexibility as far as the compute device.”

She cautioned, however, that the challenge is not about getting the Tilt Five glasses themselves to “run” from a given device, but about getting developers on board with building cross-compatible content.

“I would be sad if someone expected to grab a Tilt Five tomorrow and plug it into Android and there’s not enough content for them,” Ellsworth noted. “I don’t know exactly how to answer exact timelines because there’s work to be done and it’s complicated and there’s the content side of it too, that has to all fall into place.”

Ellsworth noted they were able to get the glasses running from a Steam Deck in about 20 minutes. Compatibility will depend on the content, she warned, but “as we get more polished on Android and iOS and other devices like that, it’ll just lower the barrier for folks that want to do solo and group gaming experiences without having to fish for wires over to a laptop or something. You wouldn’t have to go get a leading edge Android phone to run a decent experience.”

“Our long-term goals, we want this to be a device that’s so relatable that it’s like the early home pong machine where everyone had one,” Ellsworth said. “We want to cater to…call them the ‘adventurous gamers’… the gamers that have every gaming device in their house. They’ll have a Switch. They’ll have an Xbox. They probably play games on their PC. They probably have a VR headset and that’s a market that’s billions of dollars. So it’s quite large for addressable market for a first market.”

Ellsworth originally spun her technology out from Valve, noting “back in those days there was still this notion that somebody is going to just stumble onto a way to make this perfect AR system. And you won’t need to use anything like a game board and still people are still dreaming and hoping that they’ll stumble onto some way to make that happen. And it’s the laws of physics. It’s really difficult.”

“At Tilt Five, we’re all behind this mission of making a positive impact in the world that’s going to delight hundreds of millions of people,” Ellsworth said. “That’s the goal.”

The Tilt Five system can be reserved on the company’s website.