How NASA Astronauts Use VR & AR Aboard The International Space Station

A recent blog post from NASA detailed various ways that VR and AR technology is used aboard the International Space Station and back on Earth to prepare astronauts for take-off.

According to NASA, VR/AR tech is becoming an increasing strong staple in life aboard the ISS, often used to help astronauts complete tasks or activities in new or easier ways. There’s nine examples in the full blog post, alongside some photos showing a few VR and AR headsets you’re probably familiar with.

The most common headset pictured in the blog post is the HoloLens, Microsoft’s enterprise-only AR headset. The HoloLens has been used in projects such as Sidekick, which used high-definition holograms to help show crew members 3D schematics and diagrams while they work. It was also used for T2 AR, a project that will help guide astronauts through maintenance tasks for their space station treadmill, avoiding the need for real-time communication with ground crew back on Earth.

It’s not all AR though — VR is also adopted across the ISS. Some astronauts are pictured using Oculus Quests to test whether adding VR environments might improve astronauts’ exercise bike experience, while others are pictured using Oculus Rift and other headsets for a variety of tasks like operating robotic arms or piloting space vehicles.

The post also mentions The ISS Experience, a multi-episode immersive VR series by Felix & Paul Studios and TIME Studios, filmed over multiple months aboard the ISS. NASA says the series  gives people on Earth insight into life aboard the ISS, along with potentially sparking ideas and inspiration for further research and improved conditions for astronauts. The series won an Emmy Award for Outstanding Interactive Program recently, and the first two out of four planned episodes are available now through the Space Explorers app for Oculus Quest and Rift.

You can read NASA’s full blog post, detailing all the ways they use VR and AR technology aboard the ISS, here. 

Apple CEO Tim Cook: AR Is One Of ‘Very Few Profound Technologies”

In a recent interview with YouTuber iJustine, Tim Cook called himself “AR fan #1” and said he thinks it’s one of “very few profound technologies” that have the potential to permanently embed themselves in day-to-day life.

The comments on AR, prompted by a question from Justine, start around the 9:15 mark. Here’s a transcription of what he had to say (emphasis is our own):

You know, I am so excited about AR. I think AR is one of these very few profound technologies that we will look back on one day and, ‘How did we lives out without it?’ And so right now you can experience it in thousands of ways, using your iPad or your iPhone, but of course those will get better and better over time. Already, it’s a great way to shop, it’s a great way to learn, it enhances the learning process. I can’t wait for it to be even more important in collaboration and so forth. So I’m AR fan #1. I think it’s that big.

Simple things today that you can use it for, like if you’re shopping for a sofa or a chair or a lamp, in terms of experiencing it in your place… We’ve never been able to do that before until the last couple of years or so. And that’s at the early innings of AR, it will only get better

This is far from the first time Cook has been publicly bullish about AR. In April, he said it was “critically important” to Apple’s future, following on from comments two months earlier about how he thinks AR could one day attract as big of an audience as the iPhone. As far back as 2016, Tim Cook was telling Apple investors that AR will be huge but also take some time to get right.

Five years on, AR has come a long way but still has a ways to go. Facebook’s recent Ray-Ban smart glasses pave the way for that company’s public AR efforts, but Apple has yet to dip its toes into AR hardware beyond iPhone and iOS integration. It’s reported that this could change next year, with Apple rumored to launch an AR-VR headset in the second half of 2022 for anywhere between $1000-$2000 or more.

Facebook Reveals $299 Ray-Ban Stories Smartglasses With Camera And Assistant

After a week of teases, Facebook and Ray-Ban are finally announcing Ray-Ban Stories, a new line of smartglasses with built-in cameras and Facebook features.

Check out the full announcement video including an introduction to the kit from Facebook CEO Mark Zuckerberg and Rocco Basilico of Ray-Ban brand owner Luxottica below.

Ray-Ban Stories Specs And Features

As expected (and essentially confirmed in a leak earlier today), Ray-Ban Stories are sunglasses that resemble traditional Ray-Ban models with small cameras embedded in the top corners of the rims. Like the Snap Spectacles before them, you can use these cameras to instantly capture point-of-view footage, as was demonstrated by Facebook VP of VR and AR, Andrew Bosworth, in a tease earlier this week. The glasses don’t, however, feature any sort of VR or AR functionality.

The dual 5MP cameras capture up for 30 seconds of video or images. An LED lights up when the glasses are recording or taking a photo. There’s also built-in Bluetooth support and a 3-microphone audio array allowing you to connect to your phone to take calls or listen to apps.

As for Facebook features, the glasses are integrated with the Facebook Assistant, allowing you to operate the glasses with your voice. You can also use the Facebook View companion app to share media through other services including non-Facebook applications.

Ray-Ban Stories Price And Release Date

Facebook Ray-Ban Stories Group Shot

Ray-Ban Stories are launching today in the US, UK, Canada, Italy, Ireland and Australia and start at $299 USD, the same price as the entry-level 128GB Oculus Quest 2 VR headset. You can also pick them up with Polarized lenses starting at $329 or Transitions lenses starting at $379. The glasses come in three models named Wayfarer (Wayfarer L), Round and Meteor and are available in five different colors.

Are Ray-Ban Stories AR Glasses?

In a nutshell: no. AR technology has a spatial understanding of the world around you and then uses that information to project virtual images into your environment. Ray-Ban Stories won’t do this, and they won’t even have an overlay interface. They do, however, represent a form factor that Facebook is no doubt keen to one day reach with an AR headset; current devices like the Microsoft HoloLens 2 and Magic Leap One are bulky and unfit for all-day use. Facebook Reality Labs, the division of the company responsible for AR and VR technology, is using prototype AR glasses not intended for consumer use, but it will be some time before the company brings a true consumer AR headset to market.

What do you make of Ray-Ban Stories? Will you be picking up a pair? Let us know in the comments below!

Apple Uses AR To Tease Announcements For Sept. 14

Apple teased its next announcement event for Sept. 14 with an impressive demonstration of AR.

Tapping the Apple symbol on the announcement webpage launches a 3D object that, on an iPhone at least, can be positioned and resized in an AR view while casting a glow into the physical environment. If you move your iPhone closer to the symbol it reveals itself to be a portal with the text 9.14 floating inside. Using the latest iPhone 12 Pro with the teaser provided a pretty impressive demonstration of AR tracking technology — with the system robustly segmenting out people into the background with pretty believable occlusion. We know Apple is working on an advanced VR/AR headset but the company is known for cancelling projects late in development.

The event comes within days of a tease by Ray-Ban that’s expected to detail a pair of perspective-capturing Facebook smartglasses. While the device may be absent advanced AR features, we expect overlap between what the glasses do today and what features will be inside future eyewear from Facebook. In what’s likely a more near-term proposition than slim AR glasses, Facebook is pursuing a so-called “Quest Pro” high-end standalone that’s likely to incorporate more sensors for more robust tracking. Microsoft is already on its second-generation standalone AR headset (and working with the U.S. Army on a ruggedized version) while Google is unlikely to sit out the effort to power robust AR glasses. Pokemon Go maker Niantic is working on AR glasses and Snapchat recently launched an AR development kit.

The Apple announcement event on Sept. 14 will kick off at 10 am Pacific time.

Facebook/Ray-Ban Smartglasses Reveal Coming On Thursday

It looks like we’ll be getting the first details about Facebook’s collaboration with Ray-Ban on a pair of smartglasses later this week.

Ray-Ban’s official website just posted a tease with the date 09/09/2021, suggesting we’ll get more details this Thursday. We can see what’s likely a silhouette of the glasses themselves but, other than that there’s little in the way of details.

The page does allow users to sign up for a ‘Release Notification’, though it’s not clear if the smartglasses might actually launch this week, or we’ll simply get a date for later down the line. Last year Facebook said the glasses would launch in 2021 and, more recently, CEO Mark Zuckerberg reconfirmed they’d be the company’s next hardware launch.

Yesterday Facebook VP of AR and VR, Andrew Bosworth, teased a reveal of the glasses by showing off what looks like a point-of-view recording feature while on vacation with Zuckerberg. You don’t actually see the glasses themselves in the video, though.

Despite Bosworth’s tease, we’re not expecting this Ray-Ban collaboration to include any true augmented reality features, instead likely resembling one of the earlier generations of the Snap Spectacles (which, funnily enough, have already evolved into a pretty impressive prototype AR headset). That said, we’ll need to see the official details on Thursday to know what the device is really capable of. Facebook is working on its own, more advanced AR prototypes, but we don’t expect to see those for years yet.

What are you hoping to see out of the Ray-Ban and Facebook smartglasses? Let us know in the comments below.

Facebook Teases Ray-Ban Smartglasses Recording In New Video

It sure looks like Facebook is teasing its upcoming smartglasses, made it partnership with Ray-Ban.

A recent tweet from the company’s VP of AR and VR, Andrew Bosworth, features a video of the executive on what looks like a vacation with Facebook CEO, Mark Zuckerberg. But Bosworth’s tweet includes an emoji using sunglasses and the video appears to be filmed from his perspective. You can even see in some portions of the video that Bosworth has two free hands, meaning he isn’t holding a phone.

Facebook Ray-Ban Smartglasses Teased?

It’s very likely that this is demonstrating a recording feature on the confirmed Ray-Ban smartglasses, then. Last year we reported that the device would launch in 2021 and, even though the company hasn’t reconfirmed that’s the case, Zuckerberg did recently state that the glasses would be Facebook’s next major hardware launch. The ongoing chip shortage may have pushed Facebook’s plans further down the line of course.

Plus the timing works out for a tease – it’s very likely that Facebook fully reveals the glasses at its Facebook Connect developer conference in October. The company rebranded the event from Oculus Connect last year, noting at that time it would include a greater focus on AR as well as VR.

That said, it doesn’t sound like these smartglasses will feature any real AR capabilities – certainly not on the level of HoloLens or Magic Leap. Instead, features like this supposed point-of-view recording are likely to be the major sells for the device.

There are other possibilities of what could be being teased here, of course. Bosworth could be recording from Project Aria, Facebook Reality Lab’s internal AR smart glasses used for prototyping. But there’s no confirmed consumer plan for that device, so it’d be a strange tease. He could also just not be teasing anything at all, but that seems unlikely.

What for you make of this possible Ray-Ban smartglasses tease? Let us know in the comments below!

Facebook’s AR Research Aria Glasses Detailed In FCC Listing

An FCC listing for Facebook’s Aria Glasses, which are being used internally for AR research, provides new details via a user manual and various other testing and compliance documents.

The Aria Glasses were first announced at Facebook Connect last year. They are not a consumer product or a prototype for AR glasses, but are instead designed to be worn by Facebook employees as a research tool that will “help us collect data to uncover the underlying technical and ethical questions and start to look at answers to those,” as Facebook representatives explained when the project was announced. Aria Glasses do not feature a display and are intended to capture research data while the wearer goes about their day.

Project Aria “will capture the wearer’s video and audio, as well as their eye tracking and location information. The glasses’ on-device computing power will then be used to encrypt and store information that, when uploaded to separate, designated back-end storage space, will help our researchers figure out how AR can work in the real world,” Facebook explained. “Our future AR devices must be more perceptive in order to be more genuinely useful to us. In order for devices to understand where they are in relation to people and other objects, and how to make sense of any given situation, they need a virtual 3D map of the things around you. But it’s far too power-intensive to scan and reconstruct a space in real time from scratch, so AR glasses will need to tap into an existing 3D map we call LiveMaps. LiveMaps uses computer vision to construct a virtual representation of the parts of the world that are relevant to you. With these 3D maps, our future devices will be able to efficiently see, analyze, and understand the world around them and better serve those who use them. These devices will keep track of changes, like new street names, and update them in real-time. The Project Aria device is testing out how this can work in practice.”

Facebook also announced a partnership with Luxottica and Ray-Ban to produce “smart glasses”, which are unrelated to Aria and due for release later this year.

Now, we have a bit more information and our best look yet at Facebook’s Aria Glasses by way of documents within a FCC listing that was made public recently, as reported by Protocol’s Janko Roettgers.

The documents include a user manual, where the glasses are referred to as “Gemini EVT” and described as “prototype equipment … intended for limited distribution only for  purposes of testing and data collection.”

The manual also reveals that the glasses support prescription lenses, and that they do not fold down like a normal pair of glasses — the temples remain fixed at right angles from the main frame.

Project Aria AR Glasses Facebook

The user manual also includes photos labeling individual parts and external buttons. As you can see in the image embedded above, the glasses offer a mute switch, power button, proximity sensor and status LED.

When the mute switch is set to mute/privacy mode, red is visible on the LED indicator. The right hand side also features a capture button pictured below, but the document (at least at the time it was written) says it is only used to power on the device.

Project Aria AR Glasses Facebook

The glasses also come with a microfiber cloth, a power supply and a USB cord with a magnetic connector that attaches to the left temple of the glasses. In the manual, Facebook says this connector is “sensitive” and that users “may have to physically hold it together momentarily for the device to be enumerated and/or initiate charging.”

Project Aria AR Glasses Facebook

The glasses connect to an internal mobile companion app, called Ariane, and the footage collected from the cameras can be transferred and viewed on a computer as well.

Project Aria AR Glasses Facebook

While the Aria Glasses are not the consumer AR glasses many are expecting, they do give an insight into the research process Facebook is undertaking to move toward consumer AR. On the Project Aria site, Facebook is clear that the glasses are for research work only.

With Facebook Connect coming up in October, there’s a fair chance we might hear more about Project Aria and the resulting AR technology then. You can view the Aria FCC listing and all associated documents here.

Oculus Quest Apps Use AR To Teach You To Play Piano

In a video posted to Reddit earlier this week, developer Dominik Hackl showed footage of his AR app, Magic Keys, running on Oculus Quest using the experimental passthrough API.

The footage, embedded below, was posted to Reddit and shows Hackl playing a real life keyboard while using an AR overlay on the passthrough view to learn songs in real time.

The concept is similar to VRtuos, which we wrote about last year, and other similar VR piano apps available for Quest. The notes appear above the keys in sequential order, in a synthesia-like fashion, allowing you to learn a piece in real time without the need for sheet music or any existing knowledge.

Piano apps on Quest have had to use hand tracking and virtual piano calibration to line up the virtual keys with your real piano. While it worked pretty well, there was sometimes still a small feeling of disconnect between playing the real piano and seeing the virtual one. However, using AR to present the notes overlaid on a live view of a real piano has potential to be a game changer.

What’s especially notable about Magic Keys on Quest is that it’s an AR app, developed initially for tablets and then the HoloLens and Magic Leap One. Now that the Quest’s experimental passthrough API is available, it has been ported over to the Oculus Quest.

Magic Keys is not available publicly yet, but Hackl says that an APK will be available soon. However, it will have competition — existing Quest piano apps such as Grand Reality and VRtuos have indicated that experimental passthrough support on Quest may be in the works. In the case of the latter, a beta APK is already available to download on its Discord server.

Here’s VR Puzzler Gravity Lab Working In Quest 2’s AR Passthrough

Gravity Lab developer Mark Schramm has got his VR puzzler working with Quest 2’s passthrough API.

Earlier this month Facebook started to allow developers to experiment with the passthrough options on the standalone headset. Up until now, passthrough has only been used for Quest’s Guardian system, and developers haven’t had access to it. While teams still can’t release apps utilizing the API on the Quest store, we are starting to see the first experiments.

Take a look at a short gameplay clip below, first posted on Reddit.

In terms of Gravity Lab, that means bringing the game’s physics-based puzzles, in which players use different tools to transport objects to a goal zone, into AR. It works just like the core game – you play pipes, ramps and gravity manipulators in physical space. The only difference is this time the game’s happening in the real world, not a virtual environment. It’s essentially an AR version of Gravity Lab.

It’s a promising early look at what new kinds of experiences the Quest 2’s passthrough API could enable, although still clearly restricted by the black and white capture. Schramm didn’t give any indication that this experiment could one day actually be implemented into the game itself.

Gravity Lab itself is approaching its 5th birthday but is due to get a major update later this week that adds a time travel mechanic.

VR Developer Relations Leader Leaves Facebook, Joins Niantic

Dan Morris, the Head of Developer Relations at Facebook Reality Labs, left the company and is joining Niantic in a very similar role.

At Niantic, Morris will be the Director of Developer Relations, a role which will no doubt be similar in capacity to his role at Facebook, which shared a very similar name.

“There’s no other job in the world that I would have left for but this one,” said Morris, in a prepared statement posted to the Niantic blog. “Niantic’s technology unlocks something magical and uniquely exciting – an ability for people to enjoy AR experiences together, out in the real world, on whichever devices they’re carrying. Pokémon GO players are already living this, and the Lightship ARDK will enable third-party developers to build their own creations on top of our foundation.”

Niantic say that Morris is “hitting the ground running” and is already reviewing projects and nailing down the application process for the Lightshp ARDK private beta. The blog announcement post also encourages developers who haven’t signed up for the Lightship ARDK private beta to do so as soon as possible. “I look forward to identifying developers with compelling proposals and prototypes so that we can help them get to work building their dream applications ahead of general availability for the Lightship ARDK,” said Morris.

The Lightship platform is Niantic’s “planet-scale” AR platform which provides developers with an SDK to build AR experiences on a global map — a type of game popularized by Niantic’s mega-hit Pokemon Go in 2016. Several other games have since replicated the formula to varying success, including  2019’s Harry Potter: Wizards Unite and the upcoming Transformers: Heavy Metal.

Another AR-based game from Niantic is set to release in partnership with Nintendo this year as well, based on the Pikmin franchise.