Spatialify for iPhone 15 Pro Can Now Capture HDR 'Spatial Videos' At 1080p 60fps Or 4K 30fps

The Spatialify app for iPhone 15 Pro can now record 3D 'spatial videos' with HDR at 1080p 60fps or 4K 30fps.

Spatial Video is Apple's term for stereoscopic 3D video on its devices. Currently, only Apple Vision Pro and the two iPhone 15 Pro models can record spatial videos. These 3D videos are saved in the Apple HEVC Stereo Video Profile format, an implementation of MV-HEVC that only Apple Vision Pro can play back natively.

The iPhone app Spatialify released last year with the flagship feature of converting spatial videos into regular side-by-side 3D that can be played on any existing VR headset. It shipped three months before Quest officially got the ability to easily convert and sync spatial videos from your iPhone to your headset for playback.

How To Watch iPhone 15 Pro Spatial Video On Any Headset
You can already watch iPhone 15 Pro 3D ‘Spatial Videos’ on the headset you already own. Here’s how:

Since the initial release Spatialify has gained the ability to record spatial videos within the app itself, not just converting those taken in Apple's Camera app. And the latest update adds support for HDR capture as well as either higher frame rate or higher resolution.

The Apple camera app can only record spatial videos at 1080p 30fps with standard dynamic range (SDR). The new v1.3 update for Spatialify on the other hand can record at either 1080p 60fps or 4K 30fps, and both modes support high dynamic range (HDR).

You can get Spatialify for $3 on the App Store.

Meta Is Only Allowing A Few Quest Developers To Disable The Annoying Boundary In Mixed Reality

Quest mixed reality apps can disable the annoying safety boundary, but only a handful of whitelisted developers can ship this on the Quest Store and App Lab.

Boundary, formerly called Guardian, is certainly useful in virtual reality so you don't leave your playspace and bump into furniture and walls. But in most mixed reality apps it's superfluous, since you can already see the environment around you, and downright annoying because it means you can't utilize your full room as a playspace.

In the v57 system software changelog Meta said "some apps with mixed reality" will no longer have Boundary. But the company didn't say which apps this included, nor the mechanism for this happening.

UploadVR has now learned that any Quest app can disable Boundary when using passthrough by including the CONTEXTUAL_BOUNDARYLESS_APP flag in the manifest. However, the upload system for the Quest Store and App Lab will automatically reject any app version using this flag unless they're on a special Meta whitelist.

Currently whitelisted apps include:

Digital Lode tells UploadVR that the boundary existing in mixed reality mode was the #1 complaint from Espire 2: Stealth Operatives customers before this update, resulting in many 1-star reviews and refunds.

Why Not Let All Apps Do This?

So why not let all developers use this flag and get rid of the annoying boundary in mixed reality? We asked Meta's VP of VR Mark Rabkin a similar question on X around a year ago.

Rabkin pointed out that some apps flow quickly between VR and MR, blurring the boundary between the two content types. And he does have a point. If an app uses passthrough as the background but is covering most of your view with virtual objects such that you can't see your walls and furniture, shouldn't the Boundary kick in?

However, Rabkin did agree that "if you're in a mode where you can see all around you", Guardian should be "a lot more chill".

Meta's solution for now is reviewing apps on a case-by-case basis to determine whether it's safe to disable Boundary. This seems deeply unscalable, however, and the company will have to find a better solution if it's serious about making Quest a mainstream mixed reality platform, and especially if it hopes people will one day wear its headset passively throughout the day.

That solution will likely include replacing Boundary with something better altogether - as was seen in clips found in the firmware in the months before Quest 3 launched.

Quest 3 ‘Smart Guardian’ Room Scanning Setup Leaked
Clips appearing to show room scanning setup for Quest 3’s mixed reality and ‘Smart Guardian’ found in firmware:

In contrast, Apple Vision Pro's approach to this problem is to fade all virtual elements to transparent if your head moves further than 1.5 meters from where you started or gets near real-world objects. This avoids a visible boundary, but limits you to a 3-meter diameter circle.

Apple's WWDC24 Takes Place June 10 And Will Include A Showcase Of "visionOS Advancements"

Apple's annual Worldwide Developers Conference (WWDC) will take place on June 10 this year.

WWDC24 will "spotlight the latest iOS, iPadOS, macOS, watchOS, tvOS, and visionOS advancements", Apple confirmed

The main event will be a free online livestream, and there will also be "an in-person experience" that includes "special activities".

visionOS 1.1 Is Out Now, Improving Personas And More
visionOS 1.1 for Apple Vision Pro is out now. It improves Personas, Mac Virtual Display, and more. Read all the notable changes here.

Apple Vision Pro launched on February 2 with visionOS 1.0. Earlier this month Apple released visionOS 1.1 with improvements to the realism of Personas, reliability of Mac Virtual Display, positioning of Volumes, and more.

Apple is expected to preview visionOS 2 at WWDC24, which could bring much more significant improvements and new features.

Meta Reportedly Still Plans To Bring Cheaper Version Of Quest 3 To China Via Tencent

Meta still plans to bring the upcoming cheaper version of Quest 3 to China via partnership with Tencent, The Information reports.

Cheaper Version Of Quest 3?

A Meta hardware roadmap meeting leaked to The Verge in March last year revealed the company planned to release a new headset after Quest 3 in 2024 "at the most attractive price point in the VR consumer market".

Reports from The Wall Street JournalBloomberg, and a Chinese analyst who has been reliable in the past suggest this headset will feature the Snapdragon XR2 Gen 2 chipset from Quest 3 but use the old fresnel lenses from Quest 2 to hit a low enough price to replace it.

XR2 Gen 2 has a more than twice as powerful GPU, which some developers are already using to deliver much better graphics.

This upcoming headset is rumored to be called 'Quest 3 Lite' or 'Quest 3S', though there's no strong evidence for any final name yet. Evidence found in the Quest firmware suggests it will also support color mixed reality.

Coming To China Via Tencent?

To sell many kinds of products in China, foreign companies must partner with a Chinese company or set up a local subsidiary. Meta and Tencent's aim to sell Quest headsets in China was first reported by Chinese news outlet 36Kr early last year, though that report claimed the product was planned to be Quest 2.

In November The Wall Street Journal reported that the partnership, which it described as "provisional", would now instead bring the upcoming cheaper version of Quest 3 to China instead. But in January a report from Sina Finance, citing another Chinese news outlet VRTUOLUO, claimed the partnership was suspended due to unresolved details on how to handle the specifics.

Now today, The Information's Wayne Ma reports that Meta and Tencent still plan to release the cheaper version of Quest 3 in the Chinese market, in the fourth quarter of this year. Ma has a good track record of accurately reporting the future moves of Meta and Apple.

The reported structure of the partnership was that Tencent would sell and support the Quest headset in China, while both companies would work together on localization and translation of Quest Store content. Meta would get most of the device revenue, while Tencent would get most of the content revenue. This wouldn’t be a novel arrangement for Tencent. It has already been selling and supporting the Nintendo Switch in the Chinese market since 2019.

Apple Vision Pro Is Coming To China This Year
Apple Vision Pro is coming to China later this year.

If the partnership truly happens, it could be significant competition for ByteDance's Pico, which plans to release an updated Pico 4S this year. It would also offer a significantly cheaper (though less capable) alternative to Apple Vision Pro, which is also coming to China later this year and will also feature Tencent-provided apps and services.

New Apple Pencil Model With Vision Pro Support Reportedly Being Tested By Apple

A new Apple Pencil that supports Vision Pro is being tested by Apple, according to MacRumors.

MacRumors cites "a source familiar with the matter" as saying Apple has internally tested such a stylus to allow drawing in visionOS apps like Freeform. The report doesn't however go into any specific detail on how the new device works.

Supporting the report's claims, in February a VR developer found a reference to a 'Stylus' in the Unity PolySpatial package for visionOS.

0:00
/0:17

Drawing in Vision Pro is currently possible, but not precise enough for professional workflows.

If this new Apple Pencil does eventually release as a product, could it perhaps allow any table to be turned into a huge drawing canvas? Or might it even support positional tracking, allowing for precise 3D artistry and animation of the kind that currently isn't practical given Vision Pro's lack of controllers?

For now, none of these answers are known. But the Vision headsets line seems poised to eventually supplant iPads, and gaining support for styluses would be an important step in that journey.

Apple Vision Pro Review: A Heavy Portable Cinema & Monitor
At $3500, Apple sets huge expectations with Vision Pro. But does it deliver? And what’s it really like to own and use? Read our in-depth Apple Vision Pro hardware & software review here:

Apple Vision Pro Is Coming To China This Year

Apple Vision Pro will launch in China later this year.

Reuters reports that Apple CEO Tim Cook revealed the news in response to a question at the China Development Forum in Beijing, according to China Central Television.

Vision Pro is currently only available in the US. Apple previously said it will come to other countries "later" in 2024, but hasn't yet confirmed which countries this will include.

Apple Vision Pro Review: A Heavy Portable Cinema & Monitor
At $3500, Apple sets huge expectations with Vision Pro. But does it deliver? And what’s it really like to own and use? Read our in-depth Apple Vision Pro hardware & software review here:

Earlier this month MacRumors found references to 12 new languages in visionOS code:

  • Cantonese, Traditional
  • Chinese, Simplified
  • English (Australia)
  • English (Canada)
  • English (Japan)
  • English (Singapore)
  • English (UK)
  • French (Canada)
  • French (France)
  • German (Germany)
  • Japanese
  • Korean

These findings suggest Vision Pro will soon launch in Australia, Canada, China, France, Germany, Japan, Singapore, South Korea, and the UK, and perhaps Hong Kong and Taiwan too.

Late last month prominent supply chain analyst Ming-Chi Kuo claimed that Apple will launch Vision Pro in more countries before June.

And last year Bloomberg's Mark Gurman, who reliably reported many details of Vision Pro before it was officially revealed or even acknowledged to exist, reported that Apple was "discussing the UK and Canada as two of its first international markets with Asia and Europe soon after" and was working on localizing the device for Australia, Germany, France, Japan, South Korea, and China.

Meta Reportedly Cancels Plan To Bring Quest 3 Lite To China
Meta has reportedly suspended its plan to bring a ‘Quest 3 Lite’ to China via a partnership with Tencent.

Meanwhile, Meta reportedly suspended a tentative partnership with Tencent to bring the upcoming cheaper version of Quest 3 to China due to "unresolved details on how to handle the specifics", a partnership that had been in the works for at least a year.

visionOS 2 Is Reportedly Coming This Year – Will It Bring 3D Spatial Personas?

visionOS 2.0 for Apple Vision Pro will release this year, Bloomberg's Mark Gurman reports.

Apple Vision Pro launched on February 2 with visionOS 1.0. Earlier this month Apple released visionOS 1.1 with improvements to the realism of Personas, reliability of Mac Virtual Display, positioning of Volumes, and more.

visionOS 1.1 Is Out Now, Improving Personas And More
visionOS 1.1 for Apple Vision Pro is out now. It improves Personas, Mac Virtual Display, and more. Read all the notable changes here.

Gurman writes that a more substantial visionOS 2.0 update will release this year and is codenamed "Constellation". visionOS 1.0 was codenamed Borealis, suggesting an astronomy theme for the codenames.

Apple usually shares details of entirely new versions of its operating systems at its annual WWDC conference, which takes place in June.

Spatial Personas?

While it's currently unknown what new features and improvements visionOS 2.0 might bring, there is one very significant visionOS feature previewed at last year's WWDC when Vision Pro was unveiled that has yet to ship: Spatial Personas.

Personas are Apple's realistic virtual avatars in visionOS, driven in real time by the headset's eye, face, and hand tracking sensors. Your Persona will appear in any iPad or iPhone app that requests the selfie camera, as well as visionOS apps that integrate Personas. Right now that visionOS integration is limited to showing the Persona inside a 2D rectangular window, as if a webcam view. But at WWDC 2023 the company teased Spatial Personas, 3D avatars that can be positioned in your space in AR or VR.

At the time Apple said Spatial Personas would arrive as a "developer preview" in late 2023 to allow developers using the visionOS simulator to build apps using them, but this has yet to happen even as of March 2024. That suggests Spatial Personas have been pushed out to visionOS 2.

0:00
/0:13

Apple's teaser of Spatial Personas from WWDC 2023.

Spatial Personas will enable you to have a shared environment with other Vision Pro users in both AR and fully immersive apps, seeing each other in a shared coordinate space with shared context instead of as detached webcam-like views.

Apple said it will offer three Spatial Persona positioning templates for developers: Side-By-Side, Surround, and Conversational.

  • Side-By-Side is ideal for windowed apps, including co-watching content like movies and TV shows.
  • Surround places the content in the center with users facing it in a circular arrangement. It's a better choice for viewing or interacting with volumetric content, such as playing a tabletop game or viewing a CAD model.
  • Conversational places the users in a semi-circle and the content off-center. Apple says this is for when social interaction is the primary point of the experience with the content acting as a background, such as playing music.

Apple has already architected visionOS to make synchronizing app states between users as easy as possible for developers. Spatial Personas, when they arrive, should be a step-change upgrade for using Vision Pro with other owners, opening up new multi-user use cases and significantly improving the sense of social presence in existing co-watching experiences.

Quest 3 Has Higher Retention Than Previous Headsets, Meta Confirms

Meta confirmed that Quest 3 owners have higher retention than owners of its previous headsets.

Industry speculation and analysis often focuses on headset unit sales, but what really matters to both Meta and developers is how many buyers actually regularly use the headset instead of leaving it on a shelf or in a drawer. By March 2023 Meta had sold nearly 20 million Quest headsets, and in October 2022 6.37 million owners reportedly actually used their headset that month.

At GDC 2024 this week Chris Pruett, Meta's "Director of Content Ecosystem", advised developers that Quest 3 owners return to continuously use apps at a higher rate.

Back in 2019 John Carmack said that the original Oculus Quest, which had launched 4 months earlier, had higher retention than any previous headset, including the PC-based Rift and Rift S.

Carmack: Quest Is ‘By Far Our Most Retentive Hardware’, Rift S Surpasses Rift
Facebook’s new VR headsets are already seeing impressive retention, according to John Carmack. During his keynote speech at Oculus Connect 6 today, Carmack announced that Oculus Quest is “by far” Facebook’s most retentive hardware. The Oculus Rift S, however, has already surpassed the original Rift in terms of

There are a number of possible explanations for Quest 3's higher retention, and the answer may even be a combination of multiple.

It could be due to the higher quality of experience delivered by the pancake lenses and more powerful chipset. Apps and games load slightly faster, look sharper and clearer, and often run at higher refresh rate, meaning they feel smoother and have less chance of making the user feel sick.

It could also be due to the improved passthrough. Quest 3's passable quality color view of the world lets owners interact with and stay aware of their surroundings without taking the headset off, and makes using the browser and watching YouTube more widely appealing.

There could also be a degree of selection here. People who are more engaged with VR are more likely to upgrade to Quest 3, and people who are willing to spend $500 or $650 on a headset may be more likely to continue to use it than those who paid much less.

Regardless of the reasons, this news is evidence that the retention problem is not inherent to XR headsets, and improved hardware over time can solve it.

Meta AI Research: Making Headsets And Glasses Understand Your Room Layout & Furniture

Meta's latest AI research system makes headsets and glasses understand your room layout and furniture.

Called SceneScript, Meta says the system leverages the same underlying technique as large language models (LLMs) except instead of predicting the next language fragment it predicts architectural and furniture elements within a 3D point cloud capture of the same kind that headsets already capture to enable standalone positional tracking.

The output is a series of primitive 3D shapes that represent the basic boundaries of the given furniture or element.

0:00
/0:45

Meta's SceneScript in action

Quest 3 is capable of generating a raw 3D mesh of your room, and can infer the positions of your walls, floor, and ceiling from this 3D mesh. But the headset doesn't currently know which shapes within this mesh represent more specific elements like doors, windows, tables, chairs, and sofas. Quest headsets do let users mark out these objects as simple rectangular cuboids manually if they want, but since this is optional and arduous developers can't rely on users having done so.

If a technology like SceneScript were integrated into Quest 3's mixed reality scene setup developers could automatically place virtual content on or around specific furniture elements. They could replace just your windows with portals, or your chair with a virtual seat, or change your sofa into a sandbagged position. These things are possible today by asking the user to mark the furniture out manually, or sometimes with manual algorithms interpreting the scene mesh, but SceneScript could make it seamless and automatic.

0:00
/0:20

Meta's SceneScript in action

Technology like SceneScript could also have huge potential for AI assistants in future headsets and its eventual AR glasses. Meta gives the example of questions like “Will this desk fit in my bedroom?” or “How many pots of paint would it take to paint this room?”, as well as commands like “Place the [AR/MR app] on the large table”.

For now though Meta is describing SceneScript purely as research, not a near-term product feature update. While this technology may eventually come to Quest 3, there's no suggestion of that happening any time soon.

ALVR For Apple Vision Pro Is Now Available To Download & Forwards Hand Tracking To SteamVR

ALVR, the app that enables Apple Vision Pro to work on SteamVR, is now available for anyone to download.

If you're interested in VR you're probably familiar with the three most popular tools to make Meta Quest headsets act as PC VR headsets on Windows: Virtual Desktop, Steam Link, and Quest (Air) Link. But you might not be as familiar with ALVR, the open-source alternative that has been around since before any of those.

Apple Vision Pro Demonstrated Working On SteamVR Via ALVR
A developer got Apple Vision Pro working on SteamVR by porting the open-source app ALVR, though this initial implementation has some issues.

In early February we reported that ALVR had been successfully ported to visionOS by software developer Zhuowei Zhang. Though this initial port was rough with many issues, developer another programmer, Max Thomas, has been working on improving the code in the six weeks since.

Thomas has significantly improved the app's netcode to minimize jitter and hitches, increased the refresh rate to 96Hz, implemented HDR (High Dynamic Range) display output, and added support for passing Vision Pro's hand tracking through to SteamVR Skeletal Input.

And while before now using ALVR on Vision Pro required compiling the source code on a Mac using Xcode, the app has now been approved for easy download on TestFlight. If you're unfamiliar with the Apple ecosystem, TestFlight is essentially Apple's equivalent of Quest's App Lab.

Being approved on TestFlight suggests Virtual Desktop and iVRy will also be approved - both developers plan to release apps for Vision Pro.

0:00
/1:08

YouTuber Brad Lynch (SadlyItsBradley) playing Half-Life: Alyx on Apple Vision Pro with hand gestures.

Of course, the lack of tracked controllers mean Apple Vision Pro isn't a practical way to play the majority of SteamVR games out of the box. SteamVR Skeletal Input is primarily intended for the finger tracking capabilities of controllers like Valve Index, which include thumbsticks, triggers, and buttons. ALVR for Vision Pro allows you to emulate these controls via a series of gestures, but this is obviously incredibly clunky for most games.

What you can do is provide these inputs via Bluetooth controllers such as Nintendo Joy-Cons. Their inputs are passed through to SteamVR, and haptics are passed back.

All of this only works in apps that support SteamVR Skeletal Input, however. Most VR apps on Steam do not, including VRChat. And even where it works, the tracking quality is much lower than with real VR controllers like Meta Quest Touch.

0:00
/0:27

Thomas using Nintendo Joy-Conns with Apple Vision Pro to play Valve's The Lab.

To play most SteamVR games you could add SteamVR Tracking base stations and Valve Index controllers and use a tool like OpenVR Space Calibrator to manually align them. But that equipment would cost you around $600 if you don't own it already, and the alignment would need to be performed manually each time.

If you're not willing to use these methods, Vision Pro could still prove an excellent headset for use with untracked input devices, such as racing wheels for sim racing or HOTAS setups for flight simulators.

You can download ALVR from TestFlight now.