The Spatialify app for iPhone 15 Pro can now record 3D 'spatial videos' with HDR at 1080p 60fps or 4K 30fps.
Spatial Video is Apple's term for stereoscopic 3D video on its devices. Currently, only Apple Vision Pro and the two iPhone 15 Pro models can record spatial videos. These 3D videos are saved in the Apple HEVC Stereo Video Profile format, an implementation of MV-HEVC that only Apple Vision Pro can play back natively.
The iPhone app Spatialify released last year with the flagship feature of converting spatial videos into regular side-by-side 3D that can be played on any existing VR headset. It shipped three months before Quest officially got the ability to easily convert and sync spatial videos from your iPhone to your headset for playback.
Since the initial release Spatialify has gained the ability to record spatial videos within the app itself, not just converting those taken in Apple's Camera app. And the latest update adds support for HDR capture as well as either higher frame rate or higher resolution.
The Apple camera app can only record spatial videos at 1080p 30fps with standard dynamic range (SDR). The new v1.3 update for Spatialify on the other hand can record at either 1080p 60fps or 4K 30fps, and both modes support high dynamic range (HDR).
Quest mixed reality apps can disable the annoying safety boundary, but only a handful of whitelisted developers can ship this on the Quest Store and App Lab.
Boundary, formerly called Guardian, is certainly useful in virtual reality so you don't leave your playspace and bump into furniture and walls. But in most mixed reality apps it's superfluous, since you can already see the environment around you, and downright annoying because it means you can't utilize your full room as a playspace.
In the v57 system software changelog Meta said "some apps with mixed reality" will no longer have Boundary. But the company didn't say which apps this included, nor the mechanism for this happening.
UploadVR has now learned that any Quest app can disable Boundary when using passthrough by including the CONTEXTUAL_BOUNDARYLESS_APP flag in the manifest. However, the upload system for the Quest Store and App Lab will automatically reject any app version using this flag unless they're on a special Meta whitelist.
Currently whitelisted apps include:
Cubism, which has seemingly been using this flag since Quest 3 launch.
Meta's own First Encounters demo, the introductory experience for Quest 3, which implemented this flag last month.
Laser Dance, the upcoming room-scale mixed reality game from the developer of Cubism.
Digital Lode tells UploadVR that the boundary existing in mixed reality mode was the #1 complaint from Espire 2: Stealth Operatives customers before this update, resulting in many 1-star reviews and refunds.
Why Not Let All Apps Do This?
So why not let all developers use this flag and get rid of the annoying boundary in mixed reality? We asked Meta's VP of VR Mark Rabkin a similar question on X around a year ago.
Rabkin pointed out that some apps flow quickly between VR and MR, blurring the boundary between the two content types. And he does have a point. If an app uses passthrough as the background but is covering most of your view with virtual objects such that you can't see your walls and furniture, shouldn't the Boundary kick in?
However, Rabkin did agree that "if you're in a mode where you can see all around you", Guardian should be "a lot more chill".
Meta's solution for now is reviewing apps on a case-by-case basis to determine whether it's safe to disable Boundary. This seems deeply unscalable, however, and the company will have to find a better solution if it's serious about making Quest a mainstream mixed reality platform, and especially if it hopes people will one day wear its headset passively throughout the day.
That solution will likely include replacing Boundary with something better altogether - as was seen in clips found in the firmware in the months before Quest 3 launched.
In contrast, Apple Vision Pro's approach to this problem is to fade all virtual elements to transparent if your head moves further than 1.5 meters from where you started or gets near real-world objects. This avoids a visible boundary, but limits you to a 3-meter diameter circle.
Apple's annual Worldwide Developers Conference (WWDC) will take place on June 10 this year.
WWDC24 will "spotlight the latest iOS, iPadOS, macOS, watchOS, tvOS, and visionOS advancements", Apple confirmed
The main event will be a free online livestream, and there will also be "an in-person experience" that includes "special activities".
Apple Vision Pro launched on February 2 with visionOS 1.0. Earlier this month Apple released visionOS 1.1 with improvements to the realism of Personas, reliability of Mac Virtual Display, positioning of Volumes, and more.
Apple is expected to preview visionOS 2 at WWDC24, which could bring much more significant improvements and new features.
Meta still plans to bring the upcoming cheaper version of Quest 3 to China via partnership with Tencent, The Information reports.
Cheaper Version Of Quest 3?
A Meta hardware roadmap meeting leaked to The Verge in March last year revealed the company planned to release a new headset after Quest 3 in 2024 "at the most attractive price point in the VR consumer market".
Reports from The Wall Street Journal, Bloomberg, and a Chinese analyst who has been reliable in the past suggest this headset will feature the Snapdragon XR2 Gen 2 chipset from Quest 3 but use the old fresnel lenses from Quest 2 to hit a low enough price to replace it.
XR2 Gen 2 has a more than twice as powerful GPU, which some developers are already using to deliver much better graphics.
This upcoming headset is rumored to be called 'Quest 3 Lite' or 'Quest 3S', though there's no strong evidence for any final name yet. Evidence found in the Quest firmware suggests it will also support color mixed reality.
Coming To China Via Tencent?
To sell many kinds of products in China, foreign companies must partner with a Chinese company or set up a local subsidiary. Meta and Tencent's aim to sell Quest headsets in China was first reported by Chinese news outlet 36Kr early last year, though that report claimed the product was planned to be Quest 2.
In November The Wall Street Journal reported that the partnership, which it described as "provisional", would now instead bring the upcoming cheaper version of Quest 3 to China instead. But in January a report from Sina Finance, citing another Chinese news outlet VRTUOLUO, claimed the partnership was suspended due to unresolved details on how to handle the specifics.
Now today, The Information's Wayne Ma reports that Meta and Tencent still plan to release the cheaper version of Quest 3 in the Chinese market, in the fourth quarter of this year. Ma has a good track record of accurately reporting the future moves of Meta and Apple.
The reported structure of the partnership was that Tencent would sell and support the Quest headset in China, while both companies would work together on localization and translation of Quest Store content. Meta would get most of the device revenue, while Tencent would get most of the content revenue. This wouldn’t be a novel arrangement for Tencent. It has already been selling and supporting the Nintendo Switch in the Chinese market since 2019.
If the partnership truly happens, it could be significant competition for ByteDance's Pico, which plans to release an updated Pico 4S this year. It would also offer a significantly cheaper (though less capable) alternative to Apple Vision Pro, which is also coming to China later this year and will also feature Tencent-provided apps and services.
A new Apple Pencil that supports Vision Pro is being tested by Apple, according to MacRumors.
MacRumors cites "a source familiar with the matter" as saying Apple has internally tested such a stylus to allow drawing in visionOS apps like Freeform. The report doesn't however go into any specific detail on how the new device works.
Supporting the report's claims, in February a VR developer found a reference to a 'Stylus' in the Unity PolySpatial package for visionOS.
If this new Apple Pencil does eventually release as a product, could it perhaps allow any table to be turned into a huge drawing canvas? Or might it even support positional tracking, allowing for precise 3D artistry and animation of the kind that currently isn't practical given Vision Pro's lack of controllers?
For now, none of these answers are known. But the Vision headsets line seems poised to eventually supplant iPads, and gaining support for styluses would be an important step in that journey.
Apple Vision Pro will launch in China later this year.
Reuters reports that Apple CEO Tim Cook revealed the news in response to a question at the China Development Forum in Beijing, according to China Central Television.
Vision Pro is currently only available in the US. Apple previously said it will come to other countries "later" in 2024, but hasn't yet confirmed which countries this will include.
Earlier this month MacRumors found references to 12 new languages in visionOS code:
Cantonese, Traditional
Chinese, Simplified
English (Australia)
English (Canada)
English (Japan)
English (Singapore)
English (UK)
French (Canada)
French (France)
German (Germany)
Japanese
Korean
These findings suggest Vision Pro will soon launch in Australia, Canada, China, France, Germany, Japan, Singapore, South Korea, and the UK, and perhaps Hong Kong and Taiwan too.
Late last month prominent supply chain analyst Ming-Chi Kuo claimed that Apple will launch Vision Pro in more countries before June.
And last year Bloomberg's Mark Gurman, who reliably reported many details of Vision Pro before it was officially revealed or even acknowledged to exist, reported that Apple was "discussing the UK and Canada as two of its first international markets with Asia and Europe soon after" and was working on localizing the device for Australia, Germany, France, Japan, South Korea, and China.
Meanwhile, Meta reportedly suspended a tentative partnership with Tencent to bring the upcoming cheaper version of Quest 3 to China due to "unresolved details on how to handle the specifics", a partnership that had been in the works for at least a year.
visionOS 2.0 for Apple Vision Pro will release this year, Bloomberg's Mark Gurman reports.
Apple Vision Pro launched on February 2 with visionOS 1.0. Earlier this month Apple released visionOS 1.1 with improvements to the realism of Personas, reliability of Mac Virtual Display, positioning of Volumes, and more.
Gurman writes that a more substantial visionOS 2.0 update will release this year and is codenamed "Constellation". visionOS 1.0 was codenamed Borealis, suggesting an astronomy theme for the codenames.
Apple usually shares details of entirely new versions of its operating systems at its annual WWDC conference, which takes place in June.
Spatial Personas?
While it's currently unknown what new features and improvements visionOS 2.0 might bring, there is one very significant visionOS feature previewed at last year's WWDC when Vision Pro was unveiled that has yet to ship: Spatial Personas.
Personas are Apple's realistic virtual avatars in visionOS, driven in real time by the headset's eye, face, and hand tracking sensors. Your Persona will appear in any iPad or iPhone app that requests the selfie camera, as well as visionOS apps that integrate Personas. Right now that visionOS integration is limited to showing the Persona inside a 2D rectangular window, as if a webcam view. But at WWDC 2023 the company teased Spatial Personas, 3D avatars that can be positioned in your space in AR or VR.
At the time Apple said Spatial Personas would arrive as a "developer preview" in late 2023 to allow developers using the visionOS simulator to build apps using them, but this has yet to happen even as of March 2024. That suggests Spatial Personas have been pushed out to visionOS 2.
Spatial Personas will enable you to have a shared environment with other Vision Pro users in both AR and fully immersive apps, seeing each other in a shared coordinate space with shared context instead of as detached webcam-like views.
Apple said it will offer three Spatial Persona positioning templates for developers: Side-By-Side, Surround, and Conversational.
Side-By-Side is ideal for windowed apps, including co-watching content like movies and TV shows.
Surround places the content in the center with users facing it in a circular arrangement. It's a better choice for viewing or interacting with volumetric content, such as playing a tabletop game or viewing a CAD model.
Conversational places the users in a semi-circle and the content off-center. Apple says this is for when social interaction is the primary point of the experience with the content acting as a background, such as playing music.
Apple has already architected visionOS to make synchronizing app states between users as easy as possible for developers. Spatial Personas, when they arrive, should be a step-change upgrade for using Vision Pro with other owners, opening up new multi-user use cases and significantly improving the sense of social presence in existing co-watching experiences.
Meta confirmed that Quest 3 owners have higher retention than owners of its previous headsets.
Industry speculation and analysis often focuses on headset unit sales, but what really matters to both Meta and developers is how many buyers actually regularly use the headset instead of leaving it on a shelf or in a drawer. By March 2023 Meta had sold nearly 20 million Quest headsets, and in October 2022 6.37 million owners reportedly actually used their headset that month.
At GDC 2024 this week Chris Pruett, Meta's "Director of Content Ecosystem", advised developers that Quest 3 owners return to continuously use apps at a higher rate.
Back in 2019 John Carmack said that the original Oculus Quest, which had launched 4 months earlier, had higher retention than any previous headset, including the PC-based Rift and Rift S.
There are a number of possible explanations for Quest 3's higher retention, and the answer may even be a combination of multiple.
It could be due to the higher quality of experience delivered by the pancake lenses and more powerful chipset. Apps and games load slightly faster, look sharper and clearer, and often run at higher refresh rate, meaning they feel smoother and have less chance of making the user feel sick.
It could also be due to the improved passthrough. Quest 3's passable quality color view of the world lets owners interact with and stay aware of their surroundings without taking the headset off, and makes using the browser and watching YouTube more widely appealing.
There could also be a degree of selection here. People who are more engaged with VR are more likely to upgrade to Quest 3, and people who are willing to spend $500 or $650 on a headset may be more likely to continue to use it than those who paid much less.
Regardless of the reasons, this news is evidence that the retention problem is not inherent to XR headsets, and improved hardware over time can solve it.
Meta's latest AI research system makes headsets and glasses understand your room layout and furniture.
Called SceneScript, Meta says the system leverages the same underlying technique as large language models (LLMs) except instead of predicting the next language fragment it predicts architectural and furniture elements within a 3D point cloud capture of the same kind that headsets already capture to enable standalone positional tracking.
The output is a series of primitive 3D shapes that represent the basic boundaries of the given furniture or element.
Quest 3 is capable of generating a raw 3D mesh of your room, and can infer the positions of your walls, floor, and ceiling from this 3D mesh. But the headset doesn't currently know which shapes within this mesh represent more specific elements like doors, windows, tables, chairs, and sofas. Quest headsets do let users mark out these objects as simple rectangular cuboids manually if they want, but since this is optional and arduous developers can't rely on users having done so.
If a technology like SceneScript were integrated into Quest 3's mixed reality scene setup developers could automatically place virtual content on or around specific furniture elements. They could replace just your windows with portals, or your chair with a virtual seat, or change your sofa into a sandbagged position. These things are possible today by asking the user to mark the furniture out manually, or sometimes with manual algorithms interpreting the scene mesh, but SceneScript could make it seamless and automatic.
Technology like SceneScript could also have huge potential for AI assistants in future headsets and its eventual AR glasses. Meta gives the example of questions like “Will this desk fit in my bedroom?” or “How many pots of paint would it take to paint this room?”, as well as commands like “Place the [AR/MR app] on the large table”.
For now though Meta is describing SceneScript purely as research, not a near-term product feature update. While this technology may eventually come to Quest 3, there's no suggestion of that happening any time soon.
ALVR, the app that enables Apple Vision Pro to work on SteamVR, is now available for anyone to download.
If you're interested in VR you're probably familiar with the three most popular tools to make Meta Quest headsets act as PC VR headsets on Windows: Virtual Desktop, Steam Link, and Quest (Air) Link. But you might not be as familiar with ALVR, the open-source alternative that has been around since before any of those.
In early February we reported that ALVR had been successfully ported to visionOS by software developer Zhuowei Zhang. Though this initial port was rough with many issues, developer another programmer, Max Thomas, has been working on improving the code in the six weeks since.
Thomas has significantly improved the app's netcode to minimize jitter and hitches, increased the refresh rate to 96Hz, implemented HDR (High Dynamic Range) display output, and added support for passing Vision Pro's hand tracking through to SteamVR Skeletal Input.
And while before now using ALVR on Vision Pro required compiling the source code on a Mac using Xcode, the app has now been approved for easy download on TestFlight. If you're unfamiliar with the Apple ecosystem, TestFlight is essentially Apple's equivalent of Quest's App Lab.
Being approved on TestFlight suggests Virtual Desktop and iVRy will also be approved - both developers plan to release apps for Vision Pro.
Of course, the lack of tracked controllers mean Apple Vision Pro isn't a practical way to play the majority of SteamVR games out of the box. SteamVR Skeletal Input is primarily intended for the finger tracking capabilities of controllers like Valve Index, which include thumbsticks, triggers, and buttons. ALVR for Vision Pro allows you to emulate these controls via a series of gestures, but this is obviously incredibly clunky for most games.
What you can do is provide these inputs via Bluetooth controllers such as Nintendo Joy-Cons. Their inputs are passed through to SteamVR, and haptics are passed back.
All of this only works in apps that support SteamVR Skeletal Input, however. Most VR apps on Steam do not, including VRChat. And even where it works, the tracking quality is much lower than with real VR controllers like Meta Quest Touch.
To play most SteamVR games you could add SteamVR Tracking base stations and Valve Index controllers and use a tool like OpenVR Space Calibrator to manually align them. But that equipment would cost you around $600 if you don't own it already, and the alignment would need to be performed manually each time.
If you're not willing to use these methods, Vision Pro could still prove an excellent headset for use with untracked input devices, such as racing wheels for sim racing or HOTAS setups for flight simulators.