Showcased at Apple’s Worldwide Developer Conference this week, ARKit is a new core technology for iOS 11, due to launch this Fall, soon to enable augmented reality features on hundreds of millions of iPhones and iPads. As the iOS 11 developer beta is already available, we’re starting to see some interesting real-world tests of ARKit, showing off the tracking that’s achievable with nothing more than a camera.
Developer Cody Brown hacked together a quick demo using Overwatch assets as a ‘hello world’ test of ARKit running on an iPhone 6S:
Perhaps the most impressive aspect of that demo was that it was running on an iPad, using the single camera on the back of the device for tracking. This appeared to deliver fairly stable tracking, without the need for dedicated hardware, unlike Google Tango, which uses a suite of cameras and sensors.
Now that developers have their hands on ARKit, the early real-world tests are very promising, such as this clip of the Unity sample demo showing tracking points and plane estimation:
This video from Austrian augmented reality company ViewAR puts the technology through a demanding tracking test, covering the camera, moving quickly away from the virtual object, through multiple rooms with different lighting conditions to check for drift. The result is remarkable considering the limitations of using a single camera:
Apple is believed to be hard at work on AR technologies, and is likely to make screen-based AR a key selling point of the next iPhone, which is anticipated to have a near bezel-free design, which would certainly enhance the appearance of AR features.
Apple’s annual Worldwide Developer Conference (WWDC) is here, and today’s keynote saw a number of VR-specific announcements including Apple’s first VR-ready computers to go along with the launch of the company’s newest macOS High Sierra. While the company is finally going ‘VR-native’ for desktop, Apple is also zeroing in on augmented reality for iOS 11 with the entrance of their newly revealed app developer kit ‘ARKit’.
Possibly taking a swipe at Facebook’s latest AR demo at F8 in April, Senior VP of software engineering Craig Federighi said: “We’ve all seen a lot of carefully edited vision videos on this topic recently, but in the case, I’d like to show you something for real.”
Starting up a test application that will be made available to developers, Federighi explains that with the iPhone’s computer visions capabilities it’s able to map surfaces and add digital objects—replete with interactive animations and dynamic lighting. Adding a steaming coffee cup, a lamp and a vase to a bare, marker-less table, the tracking proves to be relatively solid.
Federighi says that ARKit provides fast and stable motion-tracking, plane estimation with basic boundaries, ambient lighting estimation, scale estimation, support for Unity, Unreal, SceneKit and Xcode app templates—all available on “hundred of millions of iPhones and iPads […] making overnight ARKit the largest AR platform in the world.”
Apple says iOS 11 will be made available to iPhone 5s and later, all iPad Air and iPad Pro models, iPad 5th generation, iPad mini 2 and later, and iPod touch 6th generation. iOS 11 will be released this fall, likely in tandem with iPhone 8 and iPhone 7S smartphones. A public beta is coming in June.
image courtesy Apple
Apple is working with third-parties such as IKEA, Lego, and Niantic to use ARKit, with Apple showing an improved Pokémon Go on stage that looks to actually utilize augmented reality to bring the game to life. Because ARKit uses computer vision that relies on the device’s onboard sensors and CPU/GPU, no external equipment is required to run these sorts of AR experiences.
The keynote also revealed a new AR-focused company from critically-acclaimed director and FX guru Peter Jackson called ‘Wingnut AR’. A special demo showed off the graphical and camera-based tracking capabilities of Apple’s hardware featuring a complex, real-time rendered scene digitally placed on a tabletop using an iPad. Wingnut AR is bringing an AR experience to the App Store later this year. Check out the video below to see Wingnut AR’s special Apple demo.
Apple CEO Tim Cook has once again affirmed his belief in the significance of augmented reality technology. Interviewed during a recent trip to Europe, he spoke broadly about how AR could benefit everyone’s lives in the future, highlighting the way it allows users to remain ‘present’ in the real world. Underscored by years of relevant hiring and R&D, Cook’s words appear to further set the stage for the company’s rumored foray into the AR space.
Despite many of the company’s biggest rivals and allies showing active development in the AR/VR space, Apple has yet to make a firm commitment toward or away from these immersive technologies, but there’s no doubt that the company has major research and development going on behind the scenes. The Cupertino HQ has been acquiring talent in this field for several years, filing patents at least as far back as 2007 for HMDs, and new patents for AR. Having hired top VR/AR researcher Doug Bowman a year ago, many suspect it’s only a matter of time before Apple shows their AR/VR hand.
Apple CEO Tim Cook | Photo courtesy Apple
In an interview by The Independent during a recent European tour, CEO Tim Cook reaffirmed his enthusiasm for AR, positioning it as an antithesis of VR in the way that it allows the user to remain ‘present in the world’, rather than closing the world out.
“I’m excited about augmented reality because unlike virtual reality which closes the world out, AR allows individuals to be present in the world but hopefully allows an improvement on what’s happening presently,” Cook said. “Most people don’t want to lock themselves out from the world for a long period of time and today you can’t do that because you get sick from it. With AR you can, not be engrossed in something, but have it be a part of your world, of your conversation. That has resonance.”
While we don’t agree with Cook’s assessment of AR and VR being necessarily opposite technologies (or that VR makes people sick in competently designed experiences), we can understand his view that AR’s focus on what’s already around us means the tech has the potential to integrate well with user’s daily lives (which is surely the realm where Apple likes its products to play).
Cook went on to describe AR’s potential to be as significant as the smartphone, a technology that could improve everyone’s lives.
“I regard [AR] as a big idea like the smartphone. The smartphone is for everyone, we don’t have to think the iPhone is about a certain demographic, or country or vertical market: it’s for everyone. I think AR is that big, it’s huge,” Cook told The Independent. “I get excited because of the things that could be done that could improve a lot of lives. And be entertaining. I view AR like I view the silicon here in my iPhone, it’s not a product per se, it’s a core technology. But there are things to discover before that technology is good enough for the mainstream. I do think there can be a lot of things that really help people out in daily life, real-life things, that’s why I get so excited about it.”
This isn’t the first time that Apple has appeared to show more interest in augmented reality than virtual reality, although given the challenges still facing AR, it could be argued that it would be easier for Apple to launch a VR-ready product first, perhaps a feature we’ll see promoted with the next generation of iPhone, which is expected to use an OLED panel.
Steve Jobs once said to a gathering of senior advisers that when it came to the first iPhone, “we’re going to patent it all,” (New York Times reports). While Apple has certainly changed in the years since Job’s passing, the company has undoubtedly continued to aggressively pursue patents, sometimes racking up hundreds of filings in a single month with inventions spanning everything from touch-sensitive smartwatch bands to 3D environmental mapping. This week however, the company was granted two patents that establish basic hardware and software solutions not only pointing towards a prospective Apple AR device, but marking out some fundamental territory in the process.
Both patents, spotted byApple Insider, were first filed by Metaio GmbH in 2014, a German AR firm later acquired by Apple just a year later.
Patent number 9,560,273, entitled Wearable information system having at least one camera, details a wearable device with a “display attached to his head in front of his eyes” that can use either an onboard SoC or in a wirelessly connected smartphone. The patent goes on to detail the invention’s reactive power mode, letting it switch between a battery-saving low-power mode and an active high-power mode that automatically reacts when a real-world object is matched with a reference object.
Apple AR patent detailing methods for computing 3D coordinates, image courtesy USPTO
In the figure below, the patent shows off a Google Glass-style configuration that is “especially well suited to be used with head mounted displays and a camera pointed at the space in front of the user.”
Patent 9,558,581, entitled Method for representing virtual information in a real environment, delves into the software side of AR, and is centered around virtual info overlays and proper occlusion perception.
The setup proposed in the patent paves way for a device that creates a 3D geometry model of the real environment and shows virtual information in both an occlusion mode and a non-occlusion mode, meaning objects are correctly presented to the user even if they’re hiding behind real-world obstacles. And to that, patent ‘581’ focuses on different ways of making virtual information distinct from the real world so users can quickly differentiate the real environment from the superimposed images. The patent proposes a few ways of doing this, including showing real-world objects as semi-transparent when they’re occluded by virtual info (great for driving) and using dashed lines to indicate a separation from the real and virtual.
Apple’s method of creating “proper occlusion perception,” image courtesy USPTO
While this all seems fairly basic, these sorts of patents help lay an important defensive framework that can help against future disputes from other manufacturers working in the sector; eg. Apple v. Samsung, Apple v. Microsoft, Apple v. Nokia, the list goes on. And if the recent ZeniMax v. Facebook lawsuit has taught us anything, you don’t ever want to be the one writing that half-billion dollar check after you already shelled out $2 billion for the intellectual property in the first place.
So while not giant revelations on their own, this, alongside Apple’s ever-growing library of AR patents, certainly lends credence to the idea that Apple is investing serious time and money in the field, and it doesn’t look to be ‘just in case’.
Apple’s AR Investments
Just one year after the launch of the iPhone, Apple started patenting head mounted displays with features not unlike the VR headsets of today. Of course, at the time it wasn’t feasible to produce such a unit, but again, patents aren’t really for that purpose anyway.
The company has since patented various AR/VR headset designs, like it’s Gear VR-esque phone-based headset, or the most recent patent in summer 2016 outlining a high FOV AR display.
Apple’s recent acquisitions have also pointed towards a heightened interest in AR. Besides acquiring Metaio, Apple has also taken on marker-less facial mapping and animation firm Faceshift, facial expression recognition & analysis company Emotient, and Flyby Media, a large-scale SLAM, indoor navigation, sensor fusion, image recognition, and 3D tracking company.
Another nail in the coffin: around 2014, Apple started hiring specifically for AR/VR software experts at their Cupertino HQ, first for an ‘App Engineer’ who was tasked to “create high performance apps that integrate with Virtual Reality systems for prototyping and user testing,” and then a month later four additional hires with immersive tech talent, including a ‘VR/AR Programmer’.
As we know it, Apple is a perennial ‘johnny-come-lately’ to established tech, preferring to see markets mature before they enter in with a big bet like the iPhone or Apple Watch, and the company seems famously stand-offish when it comes to VR. According to a report by Vanity Fair, Apple CEO Tim Cook says that virtual reality isn’t actually the way to go in his opinion. “Virtual reality sort of encloses and immerses the person into an experience that can be really cool but probably has a lower commercial interest over time,” said Cook.
So if VR isn’t in Apple’s commercial interest currently, the evidence thus far suggests that the company is heading eye balls-first into augmented reality. And we can’t wait to see what the world renowned electronics giant has in store.
To many, the question of an Apple AR or VR headset has become a “when” rather than an “if”. The President of Sony’s Worldwide Studios thinks 2017 is the year that Apple introduces its first immersive device.
Apple does R&D on a wide range of technologies and has been actively researching the fields of VR and AR for years, including submitting and receiving several relevant patents. And while much of Apple’s R&D doesn’t see the light of day, the company certainly excels at taking novel tech and marketing it as something that everyone can use. VR and AR are on the rise, and Apple is widely expected to jump into the immersive device space when the time is right. But exactly when that time is has been up for debate.
As for Shuhei Yoshida, President of Sony’s Worldwide Studios, 2017 is likely the year that Apple makes its first move. That’s according to Virtual Reality Pop, who queried a number of VR and AR industry insiders in a brief Q&A about their biggest predictions for the landscape in 2017.
Yoshia is a major believer in VR and has been closely involved with the creation of Sony’s PlayStation VR headset; he was the one to introduce the device (formerly called Project Morpheus) to the public for the first time at GDC 2014 (see leading photo). Since then he’s appeared numerous times to herald the headset and has carefully followed the evolution of the VR and AR landscape by attending and participating the industry’s top conferences.
It’s doubtful that Yoshida has any specific knowledge of an Apple AR or VR device announcement, but his prediction certainly contains the wisdom of a long time Sony veteran who is carefully considering PlayStation’s forward-looking VR strategy with regards to Apple’s possible entry into the marketplace.
Do you think Apple will come to the market with an AR/VR product this year? Let us know in the comments.
Robert Scoble und Austin Smith von Upload VR sprechen über mögliche neue AR/VR Produkte – auch von Apple. Sehr interessanter Video-Beitrag über künftige Entwicklungen und Perspektiven. ***** XING-Gruppe “Augmented Reality & Marketing” ***** ***** AugmentedRealityBiz.com E-Mail Newsletter ***** ***** Markus Caspari auf Twitter folgen ****
Apple CEO Tim Cook has stated that in his view, augmented reality and not virtual reality holds the ticket for commercial success in immersive technologies in the future.
Apple CEO Tim Cook [Image courtesy Mashable]On the subject of virtual and augmented reality Apple is still largely ‘mum’ on it’s intended direction, even in the wake of its latest hardware launch, the iPhone 7. As its largest competitors trail-blaze a particular path, Apple is content to be Apple and survey the chaos and decide to do things when it’s good and ready. This is what Apple has always done for the most part, and they’ve done very nicely as a result.
Up until recently we’ve only had circumstantial evidence (significant amounts mind you) that Apple is working towards an immersive technology product. Numerous acquisitions and hires point to a company who recognises mobile technology, such as its iPhone, is now reaching or indeed past its zenith in commercial potential, and (much like former mobile giant HTC has) needs to find another route to remaining relevant in the space in the future.
Now, in an interview with Good Morning America (as reported by Vanity Fair), Tim Cook has stated that, in his (and therefore Apple’s) opinion, virtual reality as he sees it is far less attractive to consumers than its close relative augmented reality. “Virtual reality sort of encloses and immerses the person into an experience that can be really cool but probably has a lower commercial interest over time,” said Cook.
Augmented reality on the other hand “gives the capability for both of us to sit and be very present, talking to each other, but also have other things—visually—for both of us to see,” says Cook, “Maybe it’s something we’re talking about, maybe it’s someone else here who’s not here present but who can be made to appear to be present.”
Cook has already gone on record with hints that Apple are working towards an immersive future, although recent comments were more supportive of virtual reality as a technology. Recode reported that Cook said on VR’s “geeky niche” image “In terms of virtual reality, no, I don’t think it’s a niche,” Cook replied. “I think it can be … it’s really cool and has some interesting applications.”
Apple meanwhile are still making significant, strategic hires which seemingly indicative of their growing commitment to an immersive future. Macrumors reported yesterday that the company have recruited two new team member, Zeyu Li from Magic Leap and Yury Petrov from Oculus. Li, who formerly served as “Principal Computer Vision Engineer” at Magic Leap has joined Apple as “Senior Computer Vision Algorithm Engineer”, which could indicate a focus on anything from inside out mobile tracking to hand gesture input. Petrov meanwhile has joined as a “Research Scientist”, having held a similar title at Oculus.
Given Apple’s biggest hardware launch this year, the iPhone 7, is now out of the way, it seems unlikely we’ll learn more about the technology giant’s plans for AR or VR this year. But, with Google’s Android-integrated Daydream VR platform due to drop very soon, Apple will indeed be playing catch up with their competitors from hereon out. But then, that’s what they do best.
Posted earlier this summer, yet another Apple job listing seeks candidates with AR and VR proficiency.
Apple says that their ‘Game Technologies Engineer’ role falls under the company’s Interactive Media Group, which “provides the media and graphics software foundation across all of Apple’s innovative products, such as iPhone, iPod touch, Apple TV, iTunes, and Mac OS X, as well as professional and consumer applications from Final Cut and Aperture to iLife and iWork.” Game Technologies Engineers create the tools and capabilities which allow developers to make great games on Apple platforms.
A job listing from the company is looking for such an engineer who is “comfortable with AR and VR concepts,” along with familiarity with programming languages and graphics pipelines commonly employed by Apple.
While the AR/VR mention is only a bullet point in the ‘Additional Requirements’ section of the job description, this job listing is just one of several spotted over the last few years that indicate the company’s continued behind-the-scenes work in the augmented and virtual reality fields. In 2014 the company posted an entirely unambiguous job listing for a ‘VR/AR Programer’ who would “develop software and tools that use VR and AR to push the state of the art to enable development of Apple’s next generation of products.”
This new job listing, along with a trail of prior evidence, makes it clear that Apple is spending considerable R&D time on virtual and augmented reality technologies.
Yesterday Apple held a keynote presentation to update the world on their latest products. While the company didn’t have any announcements on the AR/VR front, it’s clear that it’s only a matter of time before they enter the fray.
It’s too early for Apple to jump into the AR/VR game. That much was made clear on stage at the company’s latest keynote where they revealed the iPhone 7 and the Apple Watch Series 2. As an electronics company, Apple’s general strategy seems to be to wait until a market is well established before joining it, then attempt to create an Apple-idealized product for that market. Given the size of the AR/VR market today, and its practical applications for a broad groups of users, it’s too early for the company to execute that plan.
No that’s (sadly) not a stereo camera on the iPhone 7
But that doesn’t mean Apple isn’t actively engaged in AR/VR research and development. In fact, over the last few years the company has been bulking up its expertise in the field, and likely now has a sizeable operation dedicated to exploring the potential applications as applied to its product portfolio. Here’s how we know.
Then there was Apple’s hiring of Doug Bowman, Director of the Center for Human-Computer Interaction at Virginia Tech, and an influential figure in the world of virtual reality academia.
Patents
Apple is infamous for its patent strategy and you shouldn’t be surprised to find that they’re well into it in the AR/VR space. In fact, Apple’s patents in this field tell us that the company has been thinking about immersive wearables for a long time; patents filed at least as far back as 2007 show a device that looks much like the VR headsets we see today. Apple has a number of head mounted display patents, including one which shows a headset that would dock with a mobile device, and another for an augmented reality display.
Starting with Apple’s purchase of augmented reality firm Metaio in 2015, the company has made a string of AR and computer-vision related acquisitions; Faceshift (marker-less facial mapping and animation), Emotient (facial expression recognition & analysis), and Flyby Media (large-scale SLAM, indoor navigation, sensor fusion, image recognition, and 3D tracking) have all been snatched up in the last 8 months, further bolstering Apple’s AR/VR talent pool and patent portfolio.
Competition
What do you do when your biggest competitors—who also happen to be among the largest tech companies in the world—are all working on AR/VR tech? Well, you better get there too. If AR and VR catch on, the products with those features are the ones that customers will choose.
Apple has watched over the last few years as the likes of Samsung, Google, Facebook, Microsoft, Amazon, Intel, Qualcomm, and more have made initiatives in the immersive technology space into significant parts of their forward-looking strategy.
It’s early but the evidence is clear, Apple is hard at work developing practical applications for AR/VR tech, and, as long as the market develops into something significant, you can bet they will be making a splash in the space in the near future.
![Apple CEO Tim Cook [Image courtesy Mashable]](https://i0.wp.com/www.roadtovr.com/wp-content/uploads/2016/01/tim-cook.jpg)
