Apple’s Tim Cook is betting high on Augmented Reality for future product developments.
Augmented Reality is now being used in games such as the Pokemon Go. This technology allows the gamer to immerse in the real world with virtual objects. Because of the increasing demand of AR, the CEO of the tech giant has recently started a historical collaboration with AR developers.
Initial Plans
Reports have surfaced that AR is coming to Apple smartphones sooner than expected, but the CEO did not comment on this speculation. A report by Bloomberg Technology says that Apple is already working on digital spectacles that has the ability to connect with the iPhone with movies, maps, and other beam content and then to the user.
What Cook did confirm was that he already employed several AR specialists and software and hardware experts to plan this business milestone.
Why AR and not VR?
Although AR is already making waves in the gaming industry, it is still not as popular as VR. VR tends to fully immerse the gamer into a virtual environment; whereas AR somewhat mixes the real world with virtual objects.The best example is Pokemon Go wherein the gamer seeks for VR objects, being the Pokemons in the real world.
When asked why he prefers AR to VR, Cook explains that AR is more preferred because it is less intrusive compared to VR. He is also confident that AR will yield more sales based on its current demands.
The AR Market Predictions
Global Market Insights predict that by 2024, the AR global market will expand to up to 80% amounting to $165 in sales. The move to adopt this innovation by Apple is a business initiative to stay in the industry and maintain its place in the market. All because of a presumption that AR will replace smartphones in the near future.
In preparation to the realization of the gigantic plans of Cook, he hired Dolby’s tech and hardware specialist, Mike Rockwell. He currently leads Apple’s AR team. To spearhead the team, he started investing AR glasses from Meta, a small company of AR wearables.
Being one of the biggest names in tech, Apple is perhaps the chosen one in AR mobile applications considering the huge amount of expenses these AR resources and manpower will require. Cook is excited to experience AR everyday “like eating three meals a day,” the tech genius adds.
Industry professionals and tech enthusiasts always keep their ears to the pavement for any potential ripples from one of the big companies and even the smallest drop from Apple can send waves across multiple markets and publications. Many wondered if Apple would officially step into the VR or AR markets with something exhibiting their usual simple yet luxurious design and, as Bloomberg reports, it looks like the Apple CEO Tim Cook believes the future of the company will be augmented.
Cook has previously theorized that AR could have the same massive impact that smartphones themselves have had and has said positive things about the technology on multiple occasions in the past. Everything is rumor until Apple makes an official statement on the topic, but Bloomberg’s anonymous sources say that Cook believes the company could dominate the next generation of gadgets and keep people engaged with their brand all on the strength of augmented reality.
Gene Munster, a founding partner at Loup Ventures, gave a statement to the publication and he believes the actions to be more reactive than proactive. “It’s something they need to do to continue to grow,” he says, “and defend against the shift in how people use hardware.”
For this to come together, sources say that Apple has been putting together a team with former Dolby executive Mike Rockwell at the helm. This lines up with what the company did when they started working on the Apple watch and one of the former members of that team, Fletcher Rothkopf, is a part of the AR crew. Other members include the former lead engineer for Amazon Lumberyard Cody White, former Oculus researcher Yury Petrov, and design/tech leader Avi Bar-Zeev who has worked on the Hololens.
To bolster these claims, Apple has been making AR related acquisitions as far back as 2015. First, they snatched up AR software developer Metaio and that company’s CEO is now on Apple’s strategic deals team. Then, in January of last year, Apple purchased Flyby Media. Their business is founded in the creation of AR-related camera software.
If all of this turns out to be true and this supposed new AR team finds a way to create an affordable and sleek augmented headset, we could be on the verge of something game changing.
Apple CEO Tim Cook has once again affirmed his belief in the significance of augmented reality technology. Interviewed during a recent trip to Europe, he spoke broadly about how AR could benefit everyone’s lives in the future, highlighting the way it allows users to remain ‘present’ in the real world. Underscored by years of relevant hiring and R&D, Cook’s words appear to further set the stage for the company’s rumored foray into the AR space.
Despite many of the company’s biggest rivals and allies showing active development in the AR/VR space, Apple has yet to make a firm commitment toward or away from these immersive technologies, but there’s no doubt that the company has major research and development going on behind the scenes. The Cupertino HQ has been acquiring talent in this field for several years, filing patents at least as far back as 2007 for HMDs, and new patents for AR. Having hired top VR/AR researcher Doug Bowman a year ago, many suspect it’s only a matter of time before Apple shows their AR/VR hand.
Apple CEO Tim Cook | Photo courtesy Apple
In an interview by The Independent during a recent European tour, CEO Tim Cook reaffirmed his enthusiasm for AR, positioning it as an antithesis of VR in the way that it allows the user to remain ‘present in the world’, rather than closing the world out.
“I’m excited about augmented reality because unlike virtual reality which closes the world out, AR allows individuals to be present in the world but hopefully allows an improvement on what’s happening presently,” Cook said. “Most people don’t want to lock themselves out from the world for a long period of time and today you can’t do that because you get sick from it. With AR you can, not be engrossed in something, but have it be a part of your world, of your conversation. That has resonance.”
While we don’t agree with Cook’s assessment of AR and VR being necessarily opposite technologies (or that VR makes people sick in competently designed experiences), we can understand his view that AR’s focus on what’s already around us means the tech has the potential to integrate well with user’s daily lives (which is surely the realm where Apple likes its products to play).
Cook went on to describe AR’s potential to be as significant as the smartphone, a technology that could improve everyone’s lives.
“I regard [AR] as a big idea like the smartphone. The smartphone is for everyone, we don’t have to think the iPhone is about a certain demographic, or country or vertical market: it’s for everyone. I think AR is that big, it’s huge,” Cook told The Independent. “I get excited because of the things that could be done that could improve a lot of lives. And be entertaining. I view AR like I view the silicon here in my iPhone, it’s not a product per se, it’s a core technology. But there are things to discover before that technology is good enough for the mainstream. I do think there can be a lot of things that really help people out in daily life, real-life things, that’s why I get so excited about it.”
This isn’t the first time that Apple has appeared to show more interest in augmented reality than virtual reality, although given the challenges still facing AR, it could be argued that it would be easier for Apple to launch a VR-ready product first, perhaps a feature we’ll see promoted with the next generation of iPhone, which is expected to use an OLED panel.
Apple CEO Tim Cook’s interest in augmented reality is well documented, and his confidence in it seems to have only grown over the past few months.
On a recent trip to the UK, Cook was quizzed about his thoughts on the technology by The Independent. He again noted that he was more excited about AR than he was VR because the latter “closes the world out” while the former “allows individuals to be present in the world but hopefully allows an improvement on what’s happening presently.”
He explained that AR content can be “a part of your world” rather than closing you off from it. “That has resonance,” he said.
How much resonance? Cook thinks AR could appeal to as many people as Apple’s iPhone does. “The smartphone is for everyone,” he explained, “we don’t have to think the iPhone is about a certain demographic, or country or vertical market: it’s for everyone. I think AR is that big, it’s huge.”
He said that AR was a “core technology” and not its own product, but there was still use for it before the tech is “good enough” for the mainstream. “I do think there can be a lot of things that really help people out in daily life, real-life things, that’s why I get so excited about it,” he said.
We obviously don’t need to tell you just how successful the iPhone has been, and Cook’s vision for the tech is likely looking years ahead. We use our smartphones for a huge number of aspects of our daily lives now, but AR could bring the information and entertainment stored on our phones out of the screen and into the real world.
Comments like these only serve to bolster the speculation that Apple is working on its own AR device, and could skip VR headsets entirely. The real question right now is what form that device takes; could expanded AR capabilities be integrated into the next iPhone? Or does the tech giant have plans for a HoloLens or Magic Leap-like competitor?
Steve Jobs once said to a gathering of senior advisers that when it came to the first iPhone, “we’re going to patent it all,” (New York Times reports). While Apple has certainly changed in the years since Job’s passing, the company has undoubtedly continued to aggressively pursue patents, sometimes racking up hundreds of filings in a single month with inventions spanning everything from touch-sensitive smartwatch bands to 3D environmental mapping. This week however, the company was granted two patents that establish basic hardware and software solutions not only pointing towards a prospective Apple AR device, but marking out some fundamental territory in the process.
Both patents, spotted byApple Insider, were first filed by Metaio GmbH in 2014, a German AR firm later acquired by Apple just a year later.
Patent number 9,560,273, entitled Wearable information system having at least one camera, details a wearable device with a “display attached to his head in front of his eyes” that can use either an onboard SoC or in a wirelessly connected smartphone. The patent goes on to detail the invention’s reactive power mode, letting it switch between a battery-saving low-power mode and an active high-power mode that automatically reacts when a real-world object is matched with a reference object.
Apple AR patent detailing methods for computing 3D coordinates, image courtesy USPTO
In the figure below, the patent shows off a Google Glass-style configuration that is “especially well suited to be used with head mounted displays and a camera pointed at the space in front of the user.”
Patent 9,558,581, entitled Method for representing virtual information in a real environment, delves into the software side of AR, and is centered around virtual info overlays and proper occlusion perception.
The setup proposed in the patent paves way for a device that creates a 3D geometry model of the real environment and shows virtual information in both an occlusion mode and a non-occlusion mode, meaning objects are correctly presented to the user even if they’re hiding behind real-world obstacles. And to that, patent ‘581’ focuses on different ways of making virtual information distinct from the real world so users can quickly differentiate the real environment from the superimposed images. The patent proposes a few ways of doing this, including showing real-world objects as semi-transparent when they’re occluded by virtual info (great for driving) and using dashed lines to indicate a separation from the real and virtual.
Apple’s method of creating “proper occlusion perception,” image courtesy USPTO
While this all seems fairly basic, these sorts of patents help lay an important defensive framework that can help against future disputes from other manufacturers working in the sector; eg. Apple v. Samsung, Apple v. Microsoft, Apple v. Nokia, the list goes on. And if the recent ZeniMax v. Facebook lawsuit has taught us anything, you don’t ever want to be the one writing that half-billion dollar check after you already shelled out $2 billion for the intellectual property in the first place.
So while not giant revelations on their own, this, alongside Apple’s ever-growing library of AR patents, certainly lends credence to the idea that Apple is investing serious time and money in the field, and it doesn’t look to be ‘just in case’.
Apple’s AR Investments
Just one year after the launch of the iPhone, Apple started patenting head mounted displays with features not unlike the VR headsets of today. Of course, at the time it wasn’t feasible to produce such a unit, but again, patents aren’t really for that purpose anyway.
The company has since patented various AR/VR headset designs, like it’s Gear VR-esque phone-based headset, or the most recent patent in summer 2016 outlining a high FOV AR display.
Apple’s recent acquisitions have also pointed towards a heightened interest in AR. Besides acquiring Metaio, Apple has also taken on marker-less facial mapping and animation firm Faceshift, facial expression recognition & analysis company Emotient, and Flyby Media, a large-scale SLAM, indoor navigation, sensor fusion, image recognition, and 3D tracking company.
Another nail in the coffin: around 2014, Apple started hiring specifically for AR/VR software experts at their Cupertino HQ, first for an ‘App Engineer’ who was tasked to “create high performance apps that integrate with Virtual Reality systems for prototyping and user testing,” and then a month later four additional hires with immersive tech talent, including a ‘VR/AR Programmer’.
As we know it, Apple is a perennial ‘johnny-come-lately’ to established tech, preferring to see markets mature before they enter in with a big bet like the iPhone or Apple Watch, and the company seems famously stand-offish when it comes to VR. According to a report by Vanity Fair, Apple CEO Tim Cook says that virtual reality isn’t actually the way to go in his opinion. “Virtual reality sort of encloses and immerses the person into an experience that can be really cool but probably has a lower commercial interest over time,” said Cook.
So if VR isn’t in Apple’s commercial interest currently, the evidence thus far suggests that the company is heading eye balls-first into augmented reality. And we can’t wait to see what the world renowned electronics giant has in store.
