When it comes to immersive storytelling, one often thinks of holograms, interacting with posters/QR codes or images and of course virtual reality (VR). Augmented reality (AR) is more aligned to social interactions, training and simulation. Especially with mixed reality (MR) applications for Microsoft’s Hololens. AR is not immediately associated with storytelling, but new concepts are bubbling and evolving in the midst of the new era of AR, VR and MR. After seeing a prototype of an AR film, Mária Rakušanová curator of the VR Awards at Raindance Film Festival was keen to showcase it at the Festival. VRFocus spoke to Duncan Walker, the founder of Trashgames about his short AR film NEST and how he made it.
Duncan Walker studied filmmaking, animation and visual effects, but somehow ended in programming, animation and videogames. Everything has come full circle in the new era of VR and AR. The creative worlds of filmmaking and videogames are moulding together to create new potentials for storytelling. Traditional filmmaking that required VFX and CGI characters required actors in motion capture suits, a green screen and a lot of post-production technology to get a final product, taking months or even years to create the final rendered output of the film. Walker’s filmmaking method turns this traditional process on its head. Now you can film CGI characters in real locations and direct them just as you would direct an actor on a film set.
Walker explains that he 3D scanned professional dancer Aoi Nakamura through photogrammetry and put her face onto one of the cyber women characters he had created. He took these CGI characters and aliens onto the iPhone through ARKit, and then filmed them in real locations in-real time. NEST was created by combining non-specific scenes and specific scenes. With non-specific scenes, Walker directed his CGI characters through a control panel on his phone. This enabled him to direct his CGI characters; he can direct them to follow him, look at him or walk a certain direction. Specific scenes on the other hand were pre-animated, but allowed Walker to film the scene from any angle and film as many times as he wanted.
Walker explains: “Making a film is very much like making a game these days. It’s 3D assets, lots of animation, not much actual acting anymore. With what I’m doing, it’s exactly the same as making a game. Essentially I make a game, make game characters but then turn them into a film.”
Presently, these CGI characters are good at carrying out actions but are not able to emote – so we haven’t quite reached uncanny Valley. However with developments in machine learning and A.I., Walker doesn’t think we’re too far off from CGI characters being able to respond to voice, interact with objects in a scene, detecting objects in the environment and stopping or avoiding them. He believes that a CGI character would be able to interact with a real life actor on set as well. Similar to how autonomous cars are able to scan their environment and decide what to do. Walker explains that the character knows as much as your phone does, which is quite a lot these days. He hopes that software for image analysis on the iPhone would allow for live image processing that would make the end product look a lot better.
At the moment the end product would be exported to 2D traditional platforms such as YouTube, cinema or television. It would still be a 2D film, but Walker also suggests that users would be able to download an app on their phones and watch the film or action take place where they are located. The feedback Walker says differs from who sees it. Filmmakers and CGI savvy individuals will spot things and suggest how they would have done it differently whilst those who know nothing of the technology cannot quite believe NEST is created on a phone. It’s weird, strange and grounded in reality in the streets of London they recognize.
Walker hopes to continue making AR films, and potentially create a toolkit that would allow filmmakers to create their own AR films. His work can be found on his website and his AR films can be seen here. The potential for Walker’s AR filmmaking toolkit, if funded and built and then made available for public would allow the average phone user to create their own films with CGI characters. NEST is one of the first examples of the potential for future storytelling with new technologies and many more be coming.