Immersive technology can be a powerful medium when used correctly to draw attention to particular issues, such as when the United Nations (UN) and Here Be Dragons created Ground Beneath Her. Currently, the United States is going through a housing crisis with eviction cases on the rise. It’s why a new mixed reality (MR) installation These Sleepless Nights was created to highlight what’s actually going on.
Commissioned by The Next Amendment, an initiative whose goal is: “Eventually a national right to shelter could be added to the Constitution as the 28th amendment,” These Sleepless Nights is a dual piece, a documentary which premiered this week at the 76th Venice Film Festival and an MR installation involving a giant cube in San Francisco and Washinton D.C.
“A large part of US homelessness hinges on the facilitation of court-ordered evictions. These Sleepless Nights is a mixed reality documentary that uses cutting edge spatial computing technology to allow visitors to listen, connect and engage in new ways with those on the frontline of America’s eviction crisis,” explains the synopsis. When it comes to numbers the exhibition uses 2016 figures from the Eviction Lab, noting that 2,350,042 eviction cases were filed and 6,349 people were evicted per day.
These Sleepless Nights was inspired by Matthew Desmond’s Pulitzer Prize-winning book EVICTED, with the documentary directed by Gabo Arora (ZIKR: A Sufi Revival) with music by Philip Glass. At the installations, guests will be able to wear Magic Leap headsets to view imagery on the cube as they walk around it. Each side of the cube is a stage in the eviction process, so as visitors slide their hands across the walls characters appear through audio stories and spatialized audio. The locations will also be used to help raise funds towards the right to shelter, accessible via an iPhone app.
Executive produced by Fable’s Edward Saatchi (Wolves in the Walls) and co-produced by Montreal based studio DPT in association with Johns Hopkins University Immersive Storytelling and Emerging Technologies Lab, The Next Amendment aims to raise $1 million USD for local action networks providing Housing First solutions from Oakland to the Bronx.
VRFocus will continue its coverage of the latest MR projects and installations, reporting back with further announcements.
Matthew Niederhauser came from a background in anthropology and journalism. When he started dipping his toe in documentary and installations he got bit by the virtual reality (VR) bug. This led him to doing VR for three years and setting up Sensorium, an experiential studio that most recently world premiered ZIKR: A Sufi Revival at Sundance Film Festival this year.
Niederhauser explains that he was on a trip with Gabo on a United Nations special mission to Libya where they encountered some Sufi performing as part of a ritual in Tunisia. Blown away, they had never seen anything like it and decided John Fitzgerald, himself and Gabo Arora had to go back and make something out of it. Niederhauser says that Tunisia was unique as it was an Islamic nation. ZIKR: A Sufi Revival is aimed specifically at bringing interactivity to the documentary format, combining 360 degree video with volumetric interviews using Depthkit. Working together with interactive development firm Superbright, they helped use Unity to add another level of interactivity to the documentary. Enabling users to have beads that interacted with the user and those in the experience as well.
At the Sundance Film Festival, users were required to enter an empty chamber, take off their jackets and shoes, and put on an HTC Vive headset on. A special Tunisian rug was brought in and four users were put in a diamond like pattern in a ritualistic circle with HTC Vive controllers. Under two lighthouse sensors, four users were able to enjoy the experience together and interact with the content as well as each other. Everybody appears to one another as translucent avatars and the more you move your controllers, the more the environment reacts to you. Niederhauser says that people in the experience at Sundance thoroughly enjoyed it and the dancing, and this has inspired them to think about taking the experience online. “It was an amazing reaction and thinking of ways we can turn it into a home, a lobby… Come in people from around the world and participate together.”
Users were coming out of the experience laughing and smiling. Niederhauser says that it was a beautiful and joyous experience, which he believes was primarily due to people participating in it together. “We uncovered a new leaf in the documentary realm.”
With Dogwoof purchasing ZIKR: A Sufi Revival there are now discussions on improving the next iteration. Potentially through creating an online lobby so people can experience it at home, adding subpac peripherals and using HTC Vive trackers rather than controllers.
To find out more about the Sensorium, and how they used various levels of technology watch the video below.
Last week the popular Sundance Film Festival took place, showcasing some of the best films from around the world. One of those in attendance was ZIKR: A Sufi Revival which VRFocusreported on just as the event took place. Now it has been revealed that documentary producer, sales agent and theatrical distributor, Dogwoof has purchased the immersive documentary.
Directed by Gabo Arora (Ground Beneath Her), produced by Jennifer Tiexiera, and created by Sensorium, Superbright, and Tomorrow Never Knows, ZIKR: A Sufi Revival premiered 19th January in the New Frontier section of the festival. The film takes four participants on an interactive, virtual reality (VR) journey into a world of ecstatic ritual and music in order to explore the nature of faith alongside followers of this mystical Islamic tradition. By opening up an experience to Sufism, dancing and singing alongside members of the Tunisian group Association de la Renaissance du Maalouf et du Chant Soufi de Sidi Bou Saïd, it aims to shed light on the religion, revealing an Islamic practice of inclusion, acceptance, art, joy and understanding.
The deal was brokered on the last day of the Festival by Dogwoof’s Andy Whittaker and Tomorrow Never Knows’ Nathan Brown. The agreement secures funding to support additional development, including an online version of the VR experience to bring multiple players into the experience from around the world.
Dogwoof will also look to global distribution with location based installations at high profile international cultural centers, cinemas and museums. This is the first public project for Tomorrow Never Knows, a newly formed VR/AR/AI startup founded by industry veterans Arora, Brown, Saschka Unseld and Tom Lofthouse.
“There is real demand and a growing market for distinctive storytelling in VR/AR,” said Brown, CEO of Tomorrow Never Knows. “To partner with Andy and Dogwoof, one of the world’s most acclaimed producers and distributors of non-fiction stories, is more than a signal, it’s a lightning bolt for the entire industry.”
“Andy and his team at Dogwoof, have proven themselves over and over with their work in traditional documentary. And, with their first VR acquisition with ZIKR, they will bring their same tenacious spirit to making sure more and more people can experience a story very pertinent to what’s happening in the world now,” said Arora. “I couldn’t dream of a better home for ZIKR or a better team than Dogwoof for global distribution. I am honoured to be joining their award winning catalog.”
For the latest news on VR and 360-degree filmmaking, keep reading VRFocus.
Announced on the 7th October 2017 at the Immersive Media Conference, John Hopkins University has added ‘ Immersive Storytelling & Emerging Technologies’ (ISET) to its Film & Media Masters Arts Program. The course will be starting in January, so individuals looking to further their knowledge in immersive storytelling or potentially find a way to fund their project should be excited. VRFocus spoke to Gabo Arora, a filmmaker most well known for his virtual reality (VR) documentary Clouds over Sidraabout ISET, a course he designed and is leading.
ISET is a two year Masters program consisting of 10 courses that will focus on cutting-edge technology focusing on VR, augmented reality (AR) and artificial intelligence (A.I.). John Hopkin’s Film & Media of Arts program is ranked among the most diverse in America’s graduate programs across race, gender and ethnicity with nearly 70% women and 80% people of colour. Tuition is a third of what similar programs cost at institutions like New York University and University of Southern California, making it more accessible to students who might not otherwise have access to the expensive tolls necessary to explore uncharted technological territories. One of the aims of the new offering is to use the power of immersive technology for urban renewal in Baltimore as well as students working together with the John Hopkins School of Medicine and Lieber Institute for Brain development to utilize VR as a tool to treat and diagnose schizophrenia and autism. Gabo explains that the Film & Media Masters Arts Program is only two years old and unlike most traditional film courses isn’t stuck in its ways, but is open to experimentation and new opportunities.
“I began my journey in virtual reality filmmaking at the United Nations, with the goal of using this new and rich storytelling medium to create empathy for some of the world’s most pressing challenges. VR and AR can tell and teach with a depth that is unattainable in most other creative media,” said Arora. “The concentration, when combined with Johns Hopkins’ strong academic programs, will drive social impact across disciplines ranging from medicine to law and international relations.”
The Immersive Media Conference showcased a demo of Freedom Fighter, a social justice experience that uses AR to connect users with the past of Baltimore and Baltimore civil rights leader Lillie May Carrroll Jackson. This project is a perfect example of what ISET hopes students will create when on the course. The course therefore is very practical, and depending on the number of people who sign up there should be around 15-20 students per course. Gabo hopes that by the time the students finish ISET, more jobs in the fields of VR, AR and A.I. will be available for them to immediately take.
Applications are open till the end of the year and can be found here. To find out more about what Gabo hopes to do in the course, and how he intends to teach watch the video below.
Filmmaker Taura Musgrove saw The Last Goodbye , a virtual reality (VR) documentary led by Gabo Arora by Lightshed. The documentary followed Pinchas Gutter, a Holocaust survivor and takes the user to the Majdanek concentration camp in Poland. With the help of room-scale VR, the experience takes the viewer on a tour of the camp and Pincha’s experience as a child there. The compassion and empathy felt by viewers is what Musgrove wanted to tap into when she created her augmented reality (AR) experience. Gabo Arora tells VRFocusabout how with his guidance Musgrove and John Hopkins University were able to create an app that could bring the new youths in touch with their historical past in Baltimore.
Freedom Fighter is an AR app built with Apple’s ARkit that uses geofencing to bring users face-to-face with a volumetric AR model of American Civil Rights leader Dr. Lillie May Carroll Jackson. The app works only with geofencing, meaning that users can only access the content in the exact physical space where the content is made to be displayed. A little like Pokemon Go, you have to physically walk to a location to get access to certain items. Musgrove hopes that Freedom Fighter will not only connect Baltimorians with the history of their city, street corners and significant historical figures but also also show them where certain events took place with geofencing.
So if you were to download the app on an iPhone, you would physically have to walk to certain street corners in Baltimore and take out your iphone. All these locations you have to walk to, are of historical significance in the history of Baltimore and have been affected by the riots in 2015 or are in urban decay. You would hold up your phone for example and see the old National Association for the Advancement of Colored People (NAACP) appear as well as an AR model of Dr. Jackson. She would then proceed to tell you about herself, the history of this corner and its significance.
Dr. Lillie May Carroll Jackson is a herald of American Civil Rights. As head of the Baltimore chapter of the National Association for the Advancement of Colored People (NAACP) for thirty-five years, she pioneered the organization of ordinary citizens, black and white, to protest lynching, educational segregation, and police brutality. Arora, the executive producer of Freedom Fighter explains that Baltimore suffered riots in 2015, the origin of the Black Lives Matter movement and a hotbed of activism and solutions for racial and social justice. The younger generation however seem to have forgotten Baltimore’s civil rights activists and the importance of certain locations in the history of America.
The app has not been released yet, but when it is, it will be completely free to download. making it accessible to let viewers experience firsthand Dr. Jackson’s leadership, vision and strategy for activism. Arora talks about potentially bringing this into history classes in Baltimore as well as helping sustain black businesses in the community by collecting points through the app. To find out more watch the video below.
One of the most emotionally-moving VR experiences that I’ve had in VR was bearing witness to Holocaust survivor Pinchas Gutter share his experiences at the Majdanek Concentration Camp in The Last Goodbye. Gutter not only provides a guided tour, but he is able to achieve a level emotional catharsis through the process of sharing his story that his virtual presence within the experience, amplified my own sense of emotional and social presence, which is what helped to make it such a profoundly moving VR experience. The Last Goodbye uses a unique blend of photogrammetry and billboarded stereo video that helped to transport me into a room-scale experience of multiple key locations at the Majdanek Camp as Gutter shared his memories of being there as an 11-year old child.
LISTEN TO THE VOICES OF VR PODCAST
The Last Goodbye premiered at the Tribeca Film Festival, and it was an epic collaboration catalyzed by co-creators Gabo Arora and Ari Palitz and included HERE BE DRAGONS, MPC VR, and OTOY on the technical production side as well as the USC Shoah Foundation to oversee the process of capturing testimony about the Holocaust.
Arora is the founder and creative director of LightShed, and is now an advisor and former founder of the UN’s VR initiatives where was able to gain access to the Syrian refugees featured in his famous empathy piece Clouds Over Sidra. In this interview, Arora shares his collaborative process, pushing the boundaries of volumetric storytelling by blending photogrammetry-based, room-scale VR with live-action, empathy-based storytelling, as well as how he had to guide Gutter to achieve the depth of emotional presence that makes the piece so powerful.
Gabo Arora founded the United Nations VR, and has directed some of the more well-known VR empathy experiences starting with Clouds Over Sidra in December 2014 in collaboration with Chris Milk’s VR production house Within. Milk first showed Clouds Over Sidra during Sundance 2015, and featured it prominently in his VR as the Ultimate Empathy Machine TED talk in March 2015, which popularized VR’s unique abilities for cultivating empathy.
I had a chance to catch up with Arora at Oculus’ VR for Good premiere party at Sundance where we talked about producing Clouds Over Sidra, his new Lightshed production company, and the importance of storytelling in creating VR empathy experiences.
LISTEN TO THE VOICES OF VR PODCAST
Arora’s work has been at cross section of storytelling and technology, and diplomacy and humanitarian efforts. He studied film in college, but was unable to launch a successful film career in Hollywood, and instead turned towards humanitarian work with NGOs after 9/11 and eventually with the United Nations in 2009. He used his creative sensibilities to move beyond written text reports, and look to the power of new media to tell humanitarian stories. He had some success with collaborating with social media sensation Humans of New York photographer Brandon Stanton by coordinating a 50-day global trip with in 2014 in order to raise awareness of millennium development goals. He proved the power of using emerging technology to promote humanitarian goals.
After he was introduced to Within’s Chris Milk in 2014, he gathered enough support to create a virtual reality lab at UN staring with creating an experience about the Syrian refugee crisis. Clouds Over Sidra was shot in two days in December 2014 at the Za’atari Refugee Camp, which had over 80,000 Syrian refugees. Arora wanted to focus on a day in the life of a 12-year old refugee, and collaborated with his UN contacts to find the young female protagonist named Sidra. Arora said that a big key to cultivating empathy in virtual reality is to focus on the common ordinary aspects of day-to-day living whether that’s eating a meal or preparing for school. While some of these scenes would seem like non-sequiturs in a 2D film, the sense of presence that’s cultivated in VR gives the feeling of being transported into their world and a feeling of being more connected to the place and story.
Arora acknowledges that merely showing suffering of others can have the opposite effect of cultivating empathy. He cites Susan Sontag’s Regarding the Pain of Others as a book that helped provide some guidelines for how to represent the pain of others. He’s aware that we can have a lustful relationship towards violence, and that there are risks of normalizing suffering can create an overwhelming sensory overload. He’s addresses some of Paul Bloom’s arguments in Against Empathy in that there’s a bias towards empathizing with people who look or act like you. If there’s too much of a difference, then it can be difficult to connect through on any common ground. This is a big reason why Arora has typically focused on finding ways of representing the moments of common humanity within the larger context of fleeing from war or coping with a spreading disease like Ebola.
Arora was able to show that Clouds Over Sidra was able to help the United Nations beat their projected fundraising goal of $2.3 billion dollars by raising over $3.8 billion, but he’s much more confident in showing the UNICEF’s numbers of being able to double face-to-face donations from 1 in 12 without VR to 1 in 6 with VR with an increase of 10% per donation. With these types of numbers, there’s been a bit of a gold rush for NGOs to start making VR experiences for a wide range of causes, but Arora cautions that not all have been successful because not all of them have had an emphasis on good storytelling or the technical expertise that he’s enjoyed with his collaborations with Within.
Hamlet on the Holodeck author Janet H. Murray recently echoed the importance of good storytelling in VR experiences by saying that “empathy in great literature or journalism comes from well-chosen and highly specific stories, insightful interpretation, and strong compositional skills within a mature medium of communication. A VR headset is not a mature medium — it is only a platform, and an unstable and uncomfortable one at that.” The storytelling conventions of VR are still emerging, and the early VR empathy pieces have been largely relying upon conventions of traditional filmmaking.
Arora admits that there’s a certain formulaic structure that most of these early VR empathy pieces have taken that rely upon voice over narration, but he says that he started to dial back the voice overs in his most recent piece The Ground Beneath Her. He says that his recent collaboration with Milk on the U2 Song for Someone music video showed him that there’s a lot that can be communicated without resorting to voice overs.
Murray argues that “VR is not a film to be watched but a virtual space to be visited and navigated through,” and she actually recommends “no voice-overs, no text overlays, no background music.” I’ve independently come to the same conclusion, and generally agree with this sentiment because most voice over narrations or translations feel scripted and stilted. They are also often recorded within a studio that doesn’t match the direct and reflected sounds of the physical locations that are shown, which creates a fidelity mismatch that can break presence and prevent me from feeling completely immersed within the soundscapes of another place.
I’ve found that the cinéma vérité approach of having authentic dialog spoken directly within a scene works really well, or that it works best if the audio is directing me to pay attention to specific aspects to the physical locations that are being shown. After watching all ten of the Oculus for Good pieces at Sundance, one of the most common things that I saw is not having the physical location match whatever is being talked about. Sometimes they’re interesting locations to look at, but it ends up putting the majority of storytelling responsibility within the audio. If the audio were to be taken away, then the visual storytelling isn’t strong enough to stand on it’s own.
6×9’s Francesca Panetta used audio tour guides as an inspiration for how to use audio in order to cultivate a deeper sense of presence within the physical location being shown. One live-action VR piece that does this really well was a cinéma vérité piece by Condition One called Fierce Compassion, which features an animal rights activist speaking on camera taking you on a guided tour through an open rescue as it’s happening. The live delivery of narration feels much more dynamic when it’s spoken within the moment, and feels much more satisfying than a scripted narration that’s written and recorded after the fact.
A challenging limitation to many NGO empathy pieces is that they often feature non-English speakers who need to be translated later by a translator who doesn’t always match the emotional authenticity and dynamic speaking style of the original speaker. Emotional authenticity and capturing a live performance are some key elements of what I’ve found makes a live-action VR experience so captivating, but it’s been rare to find that in VR productions so far. There are often big constraints of limited time and budgets, which means that most of them end up featuring voice over narratives after the fact since this is the easiest way of telling a more sophisticated story. This formula has proven to be successful for Arora’s empathy pieces so far, but it still feels like a hybrid between traditional filmmaking techniques and what virtual reality experiences will eventually move towards, which I think Murray quite presciently lays out in her piece about emerging immersive storyforms.
Arora’s work with the UN in collaboration with Within has inspired everyone from the New York Times VR to Oculus’s VR for Good program and HTC’s VR for Impact. It also inspired Chris Milk’s TED talk about VR as the “ultimate empathy machine”, which is a meme that has been cited on the Voices of VR podcast dozens of times.
But the film medium is also a powerful empathy machine as Arora cites Moonlight as a particularly powerful empathy piece that was released in 2016. Roger Ebert actually cited movies as the “most powerful empathy machine” during his Walk of Fame speech in 2005. He said:
We are born into a box of space and time. We are who and when and what we are and we’re going to be that person until we die. But if we remain only that person, we will never grow and we will never change and things will never get better.
Movies are the most powerful empathy machine in all the arts. When I go to a great movie I can live somebody else’s life for a while. I can walk in somebody else’s shoes. I can see what it feels like to be a member of a different gender, a different race, a different economic class, to live in a different time, to have a different belief.
This is a liberalizing influence on me. It gives me a broader mind. It helps me to join my family of men and women on this planet. It helps me to identify with them, so I’m not just stuck being myself, day after day.
The great movies enlarge us, they civilize us, they make us more decent people.
Ebert’s words about film as a powerful empathy machine as just as true today as when he said it in 2005. I do believe that virtual reality has the power to create an even deeper sense of embodied presence that can trigger mirror neurons, and may eventually prove to become the “ultimate empathy machine.” VR may also eventually allow us to virtually walk in someone else’s shoes to the point where our brains may not be able to tell the difference between what’s reality and what’s a simulation. But as Murray warns, “empathy is not something that automatically happens when a user puts on a headset.” It’s something that is accomplished through evolving narrative techniques to take full advantage of the unique affordances of VR, and at the end of the day will come down to good storytelling just like any other medium.
One of the most emotionally-moving VR experiences that I’ve had in VR was bearing witness to Holocaust survivor Pinchas Gutter share his experiences at the Majdanek Concentration Camp in 