Creating great audience experiences using 360 and VR

There are more and more opportunities for artists to exploit new technologies such as 360 video and virtual reality. As part of its commissioning programme, The Space has worked with a number of organisations to develop such projects.

“Each has been wildly different and we’ve supported projects at different scales,” says Joe Bell, a former Associate Producer for The Space, now a Senior Producer at Aardman. “Yet each has confirmed the same thing: it’s the creative and the art that matters, not the technology.”

“If anything it’s the technology that can create barriers to reaching audiences,” Bell continues, “so we’re really looking to artists’ skills with narrative, choreography and immersion techniques to create something that makes best use of the form.”

Immersing the audience

Virtual Reality and 360 video are different technologies, but Luke Ritchie from the Philharmonia Orchestra, whose work Mahler 3 was commissioned by The Space, says that the distinction doesn’t matter to audiences.

“Generally people tend to distinguish between 360 and VR in that with 360 there are no computer-generated images,” explains Ritchie. “It is basically documenting in live-action film. And also you don’t have any room-based movement. If you recreate the room as a computer game in VR, instead of 360, then I would be able to move around in what we call six degrees of freedom.”

“For me, though, it’s the way that the audience engages with the content that is key. You can argue over whether the Avengers movie is CGI or live-action, but people just think of it as a film.”

Ritchie is Head of Innovation and Partnerships at the Philharmonia, and has been experimenting with audience experiences using immersive technologies for some time. Mahler 3 uses 360 video to allow an audience to watch Esa-Pekka Salonen conducting the Orchestra per-forming Mahler’s Third Symphony as if they were sitting amongst the musicians.

This new way of presenting a concert was undertaken not simply because of the technology, but because it presented an opportunity to create an intimate experience.

It also highlights how producing an experience for 360 or VR requires all the skills an arts organisation has around creating an event. Yet alongside production and artistic requirements, the Philharmonia also had to develop the technical skills to make the most of the technology.

Another Space-commissioned organisation, IOU Theatre, similarly had to capture its Rear View outdoor theatre piece in 360 video without changing or disrupting the existing project. The team wanted to document that work, but also create a new piece that could live on its own.

“We had already been touring our theatre show, Rear View, to different festivals,” says IOU Theatre’s executive director, Joanne Wain, “and the audience were taken around on the bus with a performer on the streets of the cities we went to. Our Artistic Director David Wheeler and our technical manager had already put a 360 GoPro on the bus and we were starting to look at what it might look like in a 360 world.”

“We realised that if we were to make a 360 film version of Rear View, it would help us digitise our work and extend our audiences,” says Wain.

“And if we did a professional job of it, we could then see really how effective 360 video is for our type of work,” says Wheeler. “The nature of our work is very 360, we often set the audience in unusual situations and the whole setting becomes part of the production.”

Audio in a visual medium

“What we found with VR is that your brain believes something is real in large part by its audio measurement,” says Ritchie. “If you go into a cathedral, you hear an echo and your brain goes, right, I’m in a huge space. So it’s not just the visual information, it’s the audio information that’s making your brain believe it’s real.”

In the case of the Mahler concert, the need was to capture the whole orchestra as a high-quality audio recording would. The audience member placed in the middle of the orchestra wants to hear the instruments next to them, but they must also hear the entire concert.

“One of the big frustrations I have with VR is the audio,” says Ritchie. “You tend to watch VR with a headset and headphones and ultimately from an audio perspective, classical music is not great on headphones. Particularly with Mahler 3, it starts very quiet, you can hear a pin drop, and the by the end you’ve got 114 players going hell for leather.”

“But also, from an audience perspective, it’s very isolating,” he continues. “At the moment, VR is a very solitary experience and we wanted it to be more communal. Music should be a communal, shared experience.”

For Mahler 3, The Philharmonia Orchestra worked with an experienced classical music recording firm, Floating Earth. Capturing the audio on more than 30 separate microphones, they then produced a mix that would be played not through a typical VR headset’s headphones, but on a circle of speakers.

So the audience is brought into a space where they are surrounded by around 12 speakers and 2 subwoofers. They wear the headsets to see the orchestra, but they hear the music in higher quality and without cumbersome headphones.

Alongside all of the microphones capturing audio, the visual side of the Mahler concert was cap-tured by a 360 camera.

“That camera is essentially a collection of consumer-grade GoPros,” he says, “16 GoPros in a circle, held together in a basic 3D plastic chassis, and it’s got a little Wi-Fi remote that you use to trigger it. We tested it and tested it and tested it, but when you put 2,500 people in the Festival Hall with mobile phones, it did something to the Wi-Fi remote and it just didn’t work.”

However, the team had planned for even this in advance and arranged what they thought was a last resort. “Our amazing cameraman Peter Collis went on stage to manually start the camera between movements,” recounts Ritchie. “It’s a huge credit to the orchestra and to [conductor] Esa-Pekka, who had to wait and hold their concentration for a minute.”

Unique production issues

In theory, IOU Theatre had more freedom than the Philharmonia because the performance it was capturing was one of a series, they could try again if it failed. However, for every project whether in immersive 360, VR or a regular live event, there is the question of costs. For all practical purposes, then, capturing an event in 360 or VR is almost always a one-off job, a task for a single performance.

For his 360 film documenting Liverpool’s Granby Winter Gardens project, though, producer Rob Vincent had the issue of recording what, through local government and building delays, turned into a multi-year project. The Granby Winter Gardens is an urban regeneration project created by architecture collective Assemble working alongside the local community through Granby Four Streets, the film documents the project, as a pair of abandoned, derelict homes is transformed into a public garden.

At each point when Rob shot 360 video he had to make a decision not just about when to film, but to ensure he was at precisely the right spot every time for continuity. If you are not to be in the shot yourself, you have to set up the camera and walk away from it, trusting and hoping that you’ll capture something interesting.

It’s about what’s the most appropriate way to tell the story,” says Vincent, “but when I first walked into a space like that derelict house, I was literally walking through the front door, standing in the middle of the room and turning around to look at it all. Clearly to me, the way to tell the story was to allow the user to be embodied in that space as I was.”

Heritage Storeys – Metro Boulot Dodo and Culture Coventry

Audience interaction

Currently the single best way to see a 360 or VR immersive work is through wearing a headset, but while they’re no longer as cumbersome as they were, they’re still new to most audiences. They’re also not cheap to do at scale, so it isn’t practical to fill a concert hall and give every attendee a headset.

Instead, what’s proven to be the most effective way to reach the greatest number of people is to schedule 360/VR performances like timed entry into a gallery. At the start of each session, everyone in a group of people is both given a headset and, crucially, shown how to use it – a process known as ‘onboarding’. Then the performance starts and when it’s done, those people are also shown how to remove the headsets.

“The film itself was 12 minutes long,” says IOU Theatre’s Wain, “but the installation experience [with fitting the headsets] was probably about 20-25 minutes.”

This all makes having staff who are familiar with the work and the technology present to help with onboarding essential.  There is also one issue that in theory is a problem, but in practice may help with getting immersive shows to more people. Currently the majority of people will say that they can only enjoy 360 or VR for a quite short amount of time. Perhaps it’s the unfamiliarity and perhaps it’s the disorientation of standing in one room while seeing another, but whatever it is, these projects currently tend to be quite short in duration.

Total control

While they are each very different uses of 360 video, the Philharmonia Orchestra, IOU Theatre and Assemble’s Granby project are all captures of existing work. There is an alternative, though, and that’s virtual reality where you create everything in a computer.

“[It] means basically starting from scratch building your environments,” says Paul Long, director of Metro Boulot Dodo, which created Space commission Heritage Storeys for the Herbert Museum.

“So for us that means if you build a room in VR, you can walk around that room,” he says. Metro Boulot Dodo works in entirely virtual reality which it isn’t constrained by what has or hasn’t been filmed, and it doesn’t have to capture a performance in one go. Heritage Storeys, is a piece for the Herbert Museum based around a virtual lift. The idea is that you press a button and go to the floor you want. Then the doors open and you walk out of the lift into a whole other space.”

MBD created the virtual lift with three different “floors”, each of which had its own VR world. For any one museum or other installation, those floors could readily relate to different collections. But the format of having a lift and different levels is also meant to address one of the problems of VR.

“A lot of projects are built from the ground up every time,” says Long, “whereas what we wanted to do was build an infrastructure, which is the lift. That allows you to travel to different spaces and then you, as creator, just concentrate on the content each time.”

The future

What MBD has done is create a platform that lets them continually build new works. Each of the projects that The Space has supported have been intended to be a complete work themselves yet also provide something for the future.

The Philharmonia’s Luke Ritchie says that as much as their work was about creating this 360 Mahler experience, it was also about learning and experimenting.

“Arts organisations are really doing pioneering work with technology that almost nobody else does [and] it seems to be a very positive place to create new work,” says Ritchie. “But, frankly, organisations need these kinds of funds to create new IP whether that’s the content, software, or the knowledge and expertise we got from this project. One of our key long term goals is to create enough IP that we can commercialise and generate new revenue streams for the orchestra. And we are doing that, it is helping to do that.”

“We’re at that funny intersection point between arts and technology,” he says, “and the UK is well-positioned to capitalise on that. Genuinely, it is wonderful that The Space exists and funds these kinds of projects.”

Each 360 or VR piece of work from any arts organisation pushes the form forward and does so at a time when the technology is forever improving, too.

It’s a medium with exciting, thrilling potential, but immersive 360 and VR will always be a tool for artists – and audiences.

“The advantage of the video is that you often feel as if you are standing next to the performer,” says IOU Theatre’s Wheeler. “Audiences seem to like it when it feels as if the performer is talking directly to them.”

These performance skills, the ability to create work and market it to audiences, are the same ones that are needed for 360 and VR.

“If 360 video or virtual reality is the right fit for the editorial you’re trying to explore,” says Joe Bell, “then it can be an amazingly vibrant and powerful way to immerse and engage your audiences.”

TIPS FOR 360 WORK

  • You have to put the artistic idea first and only then think of the technology: think what the technology can bring to the audience experience that wouldn’t be achieved otherwise
  • Plan how you are going to capture or create audio as much as you do visuals
  • Avoid having only one chance to capture an event
  • Plan ahead for technology to fail and have a back-up plan
  • Think carefully up front about how the audience will experience the work, including the onboarding process

Floating Earth
https://www.floatingearth.com

Granby Winter Gardens at Assemble:
https://assemblestudio.co.uk/projects/granby-winter-gardens

IOU Theatre
https://www.ioutheatre.org
YouTube: https://youtu.be/_q2Dm3urDc4

Metro Boulot Dodo
https://www.mbd.limited

Philharmonia Orchestra
https://www.philharmonia.co.uk

How useful was this resource?