More than just measuring views
When you’re evaluating digital work, it can be tempting to grab a few readily available audience statistics – views, shares, likes – to buoy up your team, funders or other stakeholders, and then quickly get back to creating the content you are passionate about.
Of course there’s nothing wrong with headline audience statistics. But if that’s all you focus on, your evaluation is likely to miss out on a wealth of insights: the chance to paint a rich picture of the impact your work is having not just on your audiences, but on your team and perhaps even on the wider world. You might also miss opportunities to strengthen your approach to this kind of work in future.
Barriers to good evaluation
In 2023, The Space commissioned the University of Exeter to conduct research into current digital evaluation practices in the cultural sector. This showed a lack of resources and budgets for effective evaluation, limited models for good practice and a lack of shared terminology, among other challenges. We therefore wanted to help reduce some of these barriers and to provide a practical framework for evaluating ‘digital cultural experiences’ (a term we’ve defined below).
Of course, there’s already a range of tools and guidance available to support the evaluation of cultural sector work and we’re keen to build on that rather than duplicating efforts. So, we’re working on this project collaboratively with the Centre for Cultural Value and the team behind the Impact and Insight Toolkit, alongside cultural sector funders and experts from various cultural organisations that regularly produce digital projects.
A practical and accessible approach
We want to offer a structured approach to digital evaluation that is easy to use and tailored to cultural sector needs. We’ve tried to keep our advice simple but also felt it was important to address the enormous range of value that digital projects can create, beyond just audience reach. We hope to provide something of a compass to help you to navigate and prioritise these areas, taking the step-by-step decisions you need to form an effective digital evaluation plan which reflects the nature of your work and your objectives.
Our approach is designed to be accessible if you are just beginning your digital journey, while also containing useful advice if you have more experience. We’ve tried to keep it jargon-free so, if you spot any terms you find confusing, please let us know.
What is distinctive about digital evaluation?
For background advice on cultural evaluation, we recommend reading the resources published by the Centre for Cultural Value, in particular their Wheel of Change Framework. Our approach takes these general principles and applies them to the particular challenges and opportunities of evaluating digital experiences. In doing this, there are two main factors which we think make digital evaluation distinctive.
Firstly, digital technology provides particular types of user experience. For example, with an interactive work such as a game, the user often gets to decide the route they take, meaning different people may experience markedly different things. Similarly, with a virtual reality project, understanding how your audiences deal with the physical hardware and also how they experience the immersive nature of the content (was it fun, nauseating or scary?) may both be aspects you want your evaluation to address.
Secondly, digital projects often allow you to gather exceptionally rich and often real-time audience data for relatively little cost. You can see exactly how many people interacted with a piece of content and what they thought much faster and more comprehensively than you would get feedback from an audience sitting in a theatre or walking round a museum. This richness of data and speed of measurement can be both a blessing and a curse. After all, this data is only ‘free’ if you set aside the cost of your time to make sense of it. What should you focus on and what should you ignore? Our advice tries to help you to think through those questions.
Defining ‘digital cultural experiences’
Let’s take a moment to define the ‘digital cultural experiences’ that we are considering here. By ‘experiences’ we are referring to digital media that people can engage with either online, via a broadcast platform, in a venue or in a live event setting. The format of this digital content can be varied e.g., linear video or audio, 360 video, binaural sound, virtual reality, augmented reality or interactive experiences such as games and websites. And digital content does not exist in isolation: experiences might be hybrid, with audiences or visitors encountering a mix of digital and non-digital elements.
We also want to interpret ‘cultural’ quite broadly to include performing arts, visual arts, literature, museum collections, heritage assets and other cultural content. Within this broad spectrum, it is important that the experience is aiming to provide cultural value itself (e.g. through entertainment or education) rather than simply promoting a separate cultural experience in the way a piece of social media marketing might. Of course, distinctions between cultural content and related marketing content can be blurred and the same evaluation approaches can be effective for both. But here our focus is on digital experiences that are intended to have their own cultural value.
The framework: three groups to consider
Now let’s turn to our proposed framework. Our suggestion is to consider the evaluation of your digital experience in terms of three groups:
- Audiences: those who have (or are intended to have) the digital experience, whether online audiences or visitors to a venue or event
- Creators: those who are involved in the creation of the digital experience (there is often an overlap between audiences and creators with participatory work)
- Wider world: anyone else who might be impacted by your project, without either directly experiencing it or participating in its creation
These three groups form a comprehensive ‘universe’. Our intention is to build out this framework to cover the evaluation of outcomes and impacts for all three. However, we’ve started by focusing on audiences, because this is the area where evaluating digital work is most distinctive, for the reasons outlined above.
Three resources for evaluation
Our article on evaluating audiences for digital experiences looks at how to consider who you are reaching, how they are interacting with your work, and what their subjective experience is.
We’ve also written about embedding evaluation into interactive projects, showcasing some interesting ways you might understand how audiences engage with an interactive experience, without breaking its flow for them.
And in case you think funders are only interested in evaluation reports that show big audience numbers, you might benefit from reading our article The funders’ perspective: how to evaluate digital work. This offers interesting insights from digital specialists at Arts Council England and Creative Scotland, covering audience evaluation as well as some broader themes.
About creators and the wider world
In the coming months, we’ll be expanding the framework to cover evaluation in relation to creators and the wider world. However, at this stage it’s worth noting a few points about these groups.
Firstly, you can consider “creators” quite broadly, both in terms of individual creative practitioners (artists, performers, designers, directors etc.) and the producers and project managers who support them at all stages of a project, from planning and production to the testing and publishing of the content. You might also consider creators at an organisational level, whether a single company or a network of partner companies collaborating on a project.
Benefits for creators of digital work might therefore include the creative and cultural impacts they want their work to have, increased skills, confidence and improved processes from producing the work; financial benefits from any primary or secondary income generated and the longer-term benefits of any new partnerships formed by the project.
Secondly, particularly with digital projects, there can be an overlap between creators and audiences. A digital project might encourage audiences to take part in the creative process through user generated content or other forms of participation. It is important to consider the overlapping impacts on this group of co-creators.
Finally, wider world impacts can be very wide ranging and therefore difficult to evaluate, particularly when working within the tight budgets and short timeframes of many publicly funded cultural projects. However, this wider sphere should not be forgotten. It might include the environmental impacts and benefits of digital work or ways in which digital production can act as a catalyst for new partnerships and innovation within a local economy.
Let us know your thoughts
The resources we’re sharing here are a starting point. Our plan is to work collaboratively to build on these, as our understanding and experience of evaluation develops.
With that in mind, we welcome your thoughts and feedback on what you’ve found useful about these resources, elements we might improve and what else we could do to make this framework as helpful as possible. Please contact us with your suggestions.
Illustrations by Jazz Rumsey
How useful was this resource?
Join our Mailing List:
Follow us on social media:
