The funders’ perspective: how to evaluate digital work

You’ve received public funding for a digital project, and now you need to report back to the funder about how you spent the money. This might be a familiar scenario, or it might be your first experience of reporting, but we know that evaluating a project can be tricky. Time, money and resources are tight and it can be difficult to decide how to measure your work. You want to give the funders the information they need, but the funders are keen that you should also use the evaluation to enhance your own understanding of what worked well about your project – and perhaps also what didn’t.   

So how do you balance all of this, when planning your digital project evaluation? And what do funders really want to see in that final report? 

We spoke to two funders, Ashley Smith-Hammond (Creative Scotland) and Paul Glinkowski (Arts Council England), to gain some perspective on how best to approach evaluation.  

1. Define what success means to you – it’s not all about ‘big numbers’ 

The success of your project depends on the values you set for it.   

Far from expecting to see ‘big numbers’ in your reporting, the funders would rather see honest reflections on whether your project met its specific goals and, if not, why not? Every project, and every artist and organisation, has a different definition of success, and your evaluation should communicate this.  

Ashley gives an example: “Have a clear idea of who your audience is, and who’s going to benefit from the work. If that audience is a very small group, and especially if it can be challenging to reach that group, then a small but highly targeted audience is still totally valid.

When reporting, it can be tempting to focus on the hard numbers – a video’s total views, for example, or final ticket sales – but Paul encourages you to consider the softer outcomes from your project, and to explore other forms of evidence for “social good” which may be unique to your work.   

You can do this by combining quantitative data with a qualitative approach. Quantitative data is often numerical data but can include other forms of measurable information such as audiences’ postcodes, or answers to ‘yes’ or ‘no questions. Qualitative data is subjective, and could be gathered through observations, focus groups or answers to open-ended questions. 

There are useful, free resources which can help you define your own values, and to find the forms of evidence which will help you measure them. The Centre for Cultural Value has resources for mixed-methods evaluation, including “questions to consider” that are helpful for thinking through what good, or useful, evaluation might look like for you and your project. 

Remember, small numbers can still result in meaningful impact – it all depends on your project’s specific goals.  

2. Know your platforms and understand your data 

Metrics are different for every platform. It is key for you to know, in advance, the kind of data you can expect to gain from your chosen digital platform or tool, and to plan accordingly. Funders expect you to be aware that most digital tools have some form of built-in analytics, and for you to draw on that data in a realistic way. 

For instance, if specific demographic data will be vital to your understanding of your audiences and your evaluation of your project, then be sure that you have a way to gather this data. Some online platforms only provide information about the country each visitor comes from; if this is not enough, you may need to run an additional audience survey to get the detail that you need.  

Also be aware that some platforms only permit you to download data within a limited timeframe. Ashley gives an example: “What if you have a six-month project, but you can only get the data from the last three months? You need to know that up front and download data when you can.” 

The data you provide should tell a story about your work. This may involve insights into streaming and download numbers, but more importantly you should explore any patterns in your data that can help to explain if your project met your goals. It is also important to be aware of what your data can’t tell you. For instance, you may have collected residential postcodes from your audience to measure demographics, but that is far from an exact science as the types of people who live within a postcode area can be very varied.   

Paul adds a note of caution: “Don’t use data if you’re not completely confident of it, or you don’t really understand the meaning of it.” In this case, he says, it is better to “get away from the focus on the quantitative and move towards different sorts of value.”  

3. Don’t be afraid to report challenges or failures  

Learning from failure is just as important as celebrating success. It’s okay if things don’t go to plan, and it’s important to include that in your reporting.  

Paul highlights that confronting any challenges and failures and evaluating them honestly is one of the most important things you can do. He says, “We want to give people a sense of permission to fail – but to fail in an enlightened way, without glossing over the negatives, and to understand how things might be done more effectively next time.” 

4. See evaluation as a rolling process 

Considering project shortcomings or opportunities for improvement is particularly beneficial if you can evaluate your project throughout its duration, rather than waiting until the end. If it’s possible to make changes to your project as you go, then this kind of honest reflection can help you to modify your approach while there’s chance to benefit from those changes.  

Ashley also points out that some types of digital project don’t have a clear ‘ending’, particularly if they will be available all the time. This makes it extra important to have a strategy for reflection throughout your process. Some funding specifically requires mid-project reporting, but even if this isn’t the case, don’t wait until your project is over to run tests, review insights and make changes if needed. 

5. Budget for evaluation from the start  

In a sector where time, money and resources can be scarce, it is important to plan the resources you need for effective evaluation from the start of the project.  

Reporting can be time-consuming and often requires specialist skills. Funders know this. To ensure an effective approach, they recommend that you budget time and money in advance, scale the evaluation to something that is proportional to your overall project budget and plan for external expert help if you need it.  

“What you budget for sends a message about what you value,” says Ashley. “The fact that you’ve taken trouble in your planning to consider your capacity for evaluation shows a quality of thinking which is reassuring.”  

“It can be easy for a small project or organisation to think they have to do all of these things themselves,” says Paul. “Often it makes sense to invest some of the budget to get an expert to do it, or to teach you how best to do it.”  

When resources are limited, be selective about what, and how, you evaluate. For example, if you are evaluating a single project, be specific about what questions you are trying to answer. Or, if you are evaluating a season of work, you could focus on a more detailed evaluation of one or two key works which seem most likely to provide you with useful insights and then take a lighter-touch approach with the others.