Data surgeries in the east: mystery shopper and qualitative workshops

View all project themes
View all Arts Nation case studies

• Partners: Audiences UK, Audiences London, Audiences South, &Co The Cultural Marketing House, Cultivate
• Themes: Improving market focus, Building capacity to engage
• Segments: Dinner and a show, Family and community focused

An event to launch Arts Nation identified four support needs amongst regularly funded organisations in the East of England. These were:

• understanding more about using customer data
• how to enrich this through the use of publically available data
• the experience organisations offer to people from the target segments
• how these audiences view the organisation's programmes and communications.

A programme with three strands was proposed:

Data surgeries

A researcher worked with organisations to help them understand the potential uses of external and internal data. Surgeries were tailored to each individual organisation's current knowledge and use of data. Where an organisation could provide customer postcode data for the last three years, they were provided with a Mosaic profile and maps of their data.

The surgeries took half a day each for preparation, and a day for the workshop and post-workshop advice. The findings from all individual workshops were analysed to identify any common issues.

Mystery shopper

Under expert guidance, a project manager recruited a group of 'shoppers' from the region whose profiles reflected those of the two key target segments. Recruitment was intended to be achieved through the use of PR and social media but it was found necessary to use commercial recruitment firms.

The Project Manager developed a welcome pack and delivered a training programme for the Mystery Shoppers. An on-line resource was created to capture and analyse responses. 'Shoppers' were not paid and in order to get the full customer experience, had to purchase their own tickets, but were able to claim these back along with other expense. By June 2011, 10 organisations had received a total of 68 visits from mystery shoppers.

The scoring of the shoppers was designed in a way that will not only allow individual organisations to measure their progress over time but will also allow benchmarking within and between regions.

Qualitative workshops

Members of the public with demographics that reflect the Dinner and a show and Family and community focused segments were recruited to take part in workshops and were paid to attend (£100, including expenses). The workshops provided an opportunity for organisations to learn more about what the two priority segments thought of the organisation's programming and communication.

Achievements

Of 32 organisations targeted 25 took part in at least one strand of the programme.

19 organisations received data surgeries, 10 participated in the mystery shopper programme and 10 in the qualitative workshops.

The training, process and scoring framework developed for use by Mystery Shoppers may have application more widely and could provide a national standard.

The data surgeries were valued by those organisations who received them and this approach may suggest a model for future delivery.

Key learning

Projects involving understanding data are a hard sell and considerable effort was required to communicate the benefits of this project to the organisations.

Data surgeries

Of 19 organisations that participated, 10 responded to a follow-up survey:

• approximately half of participants felt that the that the surgeries would make a fundamental difference to the value they place on using data in making decisions
• researchers delivering the surgeries noted a broad range of skills, experience and confidence in the use of data but very little use of external data. They felt that participants tended to overrate their own skill level
• in general, data was rarely used for strategic planning. Researchers delivering the surgeries suggest that embedding this will require greater appreciation of its role and value by senior management
• the surgeries were felt to be a practical and cost effective introduction to the effective use of data and the model may be replicable though it will be important to maintain quality
• this project highlighted that spatial geography is more important in the East than is sometimes recognised by audience development agencies
• a significant majority of respondents agreed that 'subsidising activities like these Data Surgeries should be a priority for Arts Council England'
• Audiences UK believes the Data Surgeries can replicated in other regions

Mystery shopping

• recruitment of mystery shoppers was challenging. Effort spent in recruiting respondents who, when screened, did not fit the closely defined profile of the two target segments and had to be excluded would be eased with a broader recruitment specification

At the time of writing the mystery shopping programme was still running. Further learning will be added in late 2011.

Qualitative workshops

• panel recruitment was extremely challenging. The use of a specialist qualitative recruitment company for this function is recommended
• the qualitative workshops were a very hard-sell but having experienced the workshops many participants said they would recommend them
• few of the participating RFOs make significant use of discussion groups for qualitative research and most were not systematically using basic methods of collecting audience feedback
• the workshops served as a thought provoking reminder to participants of the importance of proactively listening to their market
• researchers found it difficult to distinguish themed findings based on a limited number of workshops with small panel sizes. However, the full data from the workshops is presented in a useful report by Audiences London
• Audiences UK was surprised by the lack of decent 'self-help' materials for organisations who want to conduct their own focus groups or develop on-going customer panels