1. Introduction
The project is led centrally by Jane Finnis, Director of Culture24, with strategic input and practical guidance on the use of Google Analytics from Seb Chan, Head of Digital, Social & Emerging Technologies, Powerhouse Museum. Culture24 also brought in a social media strategist from the commercial online works – Jenni Lloyd from Nixon McInnes, a social consultancy company based in Brighton, UK. Rachel Clements, an MA student at Central St Martins studying Innovation Management, carried out the mapping of social media activities across all partners.
Seventeen individual venues are taking part, a mixture of national museums, galleries, performing arts venues and other heritage organizations: British Library, National Gallery, Science Museum, Birmingham Museums & Art Gallery, Tate, British Museum, Kew, National Maritime Museum, Imperial War Museum, National Portrait Gallery, Watershed, Royal Shakespeare Company, National Museums Scotland, National Museum Wales, Design Museum, English Heritage and Roundhouse.
Culture24 have set up a collaborative funding arrangement with all of these partners, each demonstrating their commitment through a contribution of £1,000. Further matched funding has also been secured from a number of cultural sector agencies with vested interested in the research outcomes:
Department of Culture Media and Sport (DCMS): contribution of £5,000
Arts Council of England (ACE): contribution of £4,000
Heritage Lottery Fund (HLF): contribution of £3,000
Culture24 are also working with the University of Leicester on a new Scholarship in Digital Heritage. In 2010 the first PhD scholarship was offered to Elena Villaespesa to investigate museums' online performance and approaches for measuring the value and impact of online digital culture.
Basecamp is being used to conduct the majority of the project online with logins for all partners.
The starting point for the research was to:
- Explore what common ground there may be between organizations in trying to evaluate online activity
- Benchmark current activity across the sector in such a way that any future changes can be clearly identified and mapped
- Consider what lessons can be learned that might help to shape some advice or recommendations for the wider sector
- Highlight examples of good practice, begin to define what they mean and how they can inform good practice
- Make recommendations for software and configurations standards
- Help to define a better understanding of audience behavior online
- Create, test and publish simple guidelines for organizations
- Help organizations with the process of clearly defining their target audiences
- Improve understanding of user segmentation
2. Background to this proposal
Between 2003 and 2008, Culture24 carried out three comparative surveys across UK cultural websites, and assisted in the coordination of two international comparative surveys of cultural portals (http://www.culturemondo.org/documents/surveys). These attempted to collect information about the quantity and behavior of online audiences of those organizations' websites.
This body of work and evidence clearly highlighted the lack of clarity about what to collect, how to count it, and how to interpret what it is telling you. There was also clear inconsistency in the use of tools, with organizations counting and reporting different things using different systems.
In 2009, Culture24 called together a meeting of UK national & regional museums, research & educational organizations, plus strategic public sector agencies, to explore the possibility of a joint action research project. Those attending all responded to a call that was put out using the UK MCG (Museums Computer Group) e-mail list.
The resulting action research project has been directly informed by the outcome of that meeting and by the input and ideas of the group. Further input has come from a series of 'web metrics' workshops that Culture24 held in the UK, New Zealand, Taiwan and Luxembourg, led by Seb Chan as part of the international work of the Culturemondo network.
3. Face-to-Face meetings
The partnering organizations met as a group three times over the course of the project. Each meeting took the shape of a facilitated workshop and each had a different focus.
- Brighton July 2010: Agree on scoping and priorities for project.
- London October 2010: Defining user segmentation and social media priorities for benchmarking: This informed the research work at a more practical level.
- London January 2011: Review social media benchmarking and Google Analytics work to date: This ensured a sanity check of the work to date and threw light on any key issues that needed further attention.
These meetings have been key in providing a platform for shared learning and reflection. They also ensured a genuinely collaborative approach to the project, with each partner able to have a say and express any concerns.
A fourth and final group meeting is being planned for July 2011.
The first face-to-face meeting in Brighton, July 2010
4. Quantitative site analysis
In November 2010, each of the participating institutions provided access to its web analytics accounts. Each institution's installation and basic configuration was subjected to a 'health check', partially to ascertain the degree of compatibility between quantitative data and partially to assess the general Web analytics competence of each institution.
Most institutions use Google Analytics (GA) although this is not usually used for official reporting and it runs in parallel to older log-based systems. Two institutions used Urchin on their own servers instead of Google Analytics. The earliest GA installation dated back to 5 June 2006 whilst the youngest ran from 8 February 2010. On the whole, though, all but three institutions were able to have data collected and compared across all three desired analysis periods (Oct-Dec 2008, Oct-Dec 2009, Oct-Dec 2010).
Underscoring the difficulty of comparative analytics even with the same tools in use, configurations amongst the 14 institutions varied widely. Only one institution had set Google Analytics up in a way consistent with the tool's own documentation, and less than half had basic filtering in their accounts. Only a handful had successfully segmented out their internal traffic, and only one had set up a filter to remove case-sensitivity on URLs. Some had not integrated their e-mail and other online marketing activities effectively with Google Analytics, and only half had tracking of non-HTML pages such as PDFs.
Following the health check, data collection began using a series of custom segments that were shared with the group. It had been hoped that data collection could be done by each individual institution, but given the wide variance in setup and internal skill levels, the collection was done by Seb Chan externally and then shared on Basecamp for verification and discussion. Internal traffic was segmented out at the report level as a result of the lack of top level filtering.
The data which were collected for the fourth quarters of 2008, 2009, and 2010 were:
- Internal visits and visitors (collected to identify and exclude internal traffic)
- Global visits and visitors
- UK visits and visitors
- Institutional home city (or major home city for multi-homed institutions like IWM, Tate, etc) visits and visitors
- Organic search visits
- Paid search visits
- All social media referred visits
- Facebook referred visits
- Twitter referred visits
- Flickr referred visits
- Wikipedia referred visits
Mobile data was collected for the fourth quarter of 2010 only and segmented the following:
- Global visits
- UK visits
- Institutional home city (or major home city for multi-homed institutions like IWM, Tate, etc) visits
- iPhone/iPod visits
- iPad visits
- Android visits
- Other mobile OS visits
This base-level data allows for significant cross-institutional comparisons to be made, and the social media-related quantitative data can be correlated with the specific social media research undertaken in the project.
There are a number of inconsistencies in Google Analytics, especially when using custom segments and custom reports. The worst of these are variances in visit counts between report types – however, by using the same segmentation, the effect of these across institutions is at least consistent.
For counting Twitter, the project used referral traffic from Twitter.com, even though this is known to not contain traffic coming via Twitter apps. Despite this, the overall social media data suggests that even taking this into account the overall patterns and trends are reliable.
These figures could, in the future, be programmatically harvested using the Google Analytics API and a presentation/aggregator like Museumstats.org originally developed by the Walker Art Center.
5. Quantitative and qualitative social media approach
Recent years have seen rapid growth in the usage of social media in cultural organizations, with the medium fast becoming an essential part of most audience communication strategies. The potential of social media for extending audience reach and for engaging audiences in new ways makes it an attractive tool for organizations to explore. However, the enthusiastic uptake of this medium has been accompanied by a new set of challenges for organizations, one of the most pressing of which is how to evaluate success and how to measure return of investment (ROI) in relation to social media, especially as realization of the resource cost of social media becomes apparent.
Initial discussion within the project group during mid- 2010 highlighted a need for cultural organizations to be proactive in setting up meaningful measurement frameworks for social media. Without such a framework, it is difficult to demonstrate the quality and value of social media interactions with cultural audiences. The project group also recognized the need for benchmarking data, in order to compare social media performance across organizations with similar audiences and/or missions.
6. Aims
This part of the project aimed to collect cross-organizational benchmarking data which would provide the basis of a social media framework incorporating the following principles:
- Establish a baseline for comparison across organizations, creating equivalencies in the data to allow for cross-sector benchmarking.
- Move beyond measuring numbers and begin to measure quality / value of engagement and meaning: not just usage but also interaction.
- Recognize the importance of qualitative measures as well as quantitative.
- Establish a focus on trends over time rather than snapshot numbers.
7. Methodology
Data collection took place between October and December of 2010. For each of the 17 partner organizations, data was collected about social media usage, strategy and investment in order to build a more holistic picture of each organization's approach to social media.
The data collection process was split out into two distinct stages:
Stage 1 involved desk-based research, supplemented by e-mail exchanges, Basecamp discussions and phone calls in order to review each organization's presence on the main social media channels.
Stage 2 involved each partner organization filling out a series of questionnaires via e-mail and Survey Monkey, plus further discussion via e-mail, phone and Basecamp.
Stage 1
Aim = Collect qualitative and quantitative data to determine which social media channels each of the partner organizations is using, how many people they are reaching through these channels, and the level of user engagement.
The decision was made to focus on the major social media channels only, and therefore data was collected from Twitter, Facebook and YouTube. These platforms were chosen based on an initial review which indicated that these are the most common platforms currently in use. Blogs and Flickr were omitted from the initial study since their function and setup varies significantly between institutions, adding an extra layer of complexity to any comparison.
The aim of this stage was to take a snapshot of the performance of all partners on that platform at a specific point in time (i.e. reviewing all partner organizations on the same day to ensure a fair comparison). All partner organizations provided access to their Twitter and Facebook accounts for a period of 2 months to facilitate data collection. The desk-based research was supplemented by e-mail exchanges, Basecamp discussions and phone calls.
For Twitter, additional statistics for influence and user engagement were gathered using online tools available from Klout and Tweetlevel. The problems associated with the use of automated tools to collect these statistics were readily acknowledged, and issues with the slow rates of data refresh on these platforms were also taken into account. However, it was felt that the inclusion of this data in cross-organizational benchmarking would provide for a richer case study when the results from these tools were compared with the qualitative data.
The full list of data collected was as follows:
General information
- List of social media channels that each organization uses
- Strategy for linking to social media from the main website
- Size of the organization (based on number of physical visits)
- Number of followers
- Number of mentions
- Tweet profile for the organization
- Klout stats
- Tweetlevel stats
- Number of fans
- Growth rate in fans
- Fan demographics
- Amount of organizational content
- Amount of user-generated content
YouTube
- Amount of content
- Number of viewers and subscribers
- Amount of user-generated content
For each channel, qualitative information was also gathered on the tone of content (positive or negative) and the general "feel" of the site.
Stage 2
Aim = Collect data from each of the partner organizations on a) their social media strategies, and b) the resources they invest in social media.
Each of the partner organizations was asked to respond by e-mail to a set of questions about its social media strategy, and also to provide copies of relevant documentation. Additionally, an online survey was administered via SurveyMonkey to collect information about the resources each invest in social media. Additional discussion took place via e-mail, phone and Basecamp.
The questionnaires sought to explore the following themes:
Strategy
- Percentage of organizations that have a social media strategy
- Scope of strategies
- Success measures in relation to strategy
- Impact and effect of strategy
Investment
- Staff investment in maintaining, updating and creating content
- Staff investment in monitoring and reviewing performance
- Direct costs of social medias
- Software tools used
- Concepts of value in relation to social media
8. Feedback and analysis
Once data collection was complete, individual results were made available to each partner organization for checking and feedback, and the results were discussed via e-mail and Basecamp. The aim of this stage was to capture any additional social media activity that may have been missed from the review and also to account for any other circumstances that may have influenced an organization's figures during the data collection period. Once this initial feedback period was complete, the full dataset was then shared with all partner organizations on Basecamp, for further review and discussion.
The group then met in January, 2011, in London to review and discuss the results of the research. A number of issues and questions for further investigation emerged at this stage:
Investment in social media
- Do those organizations that invest more in social media have more success with their social media channels?
Social media strategy and frameworks
- The group recognized a need to provide guidelines to aid cultural organizations in creating effective social media strategies.
- The group felt that frameworks for monitoring social media performance in cultural organizations could be developed and trialed through the adaptation of existing frameworks from the business world.
Benchmarking across the main social media channels
- Which quantitative statistics are the most useful for benchmarking social media between different cultural organizations?
- How can we measure engagement the most effectively?
Qualitative data
- What are the uses and pitfalls of sentiment analysis for cultural organizations?
- How do different organizations' account styles vary across Facebook and Twitter?
Social Media referrals to websites
- What factors drive social media referrals most effectively?
- Is there a difference in the demographics and behavior of socially engaged segments of web traffic compared to other segments?
Analysis of the results is still ongoing.
However, one of the early outcomes of this project has been the development by James Morley, Website Manager at Kew Gardens, of an automated online tool that harvests publicly available metrics from organizations' Facebook pages, YouTube channels and Twitter profiles and captures them into a table for comparison. The tool is still in its beta version, but the plan is to refine the metrics that it tracks to match the research recommendations for cross-organizational comparison, and to add in an archiving facility to allow historic data to be analyzed and trends to be studied.
9. Timescale
- April 2011: Museum & the Web workshop – International Peer Review
- July 2011: Final face-to-face meeting of all project partners in London
- July 2011: Arts Marketing Association keynote on the project: Early outcomes and recommendations
- Autumn 2011: Culture24 conference "So What and Who Cares?": Publication of final report and recommendations.
10. Conclusion
The tangible end result of the project will be a published report with a series of recommendations to funding agencies and governments and with a set of guidelines and key issues for sector organizations.
Specifically, these will include shared guidelines on:
- How to carry out a 'Health checks' on your one website and your Google Analytic set up.
- Ideas and advice about useful audience segmentation, supported by 'shared segments' available for download and use in Google Analytics
- The need and arguments for a joined up strategy – not just 'digital' or 'social media' strategies: Framework for headings and topics.
- A working framework for social media evaluation: ways to measure both engagement and popularity.
Interest in the project among cultural venues in the UK has been high, and Culture24 intend to look at how the learning might be rolled out and implemented by other groups of venues who are willing to buy into the collaborative funding model.
Culture24 is also in discussion with Seb Chan about the possibility of continuing a Phase 2 project with the same group of venues in order to take the thinking further and consider in more detail the impact of mobile devices and platforms on our understanding of online success.
