museumsandtheweb.com - evaluation

Making Sense of Numbers: A Journey of Spreading the Analytics Culture at Tate

ElenaVillaespesa — Sat, 07 Apr 2012 17:21:02 +0000

MW2012

Elena Villaespesa, Tijana Tasich, Tate, United Kingdom

Abstract

Program Item Reference:

Making sense of numbers: a journey of spreading the online metrics culture across Tate

Exploring the Relationship between Visitor Motivation and Engagement in Online Museum Audiences

silviaf — Thu, 29 Mar 2012 05:26:28 +0000

MW2012

Silvia Filippini Fantoni, Rob Stein, Gray Bowman, Indianapolis Museum of Art, USA

http://www.imamuseum.org/

Abstract

In this paper, the authors will describe the rationale, methodology, and results of a series of studies that have been conducted with visitors to the Indianapolis Museum of Art website. The objective of the studies is to better understand people’s motivation for visiting the site and whether this motivation has an impact on the way they engage online. The hope is that these results will provide a reference dataset, and a replicable model for other museums that are interested in better understanding their online audience and in conducting similar studies for their own web efforts.

Program Item Reference:

Exploring the Relationship Between Visitor Motivation and Engagement in Online Museum Audiences

Delightfully Lost: A New Kind of Wayfinding at Kew

Natasha Waterson — Sat, 24 Mar 2012 22:04:12 +0000

MW2012

Natasha Waterson and Mike Saunders, Royal Botanic Gardens, Kew, United Kingdom

www.kew.org

Abstract

In October 2010, Kew Gardens commissioned an in-depth study of visitors’ motivations and information needs around its 300-acre site, with the express aim that it should guide the development of new mobile apps. The work involved over 1,500 visitor-tracking observations, 350 mini-interviews, 200 detailed exit interviews, and 85 fulfilment maps; and gave Kew an incredibly useful insight into its visitors’ wants, needs, and resulting behaviours.

It turns out that most Kew visitors have social, emotional, and spiritual, rather than intellectual, motivations during their time here. They do not come hoping to find out more, and they don’t want or need to know precisely where they are all the time. In fact, they love the sense of unguided exploration and the serendipitous discoveries they make at Kew—they want to become “delightfully lost.”

Program Item Reference:

Delightfully lost: a new kind of wayfinding at Kew

Levelling Up: Towards Best Practice in Evaluating Museum Games

dannybirchall — Sat, 24 Mar 2012 21:41:49 +0000

MW2012

Danny Birchall and Martha Henson Wellcome Trust; Alexandra Burch and Daniel Evans, Science Museum, UK; Kate Haley Goldman, National Center for Interactive Learning at the Space Science Center, USA

Abstract

Museums make games because games can provide compelling educational engagement with museum themes and content, and the market for games is enormous. Truly understanding whether games are achieving your goals requires evaluation. In this paper, we identify the kind of games that museums make and use case studies of our own casual games to look at the benefits and means of evaluation. Beginning by identifying different kinds of evaluation within the broad framework of formative and summative practices, we suggest ways to plan an evaluation strategy and set objectives for your game. We then look in detail at evaluation methods: paper and wireframe testing, play-testing, soft launching, Google Analytics, surveys, and analysing responses “in the wild.” While we draw on our own experience for examples of best practice, we recognize that this is an area in which everyone has a lot to learn, and we conclude by suggesting some tactics for sharing knowledge across the museums’ sector.

Program Item Reference:

Levelling up: Towards best practice in evaluating museum games

What do we really know about social media in museums?

danaallengreil — Fri, 30 Sep 2011 21:32:10 +0000

MW2012

Keywords:

social media

Keywords:

twitter

Keywords:

Facebook

Keywords:

metrics

Keywords:

best practices

Keywords:

evaluation

Abstract:

With years of collective experience in social media engagement, many museums are stopping to ask themselves: what have we really learned about the audiences we are connecting with? Have we formed more meaningful relationships with our community and in the process, are we better serving the missions of our institutions? Could the future of our digital outreach be primarily based on social engagement and if so, are our organizations adapting rapidly enough to be prepared for this social future? Are our current social media practices engaging online communities to their greatest potential, or are we missing out on crowdsourcing, collaboratively learning, or other as-yet undiscovered social experiences? How can we better evaluate the success of our efforts and take our social media engagement to the next level?

This extended professional forum will use a range of social media platforms to engage museum professionals attending Museums and the Web and beyond in this discussion. This social media event component will begin before the meeting in San Diego 11-14 April using a wiki and other social media platforms, and the conversation about what we really know about social media, museum content, and cultural institution audiences will surely continue long afterwards as well. The aim is to engage fellow practitioners in defining the myths and assumptions we most want to “bust,” and to gather data in order to formulate testable hypotheses for this provocative discussion, culminating in a special professional forum at the MW2012 conference.

Type:

Professional Forum - discuss an issue

Professional Forum on "What do we really know about social media in museums?"

Site:

http://si.edu/Connect

Authors:

Elissa Frankle

Georgina Goodlander

Victoria Portway

Jeff Gates

Darren Milligan

Brian Alpert

Erin Marie Blasco

Sarah Banks

David Klevan

Amelia Wong

Delivering mobile from the ground-up: exploring the approach

hughw — Fri, 30 Sep 2011 15:29:16 +0000

MW2012

Published Paper:

If mobile is the answer, what was the question?

Keywords:

mobile

Keywords:

digital

Keywords:

framework

Keywords:

evaluation

Keywords:

interpretation

Keywords:

access

Abstract:

The challenges of delivering a meaningful ‘mobile’ experience for an institution are not always technology-related. Securing cross-departmental stakeholder investment and ensuring alignment with the institution’s core mission have been key for National Museums Scotland and National Museum Wales in developing a framework for their mobile programme. This paper will explain how these organisations sought to overcome organisational challenges from the outset; pushing forward ideas and taking a common approach to planning mobile projects that can be adopted by other institutions.

National Museums Scotland and National Museum Wales have many similarities in their structure and set-up. Both operate a main base out of their respective capital cities (Edinburgh and Cardiff) and have a mix of specialist rural and urban-based museums. Since 2009 the proportion of online visitors from mobile devices has increased significantly for both organisations; taking this on board alongside in-gallery research and predicted market trends, both have begun developing a programme of mobile interpretation using the same user-focused approach.

For the National Museum Wales this is about making information accessible, as well as creating a link between historic buildings and collections in a rural setting at the National History Museum at St Fagans. For National Museums Scotland it’s about bringing new interpretation and innovation to the recently-reopened National Museum of Scotland. Importantly, neither approach has focused on technology in the first instance - workshop sessions have concentrated on getting the right people to take a balanced view of audience needs and create design briefs that tie closely to each organisation’s strategic objectives.

Both projects have their specific challenges and the session will explore what led them to take certain decisions, what the outcomes have been to date, the broader implications for developing mobile products in terms of cost and impact on the museum technical infrastructure, and the lessons they feel will benefit other museums looking to develop their own plans.

Type:

Paper - in formal session

Optimizing engagement

Authors:

Hugh Wallace

Dafydd James

Loic Tallon

Levelling up: Towards best practice in evaluating museum games

dannybirchall — Fri, 30 Sep 2011 15:22:48 +0000

MW2012

Published Paper:

Levelling Up: Towards Best Practice in Evaluating Museum Games

Keywords:

games

Keywords:

evaluation

Keywords:

best practice

Keywords:

interaction

Keywords:

audiences

Keywords:

gamification

Abstract:

Games, and game-like experiences, are becoming increasingly important to museums seeking to engage new audiences and to provide deeper engagement with existing audiences. Demonstrating the value of commissioning and producing games as a core activity of museums is imperative. At the same time, the sector is encountering a new, cold wave of realism in evaluating its online presences. Old canards about new audiences are being thrown out the window, and a more rigorous focus is being applied to the business of gaining and measuring audience attention.

There is a risk, however, that these engagement metrics, oriented towards page-based content and measurable proxies for interaction in social media are poorly suited for the kind of immersive, deep engagement that good games can offer. This paper will identify and discuss best practice in the field of museum games evaluation in order that such evaluation might also move towards standardisation and wide acceptance, with the end result of improving games in our field.

Most museum games are web-based, and have an explicit educational intent. Examples from our own experience include High Tea, Memory (Wellcome Collection), Launchball, Rizk and Thingdom (Science Museum). These types of games will be the focus of this paper, though we expect the methodology to have applications across other gaming activities.

Drawing from our own institutional practices, across the cultural heritage and informal learning sectors, and looking at the area of ‘persuasive’ games in general, we will focus on initial objective setting and three areas of evaluation:

1. Quantitative evaluation: What are the best tools and methods to measure overall interactions, in-game interactions, and profile game achievement against audience segments?

2. Qualitative evaluation: How can we use interviews, surveys, and focus groups to gain a deeper understanding of player motivation and experience and are they sufficient? Can we find more useful and rich ‘in the wild’ where people freely discuss games? If we choose that route, what are the best approaches to gathering and analyzing that data?

3. Pre vs Post-release evaluation: How reliable or valid are expert reviews and user testing of games prior to release? How do these results compare with those from summative evaluations?

In each of these areas, we will also consider aligning evaluation approaches for different types of games: autonomous games, and gamified or gameful experiences. Technical appendices will describe both quantitative and qualitative methods.

Type:

Paper - in formal session

Pervasive Fun

Authors:

Danny Birchall

Kate Haley Goldman

Daniel Evans

Martha Henson

Delightfully lost: a new kind of wayfinding at Kew

Natasha Waterson — Fri, 30 Sep 2011 14:36:26 +0000

MW2012

Published Paper:

Delightfully Lost: A New Kind of Wayfinding at Kew

Keywords:

mobile

Keywords:

audience research

Keywords:

apps

Keywords:

evaluation

Keywords:

science

Keywords:

wayfinding

Keywords:

botanic garden

Abstract:

In October 2010, Kew Gardens commissioned an in-depth study of visitors’ motivations and information needs around its 300 acre site, with the express aim that it should guide the development of new mobile apps. The work involved over 1500 visitor tracking observations, 350 mini interviews, 200 detailed exit interviews and 85 fulfilment maps and gave Kew an incredibly useful insight into its visitors’ wants, needs and resulting behaviours.

It turns out that most Kew visitors have social, emotional and spiritual, rather than intellectual, motivations during their time here. They do not come hoping to find out more and they don’t want or need to know precisely where they are all the time. In fact, they love the sense of unguided exploration and the serendipitous discoveries they make at Kew – they want to become ‘delightfully lost’.

This paper will explore how this perhaps counter-intuitive idea – to help visitors become ‘delightfully lost’ – has influenced mobile thinking at Kew. What impact has it had on mobile concept development, design, and marketing? It will then ask whether this principle might reasonably apply in other outdoor visitor attractions, museums and galleries – using existing case studies to draw out parallels and differences.

Finally this paper will look at the results of Kew’s summative evaluation of its garden app – does quantitative and qualitative data suggest we achieved our aims? And what does this mean for future mobile developments across the Gardens?

Type:

Paper - in formal session

Pervasive Fun

Site:

http://www.kew.org/mobileapp

Authors:

Natasha Waterson

Mike Saunders

Museums in perspective: share and discuss daily data about museums and their audiences.

rui — Fri, 30 Sep 2011 14:28:14 +0000

MW2012

Keywords:

evaluation

Keywords:

social media

Keywords:

analytics

Keywords:

participation

Keywords:

engagement

Keywords:

community

Keywords:

strategy

Abstract:

It is common practice among museums to publish quarterly or annual reports that include the museum attendance figures, online visits and more recently information about their social network activities. These figures are generally accepted as indicators of the effectiveness of museums activities both online and offline. While this information can give some hints about how organizations have evolved throughout the years, it is often hard to understand what these values mean without contextualizing them with information from other organizations. For example, what does it mean to have tens of thousands of followers on Twitter? Do the changes on annual attendance numbers reflect a global trend or a new local policy? Some of these questions can be better understood once figures from different cultural organizations are analyzed comparatively.

This paper introduces an online tool that collects information about more than 3000 museums and galleries. The published information includes annual attendance numbers, exhibition attendance numbers, online visits, as well as information about popularity and engagement on social networks. The information that is automatically collected or contributed by organizations can be visualized as interactive graphs and displayed comparatively with information from other organizations. The online tool introduced in this paper is accompanied by an online platform where museum professionals can discuss topics related to the published information, such as, issues and limitations of comparative studies and adequate online and offline strategies.

The information that is now being collected on a daily basis has been previously published in a scattered manner for several years. The Art Newspaper yearly publishes the attendance numbers of the most visited museums and exhibitions world wide. More recently Culture24 has published a report that analyses both website visitors and social network in terms of popularity and engagement of several organizations. Sean Redmond has created an automated data collection tool that daily reports the popularity on Facebook and Twitter of thousands of museums and galleries. At the Walker Art Center a tool has been developed that automatically aggregates and shares information from google analytics of several museums. These initiatives were important precedents that led to the development of an online tool where all this information is cohesively presented and contextualized.

Rather than just introducing an online tool, this paper also presents new insights encountered while developing the tool and comparing data. By listening to the communication channels of more than 3000 museums and galleries, it is possible to infer information specific to the sector. For example, it is possible to answer questions such as: what are the current trends in the sector being discussed on twitter? Which content published by a museum has been most shared online? We hope that by answering these and other questions, organizations will be able to understand better their online and offline presence and finally be able to concentrate on their specific goals rather than on numbers.

Type:

Paper - in formal session

Case studies in reading web metrics

Site:

http://www.museum-analytics.org/

Authors:

Rui Guerra

Using QR codes, mobile apps and wifi tracking data to understand visitor behaviour in exhibitions

sebchan — Fri, 30 Sep 2011 05:12:29 +0000

MW2012

Keywords:

QR codes

Keywords:

mobile

Keywords:

evaluation

Keywords:

r&d

Abstract:

In the last twelve months QR codes have reached a new level of hype within the museum community - increasing as their proliferation in street advertising in the USA and Europe has increased. Even with more deployments of QR codes into gallery spaces there has been little quantitative or qualitative research thus far on how visitors actually use these codes, whether they deliver real value, and, importantly, whether their presence changes the 'gallery experience' for users and non-users alike.

In early 2009 the Powerhouse Museum experimented unsuccessfully with QR codes but again tried implementing them in a new temporary exhibition in 2011. Coming from the failures of 2009 and the need to understand more about how visitors used the codes, a three tiered approach was designed to understand the behaviour of smartphone carrying visitors in an exhibition. The approach used a combination of technology implementations - free public wifi, QR codes on every object label, a free Android and iOS mobile app containing rich exhibition catalogue content and, crucially, a built-in QR scanner, as well as wifi user tracking, and automated threshold counting.

This paper discussed the technical implementation of the project and how the design of the project evolved as the exhibition launched and then attracted visitors. The paper reveals the results of the different tracking technologies - what proportion of exhibition visitors scanned the QR codes; which objects were scanned; which objects were most viewed on the App; dwell times by wifi enabled smartphone; and, what proportion of visitors carried wifi enabled smartphones into the exhibition during the sampling period.

By collecting this data together and visualising it, the paper makes recommendations for new ways of (re)designing exhibitions with a better understanding of how smartphone users navigate them - as well as how to more successfully implement QR codes and other visitor-driven technologies in galleries.

Type:

Paper - in formal session

Tools for tapping into visitor behavior

Site:

http://www.powerhousemuseum.com/lovelace

Authors:

Sebastian Chan

Luke Dearnley

Let's Get Real

JaneFinnis — Wed, 28 Sep 2011 10:03:33 +0000

MW2012

Keywords:

evaluation

Keywords:

success measures

Keywords:

Failure

Keywords:

analysis

Keywords:

action research

Abstract:

‘Let’s Get Real: How to evaluate online success?’ is a report published by Culture24 in September 2011 at their UK conference of the same name.

It looks at the work from a unique action research project that took place in the UK between June 2010 and Sept 2011 involving 24 organizations and agencies from across the cultural heritage and arts sector. The report details the collaborative process of the project, and shares in-depth information on the data and learning gained by the organisations involved.

The key findings in the report will be the starting point for this professional forum and will be the subject for an open discussion and debate led by the report authors (and project leads) Jane Finnis and Seb Chan.

Some of the findings in the report are challenging for the cultural sector and could be seen as a wake up call to ‘get real’ about the way we measure and define value online. The report itself champions a more honest appraisal of success online in recent years and a willingness to openly admit and learn from mistakes.

Jane and Seb will present the thinking around each of the key findings from the report before chairing an open discussion on the challenges these findings present.

You can download the published report here: http://weareculture24.org.uk/projects/action-research/

Full list of partners in the action research project: British Library, Roundhouse, National Gallery, Science Museum, Birmingham Museums and Art Gallery, Tate, British Museum, Kew, National Maritime Museum, Imperial War Museum, National Portrait Gallery, Watershed, Royal Shakespeare Company, National Museums Scotland, National Museums Wales, Design Museum, English Heritage, Department for Culture, Media and Sport, Arts Council of England, Heritage Lottery Fund and Museums, Museums, Libraries and Archives Council (MLA), University of Leicester, Powerhouse Museum and Culture24

Type:

Professional Forum - discuss an issue

Professional Forum on Let's Get Real

Site:

http://weareculture24.org.uk/projects/action-research/

Authors:

Jane Finnis

Sebastian Chan

Rethinking Evaluation Metrics in Light of Flickr Commons

jdalton — Thu, 31 Mar 2011 15:15:07 +0000

MW2011

Paula Bray, and Sebastian Chan, Powerhouse Museum, Australi; Joseph Dalton, New York Public Library, USA; Dianne Dietrich, Cornell University Library, USA; Effie Kapsalis, Smithsonian Institution Archives, USA; Michelle Springer and Helena Zinkham, Library of Congress, USA

Abstract

In the past several years, cultural heritage institutions, including archives, libraries, and museums, have been placing their collections in Web spaces designed for collaboration and communication. Flickr Commons is one example of a highly visible space where cultural heritage institutions have partnered with a popular social networking site to provide greater discovery to, access of, and opportunities to interact with image collections on a large scale. It is important to understand how to measure the impact of these kinds of projects. Traditional metrics, including visit counts, tell only part of the story: much more nuanced information is often found in comments, notes, tags, and other information contributed by the user community. This paper will examine how several institutions on Flickr Commons - the Library of Congress, the Powerhouse Museum, the Smithsonian, New York Public Library, and Cornell University Library - are navigating the concept of evaluation in an emerging arena where compelling statistics are often qualitative, difficult to gather, and ever-changing.

Keywords: Flickr, metrics, evaluation, crowdsourcing, statistics, images

Program Item Reference:

Rethinking Evaluation Metrics in Light of Flickr Commons

How to Evaluate Online Success? A New Piece of Action Research

JaneFinnis — Tue, 29 Mar 2011 23:24:23 +0000

MW2011

Jane Finnis, Culture24, United Kingdom; Seb Chan, Powerhouse Museum, Australia; and Rachel Clements, Central Saint Martins, United Kingdom

Abstract

How to measure and define the success of cultural websites and online services is a problem common to all parts of the cultural sector – museums, archives, galleries, arts organizations, libraries and publishers. Worse, funding agencies and government departments lack the expertise to offer guidelines or set standards. The result is a confusing mixture of statistics and reporting that reveals little about genuine user satisfaction.

In mid-2010, a group of 24 organizations and agencies from across the cultural heritage and arts sector in the UK began collaborating on a piece of action research addressing the issues around 'evaluating online success'. Coordinated by Culture24 and working with Seb Chan, the groups of institutions have been developing a framework and best practice guide for measuring online projects both on organizational websites and those presented through social media channels.

Keywords: evaluation, metrics, evaluating cultural websites

Program Item Reference:

How to Evaluate Online Success – New action research with 24 institutions

Tales of the Unexpected: A Pragmatic and Candid View of Life Post-Launch

biblio — Thu, 13 Jan 2011 22:34:58 +0000

Mitroff Silvers, D., Sexton C., & Burnette A. (2010). Tales of the Unexpected: A Pragmatic and Candid View of Life Post-Launch. (Trant, Jennifer, David Bearman, Ed.).Museums and the Web 2010. Proceedings.

Rapid Prototyping and Collaboration Using Google Wave

biblio — Thu, 13 Jan 2011 22:34:56 +0000

Koepfler, J., & Maki J. (2010). Rapid Prototyping and Collaboration Using Google Wave. (Trant, Jennifer, David Bearman, Ed.).Museums and the Web 2010. Proceedings.

Mobile Crit Room

dbear — Mon, 29 Nov 2010 22:44:37 +0000

Type:

Session

Theme:

evaluation

Chair:

Sebastian Chan

Date & Time:

April 8, 2011 - 2:30pm - 4:30pm

Location:

Commonwealth A/B

mobile

crit room

evaluation

Required fields are marked with a red asterisk (*).

A confirmation of your registration will be sent to your email address. Please print it out and bring it to the conference.

You will not receive registration confirmation by regular mail.

If you do not receive your confirmation today, please contact us.

Archives & Museum Informatics EIN: 77-0708617; GST / BN 887978914

Description

For fifteen years we've had a 'Crit Room' to create dialogue with web site designers and explore their visions and implementations. Now that there are mobile applications with different design demands, it's time to give designers of these experiences a place to expose what they have done to the same critical review. Come to see examples of the best, and learn what other designers think about them.

Professional Forum on Rethinking Evaluation Metrics

dbear — Mon, 29 Nov 2010 22:32:40 +0000

Type:

Session

Programs:

Rethinking Evaluation Metrics in Light of Flickr Commons

Date & Time:

April 8, 2011 - 11:00am - 12:00pm

Location:

Congress

metrics

measurement

use

flickr

evaluation

Required fields are marked with a red asterisk (*).

A confirmation of your registration will be sent to your email address. Please print it out and bring it to the conference.

You will not receive registration confirmation by regular mail.

If you do not receive your confirmation today, please contact us.

Archives & Museum Informatics EIN: 77-0708617; GST / BN 887978914

Description

What are we measuring when we measure traffic on our Web site and how does it relate to our visibility on the Web? When we seek broad exposure on a range of social sites, and achieve it, how does it effect our data about use? Why are we measuring anyway, and what ought we be trying to learn?

Mini-Workshop on How to Evaluate Success

dbear — Mon, 29 Nov 2010 21:19:18 +0000

Type:

Session

Programs:

How to Evaluate Online Success – New action research with 24 institutions

Date & Time:

April 8, 2011 - 10:00am - 11:00am

Location:

Congress

metrics

evaluation

assessment

Required fields are marked with a red asterisk (*).

A confirmation of your registration will be sent to your email address. Please print it out and bring it to the conference.

You will not receive registration confirmation by regular mail.

If you do not receive your confirmation today, please contact us.

Archives & Museum Informatics EIN: 77-0708617; GST / BN 887978914

Description

Rethinking Evaluation Metrics in Light of Flickr Commons

dd388 — Thu, 30 Sep 2010 17:16:59 +0000

MW2011

Published Paper:

Rethinking Evaluation Metrics in Light of Flickr Commons

Keywords:

flickr

Keywords:

digital images

Keywords:

statistics

Keywords:

evaluation

Keywords:

outreach

Keywords:

online image collections

Abstract:

Flickr Commons launched in 2008 as a platform for enhancing the discoverability of cultural heritage collections and for allowing users to tag, annotate, and repurpose these materials. The Commons now boasts 46 members and tens of thousands of public domain images. The impact of the project has been monumental. Many of the institutions involved can boast hit counts on their Flickr images that far exceed other digital collections. While many of the numbers are certainly impressive, it can be argued that they are one of the least compelling statistics.

What, then, are the most meaningful statistics for a project that puts library collections in a highly visible space outside the library's traditional confines? Some of our standard and more established metrics, such as hit counts, only illuminate a small part of the story – the bigger picture includes the myriad ways users have interacted with our images, by adding notes and comments, by including them in their own curated galleries, by tweeting them, and referencing them on their blogs. There are limitless possibilities for users to interact with digital materials and we should be developing metrics that reflect this landscape. This forum will examine multiple approaches to assessing the impact of a digital library collection that exists beyond an institution's borders.

Type:

Professional Forum - discuss an issue

Professional Forum on Rethinking Evaluation Metrics

Authors:

Dianne Dietrich

Joseph Dalton

Effie Kapsalis

Sebastian Chan

Michelle Springer

Helena Zinkham

Paula Bray

Playing with museums - game designs to improve museum collections

mia — Thu, 30 Sep 2010 12:39:59 +0000

MW2011

Published Paper:

Playing with Difficult Objects – Game Designs to Improve Museum Collections

Keywords:

crowdsourcing

Keywords:

games

Keywords:

evaluation

Keywords:

research

Keywords:

collections

Abstract:

Crowdsourcing data through games is an attractive proposition for museums looking to maximise use of their collections online without the requirement to commit intensive curatorial resources to enhancing catalogue records. This paper investigates the optimum game designs to encourage participation and the generation of useful data.

Further detail

Projects such as Armchair Revolutionary [1], Carnegie Mellon University's 'Games with a purpose' [2] and InterroBang?! [3] are indicative of the trend for 'games for social good'. Crowdsourced projects such as the Guardian newspapers examination of MPs expense claims [4], the V&A Museum's image cropping [5], Brooklyn Museum's tagging game [6], the National Library of Australia's collaborative OCR corrections [7]; Chen's study of the application of Csikszentmihalyi's theory of 'flow' to game design; and McGonigal's ideas about multiplayer games as 'happiness engines' [8] all suggest that 'playful interactions' and crowd participation could be applied to help create specific content improvements on museum sites. Game mechanisms may help make tasks that would not traditionally be considered fun or relaxing into a compelling experience.

Games designed to crowdsource museum metadata are particularly interesting when limited resources within museums make it difficult to enhance and publish large bodies of collections data in ways that are accessible to non-specialist visitors. Further, museum metadata games are a form of collections access in their own right and have the potential to reach large, traditionally hard-to-reach audiences. Data generated may include terms to improve discoverability by adding alternate terms used by the general public (as in the steve.museum project); information about which objects are more interesting for different types of visitors; data that could be provided by curators and researchers if resources were unlimited but could also be solicited from specialists outside the museum; and experiential data for objects and events within living memory.

Based on a Masters Dissertation project ('Game mechanics for social good: a case study on interaction models for crowdsourcing museum collections enhancement', http://openobjects.blogspot.com/2010/06/game-mechanics-for-social-good-case.html, due January 2011) this paper will cover three topics:

research summary of emerging best practice about games designed to crowdsource metadata

analysis of the most effective elements of game mechanics for application to interfaces for crowdsourcing museum collections enhancement, based on the evaluation of games designed as part of the project and the evaluation of data produced as the games are played.

an emerging typology of game designs applicable for different models of collection data, based on categories such as breadth of collection data (from selected hero objects to the entire catalogue), the type of metadata desired, the level of specialist knowledge required to interpret objects, etc.

Games built with collections from museums of social history and history of science and technology will be available at http://museumgam.es/ from mid-October.

References

[1] https://www.armrev.org

[2] http://www.gwap.com/

[3] http://www.playinterrobang.com/

[4] http://mps-expenses.guardian.co.uk/

[5] http://collections.vam.ac.uk/crowdsourcing/

[6] http://www.brooklynmuseum.org/opencollection/tag_g...

[7] http://newspapers.nla.gov.au/ndp/del/home

[8] http://www.futureofmuseums.org/events/lecture/mcgo...

Type:

Paper - in formal session

Games

Authors:

Mia Ridge

How to Evaluate Online Success – New action research with 24 institutions

sebchan — Wed, 29 Sep 2010 11:49:32 +0000

MW2011

Published Paper:

How to Evaluate Online Success? A New Piece of Action Research

Keywords:

metrics

Keywords:

collaboration

Keywords:

social media

Keywords:

best practice

Keywords:

analytics

Keywords:

evaluation

Abstract:

How to measure and define the success of cultural websites and online services is a problem common to all parts of the cultural sector – museums, archives, galleries, arts organisations, libraries, publishers. Worse, funding agencies and government departments lack the expertise to offer guidelines or set standards. The result is a confusing mixture of statistics and reporting that reveals little about genuine user satisfaction.

In mid 2010 a group of 24 organisations and agencies from across the cultural heritage and arts sector in the UK began collaborating on a piece of action research addressing the issues around 'evaluating online success'. Coordinated by Culture24 and working with Seb Chan, the group of institutions have been developing a framework and best practice guide for measuring online projects both on organisational websites and those presented through social media channels.

This mini-workshop will present the findings of the project to date, along with several case studies from partner institutions, anonymised where necessary, demonstrating the impact of metrics customisations, cross-sector audience segmentation, and benchmarking.

The full list of project partners is: