By Kate Haley Goldman
Each week brings across my desk a fresh set of mobile market studies indicating how the proliferation of smartphones continues at a dramatic pace, web access is more and more mobile, phones have changed teen culture, phones have changed the culture for the rest of us, and smartphone domination is in our near future. Having established near ubiquity of phones in general, most studies foresee a continued acceleration of the adoption of the smart phone. The adoption acceleration has extended to museums, and in recent years is starting to become less talk and more actual products on the floor. Multiple recent studies have tackled the institutional perspective of mobile adoption and barriers to implementation. Among the most relevant for the museum field are the 2011 studies sponsored by AAM and Pocket Proof/Learning Times. This chapter will interpret the findings of these studies and frame questions for future studies.
The study by Fusion Analytics for AAM (supported by Guide by Cell) focused on the state of current and future institutional incorporation of mobile capabilities. The sample was somewhat larger than the study by Pocket Proof, focused primarily on the United States, and had a geographically wide and institutionally broad distribution. One key finding was that under half of American institutions within the study currently offer mobile interpretation. For those institutions that did not yet have mobile, the primary reasons were lack of budget, staff time and other resources. Most non-mobile institutions did not express a concern that they did not want their visitors using mobile devices, but they were concerned about visitor interest. Over one-third of museums without mobile products listed lack of visitor demand as why they did not offer such products. By comparison, the study from Pocket-Proof and Learning Times asked museum professionals about their attitudes towards the creation, implementation, and maintenance of mobile applications, segmenting the results by the differing challenges for those institutions who already have a mobile interpretation product compared to those who are planning to, but do not yet have such as product.
Both studies are admirable in their effort to provide baseline data and context for the proliferation of museum mobile projects. Looking at their data, it would seem that the vast majority of museums are currently working on such projects. And while that generalization may in fact be true, unfortunately we can’t be sure from the data presented within these studies.
Generalizability is a tricky proposition with research and evaluation studies. Fundamentally, research is judged on its reliability and validity. Reliability is defined as the consistency or stability of a measure from one test to the next. An accurate oven thermometer is reliable, measuring 350 degrees in the same fashion each time. Validity is the overall term used to describe whether a measure accurately measures what it is supposed to measure. An accurate oven thermometer might be consistent in temperature measurement but does not measure whether the food is hot enough. Oven temperature is not a valid measure of food temperature. It is possible to have reliable but not valid results: results that are repeatable in multiple testings but still do not measure the appropriate underlying construct. The oven might be consistent in the temperature measurement, but it still does not represent a valid measure for the temperature of the food.
Reliability is influenced by the design of the questionnaire, but most profoundly by the sampling within a project. The Pocket Proof study uses a nonprobability sampling: respondents were recruited via list-serves, social media and other outreach. While the sample sizes are large; there is no certainty that the population sampled is representative of the overall population of museums in the United States; indeed they are likely to be museums with strong connections to social media, and by extension, use of technology within the museum. This may bias the results in terms of the numbers who have mobile projects or are considering them, or bias in other ways, such as size of institution, type of mobile project attempted, etc.
The AAM study had a well-defined sampling frame: AAM member institutions and individual members. While there is likelihood of some amount of nonrandom measurement error, the issue of most concern is the response rate. Response rates are notoriously difficult on web-based surveys, even for surveys such as this one with incentives offered and a dedicated (but busy) membership. At what point can a response rate be deemed reliable? In classic survey response methodology, a response rate of 60%, though preferably 80%, is seen as acceptable for analysis and generalization purposes (Dillman, 2000). This is an extremely high bar to reach for a web survey of a general membership. Yet, with the 14% response rate of the AAM study, one could repeat the exact same study next year and have an entirely different 14% of the population respond with entirely different answers. Whether the implementation of mobile projects had gone up, down or remained constant, it would be impossible to say reliably. For the Pocket Proof study, we cannot calculate a response rate, as we don’t know how many individuals saw the survey request. We can not say with certainty what percentage of museums are currently conducting a mobile project, planning a mobile project, whether mobile projects are more common in certain types or sizes of institutions. While the concept that mobile interpretation is more prevalent in large art museums has significant face validity, due to the combination sampling strategy and response rate we have no consistent, reliable numbers.
While the results within these studies may not be generalizable, they still provide some valuable insights. One of the most useful aspects of both studies is the documentation of barriers faced by museums in implementing mobile projects. The percentages aren’t necessarily relevant here, but the rank order is. These lists provide institutions a list of most prevalent obstacles.
Setting aside the idea that some set of museums underrepresented within the surveys might face significantly different obstacles towards implementing their mobile project, the barriers faced by those that are already engaged in such projects, and by those contemplating such projects, is illuminating. As the Pocket Proof/Learning Times study notes, for institutions already using mobile interpretation, encouraging visitors to use the mobile interpretation was the largest challenge. Yet for others — vendors and researchers, as well as those considering projects — attracting new visitors via mobile was a primary goal. This disconnect represents a great opportunity for future research. Despite the numbers of institutions exploring mobile and the availability of phones, usage rates remain below 10% for permanent galleries (Proctor, 2010) For those contemplating employing a mobile interpretation strategy in the Pocket Proof study, the lack of visitor interest was at the bottom of the list of obstacles, whereas for those currently offering mobile interpretation, this issue was more of a concern. That many visitors simply find the concept of using their phone unappealing in this context (Haley Goldman, 2007) is a finding that must be further explored.
Mobile Research Opportunities
The groundwork laid by these studies promotes reflection on what sort of studies are still needed. Both of the studies here had the foresight to ask their participants what type of research is still needed within the industry. Building on those concepts from a researcher’s perspective, I propose three avenues of future potential research.
1. Visitor-based research: I know from personal experience that evaluation and research on visitor use of mobiles in museums is extraordinarily difficult. The small sample size makes it difficult to get a full picture of usage, the interpretation strategy creates difficulty in finding those who are using their mobiles; the list of challenges goes on and on. The research on mobiles in museums is overwhelming institutionally-oriented, as is the motivation for many of the museum implementation projects. Some institutions wish to avoid the overheads of providing audio tour devices to visitors, others wish to try to engage their public in more technologically current ways. The research described here focuses on the museum, their project readiness, their motivations and their obstacles. The visitors have a significantly different perspective, including their own motivations and barriers to use. The gap in visitor use can be examined through a number of lenses: how individuals adopt uses of technology, how visitors perceive their phones, and visitor motivations and desired outcomes for their museum visit.
Information-seeking is one of only many potential uses of an individual’s phone (compared to social utility, accessibility, status, etc.), and is not by any means the most common use (Wei and Lo, 2006). Thus visitors’ perception of their phones does not immediately indicate the phone’s usefulness as an interpretative device. Whether visitors are likely to use their phones for interpretation depends on their goals for their museum visit.
Similarly, while learning is a key element in many visitors’ articulations of why they choose to visit a particular museum on a particular day (Kelly, 2007), it is not the only motivation for a visit. Visitors come for destination visits (been there, done that), to spend meaningful time with family or guests, and for many other reasons. (Falk, Moussouri & Coulson, 1998). If the use of their phone for interpretation does not immediately further their goals for the visit, visitors will not make use of the opportunity. In-depth qualitative research exploring the relationship between visitor motivations, expectations of the mobile product, and resultant museum experience would help developers create better visitor personas. These personas, based on how visitors have adopted the technology, as well as their motivations and social groupings, would create better products.
2. Case Studies
As tempting (and useful) as it is to cast a wide net to look at mobile adoption across the field, analysis at a micro-level, both institutionally and from a visitor perspective, would be extremely illuminating. The AAM study notes case studies (and visitor research) at the top of the list for future studies. The generation of case studies would be an excellent complement to the analysis of barriers that each project might face. The AAM study notes that there are widely divergent goals for these projects, from increasing visitor engagement to marketing to bringing collections to a broader audience. Separate case studies by project goal would allow institutions to focus on the strategies most appropriate for their goals. The call for case studies is not new, and yet the number of viable case studies compared to the estimated number of projects occurring is very small. Pulling together a case study is quite difficult when deeply embedded within a project, and even more difficult to do in a way that is comparable with case studies from other institutions. A single collaborative project or effort to research and generate case studies from multiple institutions would provide the most comparison points for other institutions.
3. Stratified and/or Longitudinal Field-Wide Studies
For future studies designed to look at the landscape of mobiles in museums from an institutional perspective, perhaps rather than casting a field-wide net, it would be more profitable to cast a series of smaller, more fine-grained nets over time. Given that the museum field is so vast (the AAM study had thousands of responses, and yet that only results in a 14% response rate), future field-wide approaches should invest in stratified or longitudinal studies. A study of a stratified sample of museums would allow closer examination of factors that may influence development of mobile products, such as number of visitors or content focus of the institution. Longitudinal studies would also allow a more reasonable sample size, but could chart the change in mobile developments over time. Do institutions face the same barriers they did a year ago? For those just contemplating a mobile product, were they able to create one or are they still in contemplation?
In conclusion, there is much to be done, both on the product development portion of the mobiles in the museum field, and in the research components. These studies provide any important first step. The next steps should perhaps be narrower, but deeper and much more deeply tied to the visitor.
Kate Haley Goldman, director of learning research and evaluation, National Center for Interactive Learning, Boulder, Colo.
- American Association of Museums, 2011. 2011 Mobile Technology Survey. Published March 2011. Accessed March 20, 2010
- Dillman, D. A. 2000. Mail and Internet surveys: The tailored design method. 2nd ed. New York, NY: John Wiley & Sons.
- EmcArts, 2008. Biennial Exhibition Device Survey Questions. Accessed on April 3, 2011
- Falk, J., Moussouri, T. and Coulson, D., 1998. `The Effect of Visitors’ Agendas on Museum Learning’, Curator, vol. 41, no 2, 106-120.
- Haley Goldman, K. 2007. Cell Phones and Exhibitions 2.O: Moving beyond the Pilot Stage. In Museums and the Web 2007: Proceedings, ed. J. Trant and D. Bearman. Toronto: Archives & Museum Informatics, published March 1, 2007.
- Kelly, L., 2007. The interrelationships between adult museum visitors’ learning identities and their museum experiences. Published June 2007. Accessed Feburary 2010
- Proctor, N., 2010. The Museum Is Mobile: Cross-platform Content Design for Audiences on the Go. In Museums and the Web 2010: Proceedings, ed. J. Trant and D. Bearman. Toronto: Archives & Museum Informatics. Published March 31, 2010.
- Tallon, L., 2011. Museums and Mobiles Survey 2011: 738 Voices on the Objectives, Challenges, and Future of Mobile interpretation. Published January 2011. Accessed March 20, 2010
- Wei, R. & V.H. Lo. (2006). “Staying connected while on the move: Cell phone use and social connectedness”. New Media and Society. Vol. 8(1):53-72.