Assessing Museum Impact: A Pilot for Formal Research

February 27, 2020

by John W. Jacobsen, Laura Roberts, David Ellis, George Hein, and Lynn Bau

With gratitude for the support of the New England Museum Association (NEMA)

Introduction

One of the museum field’s most challenging needs is to find ways to measure, articulate, and increase a museum’s desired impacts. In response, the Assessing Museum Impact (AMI) pilot research project explored the theory that a system for using carefully selected data could help museums improve their impact and performance. With the support and partnership of the New England Museum Association (NEMA), the six participating New England museums found that using indicator data in practice did help them increase their impact. In this voluntary, small-scale pilot research project, the authors and project advisers found that the theory is promising, and that wider, deeper, and more formal research is justified. These are encouraging findings, and the authors propose that other museum associations join NEMA’s lead in supporting and disseminating these findings and encouraging further research.

“Assessing Museum Impact: From Theory to Practice – A Summary Report” summarizes the need, relevant prior work, research process, findings, conclusions, and recommendations. It details the data-informed changes that the participating museums made to their programs and operations and the methods they found most effective in the collection and application of data. This blog post focuses on the need for more formal and detailed research, building on the AMI concept test.

While there have been previous efforts to establish frameworks for museum evaluation (Anderson 1997; Baldwin 2011; Friedman 2007; Sheppard & Falk 2006), the data infrastructure to support such frameworks has not been in place until recently. Achieving this goal is now possible because of increased transparency regarding museum operating data (Anderson 2004; Stein 2009); the growing body of evaluation findings and evidence posted on http://www.visitorstudies.org/ and http://www.informalscience.org, and new national data compilations of museum operating data, such as the online database for the Association of Children’s Museums (ACM), SMU DataArts, Guidestar’s collection of IRS 990 data forms, and other online museum data. The Collaboration for Ongoing Visitor Experience Studies (COVES) is a museum-based series of specific questions that provides templates to help museums collect meaningful data about their visitors to inform decisions and relevant points of comparison with peers.

The participating museums included Gore Place (Waltham, MA); Children’s Museum of New Hampshire; Paul Revere House (Paul Revere Memorial Association, MA); Rough Point Museum (Newport Restoration Foundation, RI); Seacoast Science Center (NH), and the USS Constitution Museum (MA).

As Stephen E. Weil observed, the value of a museum lies in its contributions: What impacts did the museum achieve this year? For whom? At what cost/value to whom? How has it changed someone’s life? How has it changed its community? How does one measure the value/impact of a museum as a whole?

These conceptual foundations have implications for today’s museum leaders:

To be useful, museums need to understand the needs of their communities and be structured to address those needs.
To have an impact and create public value, museums should understand that their vast resources (means) need to be deployed to achieve their purposes (ends) and be evaluated on how effectively and efficiently they do so.
To be successful in a competitive marketplace, museums must offer experiences and services that their audiences and supporters find valuable and relevant.
To make the world better and more democratic, museums need to aspire to larger, more ambitious goals.

Synthesized, these concepts underlie museum economic theory: The community funds the museum to use its resources to provide effective services back to the community. The museum provides these services efficiently and, instead of privatizing its net revenues, contributes to community development and social good. People and organizations pay money, spend time, and make efforts to engage with a museum in return for the impacts and benefits they receive. The cumulative time, effort, and resources that a museum’s audiences and supporters provide in exchange for the benefits they get from the museum are indicators of its value to its stakeholders.

The needs, theories, and approaches were developed in John Jacobsen’s book, Measuring Museum Impact and Performance (Jacobsen 2016), which the authors of the AMI research project discussed in 2017. The authors, including Jacobsen, decided that this approach could be explored in practice with a small network of nearby museums. Out of this collegial initiative, the AMI pilot research study was organized and proposed to NEMA later that year. NEMA accepted the proposal and supported the project logistically.

The theory of action starts with community needs informing the museum’s intentional purposes and uses the museum’s resources to provide desired impacts and benefits to address those needs. The “PIID Sequence” (Purposes, Impacts, Indicators, Data fields) starts with museum leadership articulating one or more of its intentional purposes, then stating what changes or impacts they aspire to achieve for each purpose, what real-world observations might indicate that impact was happening, and what data fields might measure or document changes in that indicator.

PIID Sequence

What are our museum’s main Purposes?
What Impacts (e.g., outcomes, benefits, changes) do we wish for each purpose?
What might Indicate that an impact or change is happening?
What Data fields can measure or document that indication of impact?
How do we periodically evaluate the validity of our findings?

All participants developed their selections internally and submitted their draft choices on “PIID Sheets,” following an outline by the advisers. The advisers coached the museums through a few drafts until all the museums had sharable and largely consistent lists of their AMI test purposes, impacts, indicators, and data fields.

By the fourth workshop, all the museums had completed the following tasks, as reported on the PIID Sheets and at the previous workshops:

Selected two of their institutional purposes as the basis for the AMI project.
Listed some of the impacts and benefits that they hoped would be among the outcomes of achieving those purposes.
Figured out how they might observe whether those outcomes are happening and what indications they might measure (generating a list of indicators and data fields).

The AMI advisers coached the participants through several rounds of their PIID Sheets. The AMI process is technical and uses definitions that take some time to absorb, and the coaches wanted to get all collaborators to conform to the same system and terms. In addition, the museums often started with big ambitions, but when the coaches talked through the logistics of collecting all the desired data, the participants realized they needed to reduce the number of indicators and data fields in order to favor readily available data. A future funded study might enable the collection of more challenging data. For instance, while there was some initial interest in measuring changes in audience diversity, the participants did not have easy access to diversity metrics, databases, or collection methods.

Challenges Ahead for the Museum Field

Due to the increasing number of nonprofits seeking charitable donations, the increased interest in impact investing, and ever-increasing costs for museum operations, prospective donors are likely to want, even demand, more evidence of the impact that a particular museum has on its community. This will require more and better data.

To meet this need, museums will have to use processes that are based on verifiable approaches that specifically address their needs. The AMI project provides a way to approach this effort. In future, museums will need to be more intentional and informed about the many ways there are to collect and use meaningful data—to find the data that make the most sense for their institutions based on their goals and their communities. Working with colleague institutions may materially assist such efforts.

Conclusion

Museums need to use data to measure and improve their impact and performance. Considerable work has been done on assessing the value of a museum’s impacts and benefits, which has led to a theory of how to measure changes in a museum’s value. The purpose of the two-year AMI pilot research project was to explore whether the use of data in practice can help museums improve their impact (effectiveness) and performance (efficiency). Led voluntarily by five senior museum professionals (the authors) and coordinated by the NEMA, the project used a specific process to assist six mid-sized New England museums to use data strategically by using the PIID Sequence. The participants reported positive impacts on their management culture and actionable enhancements to their museums. All plan to continue using data strategically. The resulting findings may be useful in building capacity for incorporating data in museum administration, planning, advocacy, and fundraising. More work is needed to build on these initial steps, and the authors propose that other museum associations join NEMA in supporting and disseminating these findings and continuing this promising line of research.

ABOUT THE AUTHORS

John W. Jacobsen, President (ret), White Oak Associates, Inc.
www.WhiteOakAssoc.com; jjacobsen@whiteoakassoc.com

Laura Roberts, Principal, Roberts Consulting
www.lauraroberts.com; laura@lauraroberts.com

David W. Ellis, Consultant, President Emeritus, Museum of Science, Boston
dellis@mos.org

George Hein, Professor Emeritus, Lesley University
ghein@lesley.edu

Lynn Baum, Principal, Turtle Peak Consulting
lbaum@turtlepeakconsulting.com

WORKS CITED

Anderson, David. 1997. A Common Wealth: Museums and Learning in the United Kingdom. London: Department of National Heritage.

Anderson, Maxwell L. 2004. Metrics of Success in Art Museums. Los Angeles: The Getty Leadership Institute, J. Paul Getty Trust. Accessed September 12, 2019. http://docplayer.net/15254796-Metrics-of-success-in-art-museums.html

Baldwin, Joan H. 2011. The Challenge of “Value”: Engaging Communities in Why Museums Exist. A Museum Association of New York | Museumwise White Paper. Accessed November 4, 2014. http://manyonline.org/sites/default/files/pages/%3Cem%3EEdit%20Basic%20page%3C/em%3E%20Museums%20and%20the%20Economy/2011-The-Challenge-of-Value.pdf

Friedman, A. J. 2007. The Great Sustainability Challenge: How Visitor Studies Can Save Cultural Institutions in the 21st Century. Visitor Studies 10 (1): 3-12.

Jacobsen, John W. 2016. Measuring Museum Impact and Performance: Theory and Practice. Lanham, MD: Rowman and Littlefield.

Sheppard, Beverly, and John H. Falk. 2006. Thriving in the Knowledge Age: New Business Models for Museums and Other Cultural Institutions. Lanham, MD: AltaMira Press.

Stein, Rob. 2009. “Transparency and Museums.” Accessed October 21, 2014. Indianapolis Museum of Art. http://www.imamuseum.org/blog/2009/11/03/transparency-and-museums