Updates from the Field: Research Agendas for Informal STEM Learning

Top image: Kevin Crowley, co-PI of CAISE, discusses research agenda development at a session at the 2014 AISL PI Meeting.

The theme of this summer’s Visitor Studies Association (VSA) conference was Building Capacity for Evaluation: Individuals, Institutions, the Field. One session addressed the role that research agendas are playing in building capacity among researchers and designers of informal learning experiences and settings to gather useful evidence. The session was divided into three parts: brief presentations of three “cases” of field-driven research agenda development projects, small group discussions on the questions and issues raised by the cases, and brief report-outs, with whole-group reactions. The case studies presented were from  the experiences of the Association of Zoos and Aquariums (AZA) Committee on Education and Conservation, the Center for Advancement of Informal Science Education (CAISE), and the Association of Children’s Museums (ACM).

Research Agenda Processes

The presentations revealed three different impetuses and rationales for developing a research agenda and three processes that included some interesting parallels.

The Association of Zoos and Aquariums (AZA)

The AZA process began in 1997 when a self-selected group of zoos and aquaria researchers and practitioners met to try to develop an agenda over three days. The group discovered that the learning research in the zoos and aquaria sector was still too nascent and "silo-ed" to build consensus toward a common research areas and questions. Ten years later a subgroup of the original came together to draft a template of seven questions with sub-questions which they presented to the Conservation Education Committee (CEC), an AZA advisory group, for review and edits. The questions addressed a range of topics, from identifying the role that zoos and aquaria play in influencing social action to comparing how these types of institutions compare with others in the informal science education field in impacting lifelong learning. The CEC then spent three years living with the draft agenda, exploring how it might best be used, and further fleshing it out.

Joe Heimlich updates attendees at the 2014 AISL PI Meeting on the status of AZA's research agenda development process.

The CEC has since been developing a matrix of the questions and sub-questions that tracks which zoos and aquaria are doing evaluation or research in each of the question categories. Next steps include taking each question and tracking how it is being addressed across a diversity of institutions. The CEC plans to continue iterating and refining the research questions, as well as the matrix, which will serve a visual scan to identify gaps in data and needs for further studies. Both will also help with eventual knowledge syntheses that will be shared in a future Perspectives blog post.

The Center for Advancement of Informal Science Education (CAISE)

The Center for Advancement of Informal Science Education (CAISE) Practice-and-Research initiative was launched in response in the deep interest shown by participants in the 2012 National Science Foundation (NSF) Informal Science Education (ISE) program Principal Investigator Meeting, in bridging the worlds of designing experiences and settings with designing research that will build knowledge for improved practice.

Later that year CAISE convened a series of telephone conference calls with groups of informal science education professionals who had led or contributed to research agenda development projects in the past. These calls identified issues and challenges that became seeding questions for a weeklong online forum that followed, in order to go deeper into these topics with a wider cross section of the field.

Through iteration with NSF program officers, CAISE then used recurrent themes from the forum to draft a scope for an in-person convening to discuss parallels and mismatches between practice and research writ large and to begin exploring the feasibility of a research agenda for informal STEM learning. A heterogeneous group of ISE stakeholders at the convening helped CAISE refine its initiative to be broad and inclusive of the variety of possible ways to strengthen dialogue and connections between research and practice, and to be thoughtful about trying to develop an agenda for a wide field that includes museums and science centers, zoos and aquaria, media outlets, citizen science programs at universities, after and out of school programs, and cyberlearning and gaming platforms.

Research agendas are being tracked and shared with the field through the CAISE Research Agenda landing page on InformalScience.org.

Ultimately CAISE convened a smaller follow-up working group with participants from the initial meeting and others who suspended disbelief in the possibility of completing the task, and through a series of brainstorming and sorting activities developed what became a “roadmap” towards a research and development agenda for the field. The resulting roadmap has eight categories of potential research that CAISE has been vetting publicly and that have informed the work of others including the National Association for Research on Science Teaching (NARST) informal learning conference strand and research agenda development processes that have been initiated since then. CAISE’s combined efforts led the recent launching of a research agenda landing page that maps the progress of these types of efforts and provides easy access to background resources from the CAISE Informal Commons repository.

The Association of Children's Museums

The Association of Children’s Museums’ Learning Value of Children’s Museums: Building a Field-Wide Research Agenda project grew from the Association’s board of directors’ sense that while there is a wide variety of learning taking place in children’s museums the research and evidence gathering is not keeping pace. Hence, the board engaged researchers at the University of Washington to help identify, articulate, and prioritize the most pressing evidence needed among field professionals to guide the research. The researchers began by conducting an inventory of extant studies that became a landscape review of research already done and some currently being conducted in the children’s museum field.

Central to the process was a national symposium held in September of 2013 that brought together museum CEOs, policy makers, educators, designers, and researchers, to brainstorm, discuss, and refine a list of categories of rich, useful areas of research. As was the case with AZA and CAISE, the researchers and Association staff quickly learned that the idea of zeroing in on a few key research questions over a couple of days was unrealistic. Instead, nine categories of themes surfaced at that gathering, which the participants then weighed in on as to which areas were most in need of data and evidence, and which might be immediately addressable. Those categories were then shared for further input and feedback during three different webinars targeted for practitioners, researchers, and children’s museum CEOs, respectively. The emergent agenda is currently organized around three big “buckets”: audience, institution, and learning landscape, with seven sub-categories of areas for potential study.

The Institute for Museum and Library Services (IMLS) has awarded funding for the implementation of the agenda, which will include developing a network of children's museums with the capacity for conducting research, helping them identify areas of mutual research interest that respond to the research agenda, conducting three to five research projects across this museum network, and cultivating the network to initiate future research activities. Some uses of the agenda will be  to strengthen the infrastructure of the professional association and inform strands of the annual ACM conference.

Concerns, Observations, and Ideas for Capacity Building

Following the presentations of these case studies session participants were invited to join one of the three presenters to discuss these questions through the lens of the research agenda development project they were most interested in:

• What are concerns, issues, and pushback that the cases presented elicit?
• How do the cases inform thinking about field-wide capacity building?
• Assuming that the research agendas being developed would contribute to knowledge building, what kinds of infrastructure and professional development would they need to be implemented?

The report-outs from the discussions revealed these concerns:

• How can those not involved in these processes be sure that they were conducted rigorously? There needs to be transparency.
• The categories of research areas and questions developed by a mixed group of professionals at a “30,000 ft. level” do not always represent the priorities of the institution, program, or project. Those creating research agendas and/or frameworks for them need to continue to be flexible and open to iteration as they receive input and feedback from the wider community.
• A research agenda reflects a moment in time and the values of the field it is developed for. Values and current trends inform the “why” a field chooses particular areas for study. Hence these need to be identified and made explicit before developing the agenda.

With regard to capacity building, these points were raised:

• A research agenda itself is not for the “field” (of designers and practitioners). The agenda is for the researchers and evaluators. The gathering of the data and meaning making of it against the questions is for the field.
• There is a long history of the informal learning field grappling with what and how to research the development, design and implementation of experiences and settings. The emergence of these research agenda projects is a sign of the field’s maturity.

In terms of what these or any research agendas will need “on the ground” to be implemented, these ideas surfaced:

• Wide buy-in from the field can only be accomplished through wide dissemination of agenda drafts.
• A research agenda needs a strategy for implementation. Systems research should be part of the implementation plan.

The three research agenda development case study processes presented were all initiated for different purposes by different stakeholder groups, yet there are interesting parallels in how they are being conducted. All three convened mixed stakeholder groups of professionals to brainstorm and surface questions at different grain sizes that were then sorted, refined, and organized into what they called categories, areas, or “buckets.”  Each project built in multiple strategies and opportunities for input from a broader group than those directly involved in the initial development. Participants in all three processes also quickly learned that coming to consensus on a few areas of research is complex and cannot be accomplished in a few days.

There are also interesting parallels in their outputs so far. The first phase of the AZA and ACM processes resulted in seven categories of questions for potential study and the CAISE roadmap process produced eight. These three cases were shared again at the 2014 NSF Advancing Informal STEM Learning (AISL) program Principal Investigator meeting in August, where other processes such as those of the Giant Screen Cinema Association and the 21st Century Learning in Natural History Settings project a comparable set of categories and questions. Is a “phone number”-size list of areas/questions an optimal, manageable one for our growing field to explore at this point in time?

Overall the VSA and AISL PI Meeting sessions revealed that exploring the feasibility and development of a research agenda demands more time, reflection, critical thinking, and negotiation than one might expect. What also may be emerging is that the processes that these projects are undertaking to identify useful research questions are resulting in innovations that can be applied to other strategies for connecting research and practice. The discussion will continue at a session at the Association of Science-Technology Centers conference in Raleigh, North Carolina on Monday, October 20th. CAISE will continue to track, characterize and connect research agenda development processes here.