The starting point of the Evaluating Evaluation project was our impression that despite the substantial resources that are spent on the summative evaluation of museums and galleries the research has little impact and largely remains ineffectual.
With support from the Wellcome Trust and the Heritage Lottery Fund, we set out to see why this seems to be the case and to explore whether there are ways in which the impact of summative evaluations on knowledge and practice might be enhanced. To this end, we reviewed a substantial number of reports, undertook a range of interviews and held two
DATE:
TEAM MEMBERS:
Maurice DaviesChristian Heath
resourceevaluationProfessional Development, Conferences, and Networks
Researching the Value of Educator Actions for Learning (REVEAL) is an NSF-funded research project, led by the Oregon Museum of Science and Industry (OMSI) in collaboration with TERC and Oregon State University (OSU), which studied the impact of science center staff facilitation strategies at math exhibits. Building on and using exhibits from Design Zone—an interactive math exhibition which engaged visitors in exploring algebraic relationships and functions—the study was designed to add to the scant existing literature about the impact of staff facilitation on visitors. The study also aimed to
Co-led by the University of Washington and Science Gallery Dublin, this project aims to drive and transform the next generation of broadening participation efforts targeting teen-aged youth from communities historically underrepresented in STEM fields. This project investigates how out-of-school time (OST) programs that integrate epistemic practices of the arts, sciences, computer science, and other disciplines, in the context of consequential activities (such as creating radio segments, designing museum exhibitions, or building online games), can more broadly appeal to and engage youth who do not already identify as STEM learners. STEM-related skills and capacities (such as computational thinking, design, data visualizations, and digital storytelling) are key to productive and creative participation in many future civic and workplace activities, and are driving the 30 fastest-growing occupations in the US. But many new jobs will entail a hybrid blend of skills, such as programming and design skills that many students who have disengaged with academic STEM pathways may already have and would be eager to develop further. There is not currently a strong foundation of research-based evidence to guide the design, implementation, and evaluation transdisciplinary programs - in which STEM skills are embedded as tools for meaningful participation - or how such approaches relate to long-term outcomes. Hypothesizing that OST programs which effectively engage youth during their high-leverage teenage years can significantly impact youths' longer-term STEM learning trajectories, this project will involve: 1) Five 3-year studies documenting learning in different technology-rich contexts: Making Afterschool, Media Production, Museum Exhibition Design, Digital Arts Programs, and Pop-Up/Street Science Programs; 2) A 4-year longitudinal study, involving 100 youth from the above programs; 3) The creation of a number of practical measurement tools that can be used to monitor how programs are leveraging the intersections of the arts and sciences to support student engagement and learning; and 4) A Professional Development program conducted at informal science education conferences in the EU and US to engage the informal STEM field with emerging findings. This project is funded through Science Learning+, which is an international partnership between the National Science Foundation (NSF) and the Wellcome Trust with the UK Economic and Social Research Council. The goal of this joint funding effort is to make transformational steps toward improving the knowledge base and practices of informal STEM experiences to better understand, strengthen, and coordinate STEM engagement and learning. Within NSF, Science Learning+ is part of the Advancing Informal STEM Learning (AISL) program that seeks to enhance learning in informal environments.
Transdisciplinary, equity-oriented OST programs can provide supportive social contexts in which STEM concepts and practices are taken up as the means for meaningful participation in valued activities, building students' STEM skills in ways that can propel their future academic, career, and lifelong learning choices. This project will build the knowledge base about these emerging 21st century transdisciplinary approaches to broadening participation investigating: 1) The epistemic intersections across a range of disciplines (art, science, computation, design) that operate to broaden appeal and meaningful participation for underrepresented youth; 2) How transdisciplinary activities undertaken in the context of consequential learning (e.g., producing a radio segment, designing an exhibition for the general public) can illuminate the relevance of STEM to young people's lives, concerns, and futures; and 3) How participation in such programs can propel students' longer-term life choices and STEM learning trajectories. The project is a collaboration of the University of Washington, Science Gallery Dublin, Indiana University, Youth Radio in Oakland California, Guerilla Science in New York and London, and the London School of Economics.
Science researchers and practitioners are often challenged by how best to assess the effectiveness of science activities on young children whose language skills are still emerging. Yet, research has demonstrated the critical importance of early learning on individual potential. Building on evidence that movement is tightly intertwined with thinking, this project will investigate how thought and movement link as embodied learning to accelerate science understanding. Research will be conducted in the United States (US) and the United Kingdom (UK) with the aim to gather evidence for embodied interactions during science learning and articulate design principles about how museum exhibits can most effectively encourage cognitive and physical engagement with science. Such guidelines are largely absent in the field of informal STEM learning, and so this project seeks transformational change in how learning is understood and recognizes that changes in knowledge can be developed and revealed through body-based movements as well as verbally. Such a view is critically important given that many early learners communicate understanding through nonverbal channels before verbal. Research will be conducted with a diverse population of children and will explore the application of embodied learning to communities that are underrepresented in STEM. This project is funded through Science Learning+, which is an international partnership between the National Science Foundation (NSF) and the Wellcome Trust with the UK Economic and Social Research Council. The goal of this joint funding effort is to make transformational steps toward improving the knowledge base and practices of informal STEM experiences. Within NSF, Science Learning+ is part of the Advancing Informal STEM Learning (AISL) program that seeks to enhance learning in informal environments and to broaden access to and engagement in STEM learning experiences. During a 3-year period, researcher-practitioner teams across six museum sites will collaboratively investigate the links between movement and learning outcomes at selected science exhibits designed for young learners. Research activities will involve iteration and refinement of new instruments and protocols, through analysis of observed and automated capture of interaction data, and synthesis and interpretation of data. A design-based research methodology will be applied to address three key questions: 1) What elements of sensory and action experiences are key to informing the design of exhibits that aim to exploit embodied interactions for learning; 2) What is the role of bodily enactment /gestures in assessing children's understanding of science concepts; and 3) What cultural differences in kinds of embodied engagement emerge across diverse museum settings? Video and audio data of 400 children's exhibit interactions will be collected. Pre/post semi-structured interviews will be conducted with a subset of these participants and will focus on children's understanding of relevant science concepts as well as personal reflections on their physical and emotional experience engaging with the exhibit. This project would raise awareness of embodied approaches to learning as well as build stronger collaborations between informal STEM educators and cognitive researchers. Utilization of informal and formal dissemination networks will support wide diffusion of project outcomes. This is critically important given strong evidence pointing to the impact of preschool education in underserved populations, and ongoing national efforts by the US and UK to improve the quality of STEM learning in preschool contexts.
Project partners supported by NSF funding include The Phillip and Patricia Frost Museum of Science, University of Illinois Urbana Champaign, The Children's Museum of Indianapolis, andSciencenter (Ithaca).
Partners supported by the Wellcome Trust include University of Edinburgh, University College London, Glasgow Science Centre, Science Museum London, and Learning through Landscapes.
One way to encourage youth to pursue training in the STEM fields and enter the STEM workforce is to foster interest and engagement in STEM during adolescence. Informal STEM Learning Sites (ISLS) provide opportunities for building interest and engagement in the STEM fields through a multitude of avenues, including the programming that they provide for youth, particularly teens. Frequently, ISLS provide opportunities to participate in volunteer programs, internships or work, which allow teens both to learn relevant STEM knowledge as well as to share that knowledge with others through opportunities to serve as youth educators. While youth educator programs provide rich contexts for teens to engage as both learners and teachers in these informal STEM environments, research to date has not yet identified the relationship between serving as youth educators and STEM engagement. Thus, the goal of this project is to document the impact of youth educators on visitor learning in ISLS and to identify best practices for implementing youth educator programs. The project studies STEM interests and engagement in the youth participants and the visitors that they interact with at six different ISLS in the US and UK. This project is funded through Science Learning+, which is an international partnership between the National Science Foundation (NSF) and the Wellcome Trust with the UK Economic and Social Research Council. The goal of this joint funding effort is to make transformational steps toward improving the knowledge base and practices of informal STEM experiences. Within NSF, Science Learning+ is part of the Advancing Informal STEM Learning (AISL) program that seeks to enhance learning in informal environments and to broaden access to and engagement in STEM learning experiences.
This project examines youth educator experiences related to STEM identity, educational aspirations, and motivation. The project also identifies outcomes that the youth educators have on visitors to ISLS in terms of knowledge, interest, and engagement in STEM. The specific aims are: 1) Outcomes for Teens - To measure the longitudinal impact of participation in an extended youth educator experience in an ISLS; 2) Outcomes for Visitors - To compare visitor engagement with and learning from exhibits in ISLS when they interact with a youth educator, relative to outcomes of interacting with an adult educator or no educator; and 3) Outcomes Across Demographics and STEM Sites - To examine differences in visitor engagement based on participant characteristics such as socio-economic status (SES), age, gender, and ethnicity and to compare outcomes of youth educator experiences across different types of ISLS. This research, which draws on expectancy value theory and social cognitive theory, will follow youth participants longitudinally over the course of 5 years and use latent variable analyses to understand the impact on the youth educators as well as the visitors with whom they interact. Importantly, the results of this research will be used to develop best practices for implementing youth educator programs in ISLS and the results will be disseminated to both academic and practice-based communities.
This project has clear and measurable broader impacts in a variety of ways. First, the project provides guidance to improve programming for youth in ISLS, including both the sites involved directly in the research and to the larger community of ISLS through evaluation, development, and dissemination of best practices. Additionally, this project provides rigorous, research-based evidence to identify and describe the outcomes of youth educator programs. This study directly benefits the participants of the research, both the visiting public and the youth educators, through opportunities to engage with science. The findings speak to issues of access and inclusivity in ISLS, providing insight into how to design environments that are welcoming and accessible for diverse groups of learners. Finally, this project provides evidence for best practices for ISLS in developing programs for youth that will lead to interest in and pursuit of STEM careers by members of underrepresented groups.
DATE:
-
TEAM MEMBERS:
Adam Hartstone-RoseMatthew IrvinKelly Lynn MulveyElizabeth ClemensLauren ShenfeldAdam RutlandMark WinterbottomFrances BalkwillPeter McOwanKatie ChambersStephanie TylerLisa Stallard
Front-line staff are an integral part of the visitor experience at museums and science centers across the country, facilitating activities and programs, leading classes, and more. But do these staff make a difference for visitor learning? And what are the most effective facilitation strategies and approaches? In 2013, the Oregon Museum of Science and Industry (OMSI) received funding from the National Science Foundation for a three-year study, Researching the Value of Educator Actions for Learning (REVEAL), to begin to address these questions. Building on the Design Zone exhibition, REVEAL
This document is a list of culturally responsive research resources developed by the Researching the Value of Educator Actions for Learning (REVEAL) project.
This document outlines the culturally responsive research (CRR) framework developed to guide planning, data collection and analysis, and dissemination throughout the Researching the Value of Educator Actions for Learning (REVEAL) project. REVEAL was a National Science Foundation-funded initiative that studied the impact of staff facilitation by museum educators on family learning at interactive exhibits in a science center. Led by the Oregon Museum of Science and Industry, the project was conducted in partnership with Oregon State University and TERC. The project included a design-based
The Oregon Museum of Science and Industry (OMSI), located in Portland, is a hands-on science museum. In 2013, OMSI received funding from the National Science Foundation for the project Researching the Value of Educator Actions for Learning (REVEAL) to study how museum educators can better help families learn math while interacting with museum exhibits. Through REVEAL, OMSI was able to partner with Adelante Mujeres, a non-profit community organization located in Forest Grove that educates and empowers Latina women and their families. Here we share some of the lessons learned from the
This summative evaluation report details the Broad Implementation of the Living Laboratory model--an initiative to promote partnership between museums and cognitive science researchers in order to promote professional learning and involve the public in scientific research. The evaluation investigated the extent of the dissemination effort’s depth, spread, sustainability, and shift in ownership, based on Coburn’s criteria for scale-up (2003). Evaluators collected data from surveys, interviews, focus groups, document review, and observations. Findings about depth suggest that adopters fully
Informal learning, and by extension, museums, are inherently emotional experiences, evoking feelings of awe, excitement, and curiosity. Oftentimes, museum professionals have prioritized traditionally positive emotions such as excitement and interest as being the most desirable and useful in supporting museum learning. However, prior research into naturally occurring emotions at museums found that some visitors who experienced negative emotions, such as confusion or frustration, at exhibits also reported deeper engagement and overall feelings of satisfaction (Rappolt-Schlichtmann et al., 2017). Based on these findings and similar results from formal education (D’Mello et al., 2014), this project team has worked to develop and refine a framework of strategies for creating exhibits that invoke and support visitors through the complex emotional state called productive struggle (PS) which is defined as a three-part emotional arc characterized by: 1) disequilibrium (experienced emotionally as emotions like confusion, frustration, surprise, or unease) that arises from encountering a challenging task, phenomenon, or idea, 2) persistence through the task which is supported by exhibit design scaffolds, and 3) an emotionally productive resolution tied to the source of disequilibrium or an overall sense of effortful achievement. In deliberately attending to and supporting a range of negative and positive emotions in museums, visitors can gain access to a wider variety of complex emotional experiences, including those critical to STEM learning, and have potential to broaden participation in STEM by supporting learners' diverse emotional needs and preferences.
This multidisciplinary project team consisting of researchers and exhibit professionals utilized a design-based research (DBR) process to develop, test, and refine a definition of museum-based productive struggle, and create a framework of design strategies that support PS. Three physical exhibits and a virtual exhibit were created during this multi-year project. Additionally, a summative research study was conducted with 105 youth ages 10-17 to explore: 1) whether visitors experienced the expected emotional arc of PS; 2) how exhibit design strategies supported PS; and 3) the extent of visitors’ learning and engagement at these PS exhibits.
In late 2012, Providence Children’s Museum began a major three-year research project in collaboration with The Causality and Mind Lab at Brown University, funded by a grant from the National Science Foundation (1223777). Researchers at Brown examined how children develop scientific thinking skills and understand their own learning processes. The Museum examined what caregivers and informal educators understand about learning through play in its exhibits and how to support children’s metacognition – the ability to notice and reflect on their own thinking – and adults’ awareness and appreciation of kids’ thinking and learning through play. Drawing from fields like developmental psychology, informal education and museum visitor studies, the Museum’s exhibits team looked for indicators of children’s learning through play and interviewed parents and caregivers about what they noticed children doing in the exhibits, asking them to reflect on their children’s thinking. Based on the findings, the research team developed and tested new tools and activities to encourage caregivers to notice and appreciate the learning that takes place through play.