Inverness Research and Oregon State University, with support and input from CAISE, conducted an evaluation of the 2021 NSF AISL Awardee Meeting which was held virtually October 19-21, 2021. The evaluation effort included observing the meeting, participating in debriefing the meeting with CAISE co-PIs, the CAISE equity audit committee, and NSF Program Officers; developing and administering a post-event survey;1 and analyzing data collected through both the survey2 and Pathable, the virtual platform.
The meeting specifically focused on inviting and including community partners, and on
This project is designed to support collaboration between informal STEM learning (ISL) researchers, designers, and educators with sound researchers and acoustic ecologists to jointly explore the role of auditory experiences—soundscapes—on learning. In informal STEM learning spaces, where conversation advances STEM learning and is a vital part of the experience of exploring STEM phenomena with family and friends, attention to the impacts of soundscapes can have an important bearing on learning. Understanding how soundscapes may facilitate, spark, distract from, or even overwhelm thinking and conversation will provide ISL educators and designers evidence to inform their practice. The project is structured to reflect the complexity of ISL audiences and experiences; thus, partners include the North Park Village Nature Center located in in a diverse immigrant neighborhood in Chicago; Wild Indigo, a Great Lakes Audubon program primarily serving African American visitors in Midwest cities; an after-school/summer camp provider, STEAMing Ahead New Mexico, serving families in the rural southwest corner of New Mexico, and four sites in Ohio, MetroParks, Columbus Zoo and Aquarium, Franklin Park Conservatory and Botanical Gardens, and the Center of Science and Industry.
Investigators will conduct large-scale exploratory research to answer an understudied research question: How do environmental sounds impact STEM learning in informal learning spaces? Researchers and practitioners will characterize and describe the soundscapes throughout the different outdoor and indoor exhibit/learning spaces. Researchers will observe 800 visitors, tracking attraction, attention, dwell time, and shared learning. In addition to observations, researchers will join another 150 visitors for think-aloud interviews, where researchers will walk alongside visitors and capture pertinent notes while visitors describe their experience in real time. Correlational and cluster analyses using machine learning algorithms will be used to identify patterns across different sounds, soundscapes, responses, and reflections of research participants. In particular, the analyses will identify characteristics of sounds that correlate with increased attention and shared learning. Throughout the project, a team of evaluators will monitor progress and support continuous improvement, including guidance for developing culturally responsive research metrics co-defined with project partners. Evaluators will also document the extent to which the project impacts capacity building, and influences planning and design considerations for project partners. This exploratory study is the initial in a larger research agenda, laying the groundwork for future experimental study designs that test causal claims about the relationships between specific soundscapes and visitor learning. Results of this study will be disseminated widely to informal learning researchers and practitioners through workshops, presentations, journal articles, facilitated conversations, and a short film that aligns with the focus and findings of the research.
DATE:
-
TEAM MEMBERS:
Martha MersonJustin MeyerDaniel Shanahan
resourceprojectProfessional Development, Conferences, and Networks
Each year, millions of Americans visit science centers and museums, children’s museums, zoos, aquariums, nature centers, planetariums, and similar institutions. Recognized as trusted and popular places for educational and leisure experiences, these institutions are uniquely capable of engaging people across a spectrum of beliefs, experiences, and identities on topics related to science, technology, engineering, and mathematics (STEM), as well as addressing pressing societal problems related to science, technology, and innovation. However, the potential impacts of these institutions are largely dependent on the skills, knowledge, and abilities of the professionals working at them. Following the onset of the COVID-19 pandemic and the need to shift the nature of work, the informal STEM learning workforce was dramatically impacted. The recent period of disruption is a time for innovation. The field is well-positioned to promote new models of professional learning and development that are grounded in the values and practices of informal learning. This project will benefit local communities across the United States and society at large by advancing the capacity of science-engagement professionals to respond to societal needs, concerns, and interests more effectively through their institutions’ exhibitions, education and learning programs, and various forms of public engagement (e.g., community outreach events, supports for teachers/educators and schools). Led by the Association of Science and Technology Centers in collaboration with the Center of Science and Industry’s Center for Research and Evaluation and Oregon State University’s STEM Research Center, this work will build on the Informal STEM Learning (ISL) Professional Competency Framework developed and validated with prior National Science Foundation funding. Competency frameworks are increasingly used across many sectors to identify the suite of skills, knowledge, and capabilities necessary to be successful in a particular area of work. Given dramatic changes to the ISL workplace in the last two years, there is an even stronger potential for the Framework, particularly if newly developed supports can link the Framework to the current, emerging, and continuing needs of the workforce.
Guided by a systematic process for designing training and instructional materials, this project will first capture changes in the ways diverse informal STEM learning professionals describe, pursue, and achieve competencies to produce a revised Framework. Next, the project will collaboratively develop companion resources with diverse professionals through a series of participatory design workshops, using a sequential and iterative approach. The resources are expected to include indicators of professional competencies, self-assessments, training tools, and other types of resources that have the potential to significantly advance the professional learning (as undertaken by individuals and institutions) and the professional development capacity (as provided by institutions, associations, and other organizations) of the informal STEM learning field. Next, the project will share final products–including a refreshed website that hosts the updated Framework and newly-developed suite of tools–with professional association and network partners who can disseminate directly to the institutions and professionals who are developing and practicing these competencies. The project will gather evidence from a small number of early adopters, providing data on specific use-case scenarios. Finally, the project will document the potential impact of the Framework on the field by measuring how the Framework is perceived by informal STEM learning professionals as usable, useful, and beginning to be “in-use." Over time—and with increased use—the Framework and its companion resources hold the promise of contributing to the opening of the field to professionals with identities currently underrepresented through more transparent expectation and clearer growth pathways.
Diversity, equity, access, inclusion, and belonging-related change is often difficult to achieve in organizations. In the context of Informal STEM Learning (ISL), this results in inequitable opportunities for both ISL professionals and learners to engage in STEM environments and experiences. For people to thrive in these settings, creative and innovative approaches that address historical and current realities of intersectional marginalization and inequitable norms within informal STEM institutions are necessary to disrupt conventions, institutional barriers, and patterns of inequities. TERC and the Detroit Zoological Society will develop and implement an ISL Equity Resource Center (the Center). The Center will advance equity within the ISL field by cultivating a multi-sector, diverse learning community that designs and conducts evidence-based research and practice, including but not limited to equity-focused leadership, decision-making, theory, methods, project topic selection and design, and budget management.
The Center will cultivate lasting change in the ISL and broader STEM learning ecosystem via (a) sharing more inclusive, culturally relevant, and responsive ISE research and practice; (b) identifying scalable, equity-focused research findings useful in ISL programming; and (c) promoting greater public awareness of the importance of broadening participation in STEM. The Center's primary stakeholders are ISL professionals, including researchers, practitioners, and evaluators. The Center will maintain and expand a digital infrastructure to support innovation and sharing across the ISL field. Through its combined efforts, the Center will raise the visibility and impact of ISL equity-focused research and practice and its contributions to the overall STEM endeavor. The Center will also organize and host the biennial AISL Awardee Meeting. The Center is funded by the Advancing Informal STEM Learning program, which seeks to (a) advance new approaches to and evidence-based understanding of the design and development of STEM learning in informal environments; (b) provide multiple pathways for broadening access to and engagement in STEM learning experiences; (c) advance innovative research on and assessment of STEM learning in informal environments; and (d) engage the public of all ages in learning STEM in informal environments.
DATE:
-
TEAM MEMBERS:
Stephen AlkinsDiane MillerLisette Torres-GeraldPati Ruiz
resourceprojectProfessional Development, Conferences, and Networks
The National Girls Collaborative Project and Education Development Center are convening “Advancing the Conversation on Scaling National Informal STEM Programs,” a two-and-a-half day knowledge-building conference that brings together key stakeholders in informal STEM education (ISE) to examine what scale looks like across informal learning settings. Currently, there is not a common definition or set of dimensions related to what it means to scale programs in informal learning settings. Approaches to scale in ISE too often center on the perspectives and needs of people who are developing and spreading programs while less consideration is given to the realities of those responsible for operationalizing programs in hyper-local contexts. This conference approaches the question of scale from the perspective of the program implementers, who are the beneficiaries of capacity building and serve as facilitators of these scaled programs. The conference also gives voice to program developers, researchers, evaluators, and funders of national informal STEM programs who study and support scale in education. The aim of the conference is to develop a new framework for scale in ISE that centers partnership and capacity building of informal educators. Such an approach to scale addresses issues of local access and diversity, equity, and inclusion, and promotes sustainability of ISE in high-need communities.
Conference discussions challenge three common misperceptions of scale across ISE: (1) simple spread and replication of turnkey programs lead to effective scale in ISE; (2) definitions of scale derived from formal learning settings should be used to scale across ISE; and (3) scale across ISE should be defined by program developers and those that seek to study it. Participants with a wide range of perspectives and who represent a diversity of organizational types will attend the conference and work together to articulate scaling success factors, barriers, diversity, equity, and inclusion strategies, and intended outcomes, distilling themes, questions, and concerns about current approaches to evaluating and researching scale in ISE. Together, conference participants will co-create a framework for scale in ISE which will define new and expanded dimensions of scale that center on capacity building and diversity, equity, and inclusion, as well as a Program Developer’s Guide for scaling ISE programs through the lens of the framework. These products will increase the knowledge and capacity of informal learning organizations involved in nationally scaled initiatives, STEM-rich institutions wanting to scale their own locally developed programs, informal STEM researchers and evaluators, and the broader field of ISE including program funders. Conference findings will be broadly disseminated through publications, conferences, and a national webinar co-hosted by the National Girls Collaborative Project and Education Development Center.
The project will refine, research and disseminate making exhibits and events that the museum has developed and tested to support early engineering skill development. The project will use cardboard, a familiar and flexible material, to support the activities. The goal is to develop insights and resources for informal educators across the museum field and beyond into how to effectively structure and facilitate open-ended maker education experiences for visitors that expand the number and kinds of museums and families who can engage in these activities. Maker education is often linked to Science, Technology, Engineering and Mathematics (STEM) learning and uses hands-on and collaborative approaches to support activities and projects that foster creativity, interest, and skill development. To address patterns of inequitable access to and participation in both formal and informal learning opportunities, the project will be designed to engage families from under-represented communities and research how they participate in informal engineering activities and environments. The project will make a suite of resources available for museums and other ISE practitioners that will be developed through iterative testing at all of the different settings. These resources will be made widely available via an open access online portal.
The project will research how effectively the use of cardboard making exhibits and events engage families, particularly families from underrepresented groups, in STEM and early engineering. The project's theoretical framework combines elements of: (1) learning sciences theories of family learning in museums; (2) making as a learning process; (3) early engineering practices and dispositions, and (4) equity in museums and the maker movement. The research will be conducted within two multi-month implementations of a large-scale Cardboard Engineering gallery at the Science Museum of Minnesota and two-week scaled implementations of the gallery at each of three recruited partner museum sites. The project design interweaves evaluation and research aims. Paired observations and surveys will be used to research how effectively the project is working in different venues. This integration of research and evaluation will generate a large data set from which to generalize about cardboard making across contexts. Case studies will be used to identify barriers to engagement that can be remedied, but they will provide a rich data set for understanding family learning and engineering in making. Research findings and products will be posted on the Center for Informal Science Education website and submitted for publication in peer-reviewed journals such as Visitor Studies, ASTC Dimensions, the Journal of Pre-College Engineering Education Research and others.
This project is funded by the National Science Foundation's (NSF's) Advancing Informal STEM Learning (AISL) program, which supports innovative research, approaches, and resources for use in a variety of learning settings.
Robots and robotics excite and challenge youths and adults. Unfortunately, the cost of purchasing robots or building useful robots is prohibitive for many low resource individuals and groups. This project will relieve this expense and provide an opportunity for resource limited individuals to experience the thrilling aspects of robotics by building a computer game that simulates robotic action. This project uses co-robotics wherein the participating player programs an avatar to assist in a symbiotic manner to achieve the goals of the game and participant. The game will provide access to the ideas and concepts such as programing, computational thinking and role assumption. The overarching goals are (1) to engage low-resource learners in STEM education through robotics in out-of-school spaces, and (2) to update the field of robotics-base STEM education to integrate the co-robotics paradigm.
This project is designed to gain knowledge on how co-robotics can be used in the informal education sector to facilitate the integration of computational science with STEM topics and to expand the educational use of co-robotics. Because the concept of co-robotics is new, a designed-based research approach will be used to build theoretical knowledge and knowledge of effective interventions for helping participants learn programing and computational thinking. Data will be collected from several sources including surveys, self-reports, in game surveys, pre and post-tests. These data collection efforts will address the following areas: Technology reliability, Resolution of cognitive tension around co-play, Accelerate discovery and initial engagement, Foster role-taking and interdependence with co-robots, Investigate social learning, and Validate measures using item response theory analysis. The DBR study questions are:
1.What design principles support the development of P3Gs that can effectively attract initial engagement in a free-choice OST space that offers large numbers of competing options? 2.What design principles support a P3G gameplay loop that enables learning of complex skills, computational thinking and co-robotics norms, and building of individual and career interest over the course of repeated engagement?
3.What design principles support P3Gs in attaining a high rate of re-engagement within low-resource OST settings? 4.What kinds of positive impact can P3Gs have on their proximal and distal environment? In addition, the project will research these questions about design: 1.What technical and game design features are needed to accommodate technological interruption? 2.What design elements or principles mitigate competition for cognitive resources between real-time play and understanding the co-robotic's behavior in relation to the code the player wrote for it? 3.What design elements are effective at getting learners in OST settings to notice and start playing the game? 4.What designs are effective at encouraging learners to engage with challenging content, particularly the transition from manual play to co-play? 5.What design elements help players develop a stake in the role the game offers? 6.What social behaviors emerge organically around a P3G prototype that is not designed to evoke specific social interactions?
This project is funded by the National Science Foundation's (NSF's) Advancing Informal STEM Learning (AISL) program, which supports innovative research, approaches, and resources for use in a variety of learning settings.
This Research in Service to Practice project will bring together representatives from six long-standing youth programs, experts in the field of out-of-school-time youth programming, and education researchers to collaboratively explore the long-term (15-25 years) impact of STEM-focused, intensive (100+ hours/year), multi-year programming. The six partnering programs have maintained records with a combined total of over 3000 alums who participated between 1995 and 2005. This four-year research project uses an explanatory, sequential, mixed-method design to carry out four steps: (1) identify and describe the impact on the lives of program alums who are now ages 30 to 45; (2) identify causal pathways from program strategies to long-term outcomes; (3) develop an understanding of these pathways from the perspective of the people who experienced them; and (4) disseminate this knowledge broadly to those associated with STEM-focused programming. Research questions include: How did these programs affect youth's lives as they progressed toward and into adulthood? What program strategies and what participant attributes contributed most to the staying power of these effects? What life events and social structures supported and inhibited participant outcomes? This project describes the effects, identifies the causal pathways, and produces materials that programs can use for both strategic planning and generating support resources. Additionally, this project provides research methodology for organizations that want to conduct their own retrospective research and lays a foundation for a more comprehensive study that includes programs without historical documentation. The project aligns with NSF's Big Idea "NSF INCLUDES: Transforming education and career pathways to help broaden participation in science and engineering" by providing essential information about the long-term effect of interventions on educational and career pathways in STEM.
The project's approach involves three phases: (1) research preparation, (2) causal structural modeling of survey data from approximately 2,000 respondents, and (3) rich qualitative follow-up. Human ecological and self-determination theories inform data collection and analyses at every project phase. In the preparation phase, program staff complete program profiles from an historic perspective by identifying program strategies that may have included, for example, scientific research, robotics development, teaching science in informal settings, and working in scientific research labs. In the quantitative phase, the project will recruit alums who attended one of the 6 youth programs between 1995 and 2005 to submit a current resume and complete an online questionnaire, based on the following scaled variables: retrospective recall of basic psychological need satisfaction and frustration in relation to perceived program strategies; STEM identity (at three time periods: pre-program; post-program; and current); current well-being; career influences; and career barriers. The questionnaire also includes open-ended questions about life events related to the following categories: family and friends, school and work, and living conditions. Analysis of the questionnaire will lead to development of a causal structural model. In the qualitative phase, data will be collected from a purposefully selected sample of 30 alums based on findings from the quantitative phase. Methods include interviews, photo journals, and STEM pathways maps. Analysis of interviews, resumes, and photo journals take place within the structure of basic psychological need satisfaction and motivational quality across ecological systems over time. Qualitative analysis uses the constant comparative method, and findings are used to update and refine the final causal structural model and inform overall findings, conclusions, and recommendations of the project.
Since the 1990s, out-of-school time programs have engaged youth from underserved communities in STEM learning and in building interest in STEM careers, yet these programs often based on untested assumptions that participation has lasting effects on education, career, and life choices related to STEM. This Research in Service to Practice project has the potential to 1) guide practitioners in program improvement and improved program outcomes; 2) provide insight into achieving program goals, such as equity, increased well-being of participants, an informed citizenry, and a diversified STEM workforce; and 3) inform multi-stakeholder decision-making with respect to this type of programming. This research also builds a foundation of research data collection and analysis methods to guide and support future research on long term-impacts and youth STEM programming. Dissemination strategies include a website, webinars, video, infographics, conference presentations, and written reports to reach stakeholders including practitioners, researchers, administrators, and funders.
This project is funded by the National Science Foundation's (NSF's) Advancing Informal STEM Learning (AISL) program, which supports innovative research, approaches, and resources for use in a variety of learning settings.
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program funds innovative research, approaches and resources for use in a variety of settings. Informal STEM educational activities have proliferated widely in the US over the last 20 years. Additional research will further validate the long-term benefits of this mode of learning. Thus, elaborating the multitude of variables in informal learning and how those variables can be used for individual learning is yet to be defined for the circumstances of the learners. Thus, the primary objective of this work is to produce robust and detailed evidence to help shape both practice and policy for informal STEM learning in a broad array of common circumstances such as rural, urban, varying economic situations, and unique characteristics and cultures of citizen groups. Rather than pursuing a universal model of informal learning, the principal investigator will develop a series of comprehensive models that will support learning in informal environments for various demographic groups. The research will undertake a longitudinal mixed-methods approach of Out of School Time/informal STEM experiences over a five-year time span of data collection for youth ages 9-19 in urban, suburban, town, and rural communities. The evidence base will include data on youth experiences of informal STEM, factors that exert an influence on participation in informal STEM, the impact of participation on choices about educational pathways and careers, and preferences for particular types of learning activities. The quantitative data will include youth surveys, program details (e.g. duration of program, length of each program session, youth/facilitator ratio, etc.), and demographics. The qualitative data will include on-site informal interviews with youth and facilitators, and program documentation. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program funds innovative research, approaches, and resources for use in a variety of settings. This project will develop a national infrastructure of state and regional partnerships to scale up The Franklin Institute's proven model of Leap into Science, an outreach program that builds the capacity of children (ages 3-10) and families from underserved communities to participate in science where they live. Leap into Science combines children's science-themed books with hands-on science activities to promote life-long interest and knowledge of science, and does so through partnerships with informal educators at libraries, museums, and other out-of-school time providers. Already field-tested and implemented in 12 cities, Leap into Science will be expanded to 90 new rural and urban communities in 15 states, and it is estimated that this expansion will reach more than 500,000 children and adults as well as 2,700 informal educators over four years. The inclusion of marginalized rural communities will provide new opportunities to evaluate and adapt the program to the unique assets and needs of rural families and communities.
The project will include evaluation and learning research activities. Evaluation will focus on: 1) the formative issues that may arise and modifications that may enhance implementation; and 2) the overall effectiveness and impact of the Leap into Science program as it is scaled across more sites and partners. Learning research will be used to investigate questions organized around how family science interest emerges and develops among 36 participating families across six sites (3 rural, 3 urban). Qualitative methods, including data synthesis and cross-case analysis using constant comparison, will be used to develop multiple case studies that provide insights into the processes and outcomes of interest development as families engage with Leap into Science and a conceptual framework that guides future research. This project involves a partnership between The Franklin Institute (Philadelphia, PA), the National Girls Collaborative Project (Seattle, WA), Education Development Center (Waltham, MA), and the Institute for Learning Innovation (Corvallis, OR).