Since the late 2000s, interest in the development and use of shared measures in the informal science, technology, engineering, and mathematics (STEM) education (ISE) field has increased. The intent is to build the capacity of evaluators to measure common outcomes of ISE experiences. We begin this chapter with a definition of shared measures, a description of related technical qualities of these measures, and a discussion of benefits and concerns around the use of shared measures. We then review recent conversations and developments around shared measures, including examples of observational
This research in service to practice project will examine the impact of a 12-year statewide science field trip program called LabVenture. This hands-on program in discovery and inquiry brings middle school students and teachers across the State of Maine to the Gulf of Maine Research Institute (GMRI) in Portland, Maine to become fully immersed in explorations into the complexities of local marine science ecosystems. These intensive field trip experiences are led by informal educators and facilitated entirely within informal contexts at GMRI. Approximately 70% of all fifth and sixth grade students in Maine participate in the program each year and more than 120,000 students have attended since the program's inception in 2005. Unfortunately, little is known to date on how the program has influenced practice and learning ecosystems within formal, informal, and community contexts. As such, this research in service to practice project will employ an innovative research approach to understand and advance knowledge on the short and long-term impacts of the program within different contexts. If proven effective, the LabVenture program will elucidate the potential benefits of a large-scale field trip program implemented systemically across a community over time and serve as a reputable model for statewide adoption of similar programs seeking innovative strategies to connect formal and informal science learning to achieve notable positive shifts in their local, statewide, or regional STEM learning ecosystems.
Over the four-year project duration, the project will reach all 16 counties in the State of Maine. The research design includes a multi-step, multi-method approach to gain insight on the primary research questions. The initial research will focus on extant data and retrospective data sources codified over the 12-year history of the program. The research will then be expanded to garner prospective data on current participating students, teachers, and informal educators. Finally, a community study will be conducted to understand the potential broader impacts of the program. Each phase of the research will consider the following overarching research questions are: (1) How do formal and informal practitioners perceive the value and purposes of the field trip program and field trip experiences more broadly (field trip ontology)? (2) To what degree do short-term field trip experiences in informal contexts effect cognitive and affective outcomes for students? (3) How are community characteristics (e.g., population, distance from GMRI, proximity to the coast) related to ongoing engagement with the field trip program? (4) What are aspects of the ongoing field trip program that might embed it as an integral element of community culture (e.g., community awareness of a shared social experience)? (5) To what degree does a field trip experience that is shared by schools across a state lead to a traceable change that can be measured for those who participated and across the broader community? and (6) In what ways, if at all, can a field trip experience that occurs in informal contexts have an influence on the larger learning ecosystem (e.g., the Maine education system)? Each phase of the research will be led by a team of researchers with the requisite expertise in the methodologies and contexts required to carry out that particular aspect of the research (i.e., retrospective study, prospective study, community study). In addition, evaluation and practitioner panels of experts will provide expertise and guidance on the research, evaluation, and project implementation. The project will culminate with a practitioner convening, to share project findings more broadly with formal and informal practitioners, and promote transfer from research to practice. Additional dissemination strategies include conferences, network meetings, and peer-reviewed publications.
The potential insights this research could garner on intersectionality between formal and informal STEM learning are substantial. As a consequence, this project is co-funded by the Advancing Informal STEM Learning (AISL) and Discovery Research K-12 (DRK-12) Programs. The Advancing Informal STEM Learning (AISL) program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. Likewise, the Discovery Research-K12 Program seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program funds innovative research, approaches and resources for use in a variety of settings. Informal STEM educational activities have proliferated widely in the US over the last 20 years. Additional research will further validate the long-term benefits of this mode of learning. Thus, elaborating the multitude of variables in informal learning and how those variables can be used for individual learning is yet to be defined for the circumstances of the learners. Thus, the primary objective of this work is to produce robust and detailed evidence to help shape both practice and policy for informal STEM learning in a broad array of common circumstances such as rural, urban, varying economic situations, and unique characteristics and cultures of citizen groups. Rather than pursuing a universal model of informal learning, the principal investigator will develop a series of comprehensive models that will support learning in informal environments for various demographic groups. The research will undertake a longitudinal mixed-methods approach of Out of School Time/informal STEM experiences over a five-year time span of data collection for youth ages 9-19 in urban, suburban, town, and rural communities. The evidence base will include data on youth experiences of informal STEM, factors that exert an influence on participation in informal STEM, the impact of participation on choices about educational pathways and careers, and preferences for particular types of learning activities. The quantitative data will include youth surveys, program details (e.g. duration of program, length of each program session, youth/facilitator ratio, etc.), and demographics. The qualitative data will include on-site informal interviews with youth and facilitators, and program documentation. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
This pilot study will examine the effectiveness of an innovative applied social change, community and technology based program on marginalized youths' access, interest, efficacy and motivation to learn and engage in digital technology applications. Using stratified near-peer and peer-to-peer mentoring approaches, the pilot builds on extant literature that indicates that peer-supported hands-on mentoring and experiences can alleviate some barriers to youth engagement in digital technologies, particularly among underrepresented groups. In this project, undergraduate students will mentor and work collaboratively with high school youth primarily of Hispanic descent and community-based organizations to develop creative technology-based solutions to address social issues and challenges within their local communities, culminating in events called Impactathons. These community-hosted local and state-wide events set this pilot project apart from similar work in the field. The Impactathons not only provide a space for intellectual discourse and problem-solving among the undergraduate-youth-community partners but the Impactathons will also codify expertise from scientists, social scientists, technologists, community leaders, and other stakeholders to develop technology-based solutions with real world application. If successful, a distal outcome will be increased youth interest in digital technologies and related fields. In the short term, favorable findings will provide preliminary evidence of success and lay the foundation for a more extensive study in the future.
This pilot project is a collaboration between the Everett Program, a student-led program for Technology and Social Change at the University of California Santa Cruz - a Hispanic Serving Institution - and the Digital NEST, a non-profit, high-tech youth career development and collaboration space for young people ages 14-24. Through this partnership and other recruitment efforts, an estimated 70-90 individuals will participate in the Impactathon pilot program over two years. Nearly two-thirds of the participants are expected to be undergraduate students. They will receive extensive training in near-peer and peer-to-peer mentoring and serve as mentors for and co-innovation developers with the high school youth participants. The undergraduates and youth will partner with local community organizations to identify a local social challenge that can be addressed through a technology-based solution. The emergent challenges will vary and could span the spectrum of STEM and applied social science topics of interest. Working in informal contexts (i.e., afterschool. weekend), the undergraduate-youth-community partner teams will work collaboratively to develop practical technology-based solutions to real world challenges. The teams will convene three times per year, locally and statewide, at student and community led Impactathons to share their work and glean insights from other teams to refine their innovations. In parallel, the research team will examine the effectiveness of the Impactathon model in increasing the undergraduate and youths' interest, motivation, excitement, engagement and learning of digital technologies. In addition to the research, the formative and summative evaluations should provide valuable insights on the effectiveness of the model and its potential for expansion and replication.
The project is co-funded by the Advancing Informal STEM Learning (AISL) Program and STEM +C. The AISL program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. STEM + C focuses on research and development of interdisciplinary and transdisciplinary approaches to the integration of computing within STEM teaching and learning for preK-12 students in both formal and informal settings.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants. The three-year project, Montana Models: Connecting Local and Disciplinary Practices through University-Community Partnerships, focuses on creating, implementing, and studying several learning outcomes associated with youth engagement in mathematical modeling contexts. The project builds on existing partnerships between the state's two research universities and Montana 4-H to target outreach to rural youth and bring them into a network of people who can inspire, support, and sustain STEM learning. Middle school and high school students from rural communities will be invited to a university campus for a residential modeling-based summer program l focused on mathematics and mathematical modeling. Activities at the summer program are designed to engage them in problems relevant to their own backgrounds and experiences and to honor their local funds of knowledge. The primary goal of Montana Models is to use mathematical modeling as a mechanism for bringing everyday mathematical practices already present in rural communities into contact with disciplinary practices. The project focuses on the following research questions: (1) What are the everyday mathematical practices in Montana communities? (2) How can everyday mathematical practices be leveraged and brought into contact with disciplinary practices in service of mathematizing meaningful questions within the community? (3) How do youth identify and get identified with respect to mathematics and with respect to their role in the world? (4) How does participation in project activities affect participants' knowledge of mathematical practices and content? The project uses social design experimentation, a hybrid research methodology which combines the traditions of design-based research with forms of inquiry that involve collaboration among participants, researchers, and other stakeholders, such as critical ethnography. Data sources include field notes from ethnographic observations, interviews, videos of students engaging in modeling activities, artifacts that show their mathematical work, and results from the Attitudes Towards Mathematics Inventory. Through its collaboration with 4-H, Montana Models targets outreach to rural youth across the state, especially those from groups that are typically underrepresented in STEM fields. The project is poised to impact ways in which formal and informal educators understand the knowledge bases that are already present in rural communities and how those bases may inform, support, and sustain STEM learning. Findings and deliverables will be disseminated through a public-facing website and through the 4-H infrastructure. This infrastructure includes Montana 4-H's Clover Communication Contest that will allow participating youth to showcase their projects. Research findings will be shared through local and national conferences and peer-reviewed publications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants. This Innovations in Development project will develop and test a model for audience assessment that STEM professionals can use during their public engagement efforts. These on-the-spot assessments will allow scientists to monitor audience understanding and use this to make immediate improvements to their activity. This project fills a critical gap in the field of public science communication as expressed by outreach scientists and by the professional and academic organizations that train them. Project partners include The Astronomical Society of the Pacific, Oregon State University, Pacific Science Center, and the National Radio Astronomy Observatory. Research questions include: (1) How and under what conditions can scientists, regardless of discipline, learn to build assessment into outreach activities? (2) To what degree are scientists willing and able to change their outreach activities to include assessment to ascertain audience attentiveness or understanding? (3) To what degree will scientists be able to adjust their outreach and engagement efforts based on audience feedback, and what support do they need to do so? (4) With constraints on their time and resources, how can the model help scientists conduct audience assessment on their own? (5) Are audience members finding events more enjoyable and fruitful that use assessment techniques? Several iterative rounds of R&D will be used in the spiral design-based approach. A group of Design Testers will start each of the iterative cycles for the initial development and testing techniques, associated training, and support. Later phases will use Field Testers using a rubric for usability, validity, and reliability of various strategies. The summative evaluation of the project will begin early and add data over the life of the project. A mixed methods approach will be used including surveys of scientists once each year after they have participated in the Prototype training. Descriptive and inferential statistics will be used to document the extent to which the project has achieved its intended outcomes. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program funds innovative research, approaches and resources for use in a variety of settings. This project examines the conditions in which families and young learners most benefit from "doing science and math" together among a population that is typically underserved with respect to STEM experiences--families experiencing poverty. This project builds on an existing program called Teaching Together that uses interactive parent-child workshops led by a museum educator and focused on supporting STEM learning at home. The goal of these workshops is to increase parents'/caregivers' self-perception and ability to serve as their child's first teacher by supporting learning and inquiry conversations during daily routines and informal STEM activities. Families attend a series of afternoon and evening workshops at their child's preschool center and at a local children's museum. Parents/Caregivers may participate in online home learning activities and museum experiences. The project uses an experimental design to test the added value of providing incremental supports for informal STEM learning. The study uses an experimental design to address potential barriers parents/caregivers may perceive to doing informal STEM activities with their child. The project also explores how the quantity and quality parent-child informal learning interactions may relate to changes in children's science and mathematics knowledge during the pre-kindergarten year. The project partners include the Children's Learning Institute at the University of Texas Health Science Center at Houston and the Children's Museum of Houston.
The project is designed to increase understanding of how parents/caregivers can be encouraged to support informal STEM learning by experimentally manipulating key aspects of the broader expectancy-value-cost motivation theory, which is well established in psychology and education literatures but has not been applied to preschool parent-child informal STEM learning. More specifically, the intervention conditions are designed to identify how specific parent supports can mitigate potential barriers that families experiencing poverty face. These intervention conditions include: modeling of informal STEM learning during workshops to address skills and knowledge barriers; materials to address difficulties accessing science and math resources; and incentives as a way to address parental time pressures and/or costs and thereby improve involvement in informal learning activities. Intervention effects will be calculated in terms of effect sizes and potential mediators of change will be explored with structural equation modeling. The first phase of the project uses an iterative process to refine the curriculum and expand the collection of resources designed for families of 3- to 5-year-olds. The second phase uses an experimental study of the STEM program to examine conditions that maximize participation and effectiveness of family learning programs. In all, 360 families will be randomly assigned to four conditions: 1) business-as-usual control; 2) the Teaching Together core workshop-based program; 3) Teaching Together workshops + provision of inquiry-based STEM activity kits for the home; and 4) Teaching Together workshop + activity kits + provision of monetary incentives for parents/caregivers when they document informal STEM learning experiences with their child. The interventions will occur in English and Spanish. A cost analysis across the interventions will also be conducted. This study uses quantitative and qualitative approaches. Data sources include parent surveys and interviews, conversation analysis of home learning activities, parent photo documentation of informal learning activities, and standardized assessments of children's growth in mathematics, science, and vocabulary knowledge.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
This award was provided as part of NSF's Social, Behavioral and Economic Sciences Postdoctoral Research Fellowships (SPRF) program and is supported by SBE's Developmental Sciences program and the Directorate for Education and Human Resources' (EHR) Advancing Informal STEM Learning program. The goal of the SPRF program is to prepare promising, early career doctoral-level scientists for scientific careers in academia, industry or private sector, and government. SPRF awards involve two years of training under the sponsorship of established scientists and encourage Postdoctoral Fellows to perform independent research. NSF seeks to promote the participation of scientists from all segments of the scientific community, including those from underrepresented groups, in its research programs and activities; the postdoctoral period is considered to be an important level of professional development in attaining this goal. Each Postdoctoral Fellow must address important scientific questions that advance their respective disciplinary fields. Under the sponsorship of Dr. Sandra D. Simpkins at the University of California, Irvine, this postdoctoral fellowship award supports an early career scientist exploring high-quality and culturally responsive, math afterschool program (ASP) practices for under-represented minority (URM) youth. Mathematical proficiency is the foundation of youth's STEM pursuits. Yet today, far too many youth do not pursue STEM based on a perception that they are "not good at math". Students need to engage in contexts that spark their interest and their continued mastery and growth. ASPs are settings for such dynamic opportunities, particularly for URM students such as Latinos who attend lower quality schools and do not feel supported. In college, URM students often struggle with uninspiring and culturally incongruent STEM learning environments. The intergenerational nature of university-based STEM ASPs, whereby younger students are paired with undergraduate (UG) mentors, are opportunities to support both K-12 and UG students' motivational beliefs in math and STEM more broadly. This project will examine these intergenerational developmental processes in the context of a math enrichment ASP located at a Hispanic-Serving Institution. By studying how ASPs can serve as an important lever for promoting URM students' access and success in STEM, this project seeks to meaningfully inform efforts to broaden the participation of underrepresented groups in these fields.
This project seeks to understand how participating in a math enrichment ASP supports both youth participants' and UG mentors' motivational beliefs in math; to describe high-quality and culturally responsive practices; and to understand how to support the effectiveness of youth-staff relationships. To accomplish these research objectives, data will be collected from both youth participants and UG mentors through multiple methods including surveys, in-depth interviews, participant-observations, and video observations of youth-staff interactions. This project will add to our understanding of university-ASP partnerships. Further, the knowledge gained from this study will impact the larger landscape of practice and research on STEM ASPs by 1) addressing critical gaps in the current literature on high-quality and culturally responsive STEM ASP practices and 2) informing ASP staff development training. Overall, this mixed methods project will provide critical and rich information on the ways that ASPs can effectively deliver on its promise of promoting positive development for all youth, especially URM youth who may need and benefit from these spaces the most. The invaluable insight garnered from this study will be disseminated to traditional academic audiences to advance knowledge, as well as to local, state, and national organizations to inform the larger landscape of practice in STEM ASPs.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
Informal STEM education needs high quality program evaluation. Research is particularly needed on the relationship between STEM outcomes and positive youth development (PYD) / socio-emotional learning (SEL). This pilot and feasibility project involves a collaboration between experts in STEM education, out-of-school time programs (OST), PYD, SEL, evaluation, and program development. STEM Scouts helps youth grow in character and develop skills using experiential activities and interaction with STEM professionals. This project expands the implementation of the Systems Evaluation Protocol (SEP), an evidence-based approach to evaluation planning.
The goals are to: 1) develop a theory of change for STEM Scouts highlighting the relationship between STEM outcomes and PYD/SEL outcomes; 2) pilot enhancements to the SEP (System Mapping, Ecosystem Modeling, and Model Validation); and, 3) determine the feasibility of conducting a national STEM Scouts study. STEM Scouts leadership and project researchers will work through the SEP to generate a stakeholder map, logic model, and pathway model (PM). Five STEM Scouts Labs across the country will participate in focus groups where they will complete SM to identify the system in which the Lab exists (e.g., stakeholders and decision-makers), review and revise their system maps, identify key outcomes and connections in the PM, and discuss how the PM reflects their STEM Scouts experiences. It is hypothesized the enhanced SEP will enable the working group to better understand factors hindering or enabling program and evaluation feasibility and success. Findings will be disseminated to the evaluation/research community, OST program providers/developers and the public.
This project is funded by the National Science Foundation's Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments.
Techbridge Girls’ mission is to help girls discover a passion for science, engineering, and technology (SET). In August 2013, Techbridge Girls was awarded a five-year National Science Foundation grant to scale up its afterschool program from the San Francisco Bay Area to multiple new locations around the United States. Techbridge Girls began offering afterschool programming at elementary and middle schools in Greater Seattle in 2014, and in Washington, DC in 2015.
Education Development Center is conducting the formative and summative evaluation of the project. To assess the implementation
Techbridge Girls’ mission is to help girls discover a passion for science, engineering, and technology (SET). In August 2013, Techbridge Girls was awarded a five-year National Science Foundation grant to scale up its afterschool program from the San Francisco Bay Area to multiple new locations around the United States. Techbridge Girls began offering afterschool programming at elementary and middle schools in Greater Seattle in 2014, and in Washington, DC in 2015.
Education Development Center is conducting the formative and summative evaluation of the project. To assess the implementation
Techbridge Girls’ mission is to help girls discover a passion for science, engineering, and technology (SET). In August 2013, Techbridge Girls was awarded a five-year National Science Foundation grant to scale up its after-school program from the San Francisco Bay Area to multiple new locations around the United States. Techbridge Girls began offering after-school programming at elementary and middle schools in Greater Seattle in 2014, and in Washington, DC in 2015.
Education Development Center is conducting the formative and summative evaluation of the project. To assess the