In fall 2019, the Bell Museum received funding via a NASA TEAM II grant to create Mars: The Ultimate Voyage, a full-dome planetarium show and accompanying hands-on activities that focus on the interdisciplinary roles that will be needed to send humans to Mars. This report from Catalyst Consulting Group presents the findings from the summative evaluation completed in March–May 2023.
This project is expanding an effective mobile making program to achieve sustainable, widespread impact among underserved youth. Making is a design-based, participant-driven endeavor that is based on a learning by doing pedagogy. For nearly a decade, California State University San Marcos has operated out-of-school making programs for bringing both equipment and university student facilitators to the sites in under-served communities. In collaboration with four other CSU campuses, this project will expand along four dimensions: (a) adding community sites in addition to school sites (b) adding rural contexts in addition to urban/suburban, (c) adding hybrid and online options in addition to in-person), and (d) including future teachers as facilitators in addition to STEM undergraduates. The program uses design thinking as a framework to engage participants in addressing real-world problems that are personally and socially meaningful. Participants will use low- and high-tech tools, such as circuity, coding, and robotics to engage in activities that respond to design challenges. A diverse group of university students will lead weekly, 90-minute activities and serve as near-peer mentors, providing a connection to the university for the youth participants, many of whom will be first-generation college students. The project will significantly expand the Mobile Making program from 12 sites in North San Diego County to 48 sites across California, with nearly 2,000 university facilitators providing 12 hours of programming each year to over 10,000 underserved youth (grades 4th through 8th) during the five-year timeline.
The project research will examine whether the additional sites and program variations result in positive youth and university student outcomes. For youth in grades 4 through 8, the project will evaluate impacts including sustained interest in making and STEM, increased self-efficacy in making and STEM, and a greater sense that making and STEM are relevant to their lives. For university student facilitators, the project will investigate impacts including broadened technical skills, increased leadership and 21st century skills, and increased lifelong interest in STEM outreach/informal science education. Multiple sources of data will be used to research the expanded Mobile Making program's impact on youth and undergraduate participants, compare implementation sites, and understand the program's efficacy when across different communities with diverse learner populations. A mixed methods approach that leverages extant data (attendance numbers, student artifacts), surveys, focus groups, making session feedback forms, observations, and field notes will together be used to assess youth and university student participant outcomes. The project will disaggregate data based on gender, race/ethnicity, grade level, and site to understand the Mobile Making program's impact on youth participants at multiple levels across contexts. The project will further compare findings from different types of implementation sites (e.g., school vs. library), learner groups, (e.g., middle vs. upper elementary students), and facilitator groups (e.g., STEM majors vs. future teachers). This will enable the project to conduct cross-case comparisons between CSU campuses. Project research will also compare findings from urban and rural school sites as well as based on the modality of teaching and learning (e.g., in-person vs. online). The mobile making program activities, project research, and a toolkit for implementing a Mobile maker program will be widely disseminated to researchers, educators, and out-of-school programs.
This Innovations in Development project aims to foster the development of STEM identity among a diverse group of middle school students and, in turn, motivate them to pursue in STEM interests and careers. Vegas STEM Lab, led by a team of investigators from the University of Nevada, Las Vegas, will employ a mix of online and on-site activities to introduce students to engineering methods in the context of the entertainment and hospitality (E&H) industry that is the lifeblood of Las Vegas. Investigators will collaborate with local resorts, multimedia designers, and arts institutions to offer field experiences for students to interview, interact with, and learn from local experts. The Lab will help youth overcome prevailing beliefs of STEM as boring and difficult, boost their confidence as STEM-capable individuals, and expose them to the exciting STEM careers available in their hometown. UNLV engineering undergrads will serve as near-peer mentors to the middle school students, guiding them through Lab activities and acting as role models. Investigators will measure student learning and engagement over the course of the Vegas STEM Lab experience with the aim of understanding how the Lab model—with its rich set of activities and interpersonal interactions set in the local E&H industry—can cultivate STEM identity development and encourage students to pursue STEM pathways. Despite the project’s hyperlocal focus on the Las Vegas community, if successful, other cities and towns may learn from and adapt the Lab model for use in their youth development programs.
Vegas STEM Lab will provide online materials for students’ STEM learning during the academic year followed by on-site visits and hands-on project development during a three-week summer experience. The Lab will run for three years with cohorts of 40 students each (N=120) with the aim of iteratively improving its activities and outcomes from year to year. The local school district will help recruit middle school students who have demonstrated low interest in STEM to participate in the Lab, ensuring that participants reflect the demographic makeup of the Las Vegas community in terms of race and ethnicity, socio-economic status, and gender. Summer activities will take students behind the scenes of the city’s major E&H venues; investigate the workings of large-scale displays, light shows, and “smart hospitality” systems; and then build their own smaller scale engineering projects. Investigators will employ the Dynamic Systems Model of Role Identity (DSMRI) framework to study how intentionally designed Lab experiences shape students’ understanding of themselves, their future aspirations, and their grasp of the scientific enterprise. Summer activities will be integrated into the online learning platform at the end of each year of Vegas STEM Lab, and in the final year of the project, workshops will train local educators to use the platform in either formal or informal learning settings. Materials and research findings produced through this work will be disseminated to middle school teachers and afterschool care providers, and shared with researchers through academic publications and conferences.
This Innovations in Development project is funded by the Advancing Informal STEM Learning (AISL) program.
DATE:
-
TEAM MEMBERS:
Emma RegentovaVenkatesan MuthukumarJonathan HilpertSi Jung Kim
This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).
Research shows that Black girls and women, regardless of their academic achievements and STEM interests, often encounter academic under-preparation, social isolation, exclusion, and race-gender discrimination that negatively impacts their ongoing engagement and retention in STEM. This project will provide innovative, culturally relevant learning environments to middle and high school Black girls to counter these negative trends. Using hands-on coding and robotics activities, project participants will develop positive attitudes toward science, technology, engineering, and mathematics (STEM). The project emphasizes peer-mentoring by providing opportunities for Black female high school (assistant coaches) and Black college students (coaches) to serve as counselors and mentors to participants. Additionally, engineers, scientists, and executives from STEM industries will serve as mentors and share their experiences to broaden participants’ STEM career aspirations. The project is a three-year collaborative effort between the University of California Davis C-STEM Center, the Umoja Community Education Foundation, and the 66 affiliated California community colleges, industry partners, and school districts in California. Over three years, nearly 2,000 females will participate in the project.
Learning environments for Black girls and women led by other Black girls and women are referred to as “counterspaces” where they are free to engage in STEM in ways that value their identities while promoting STEM engagement, interests, and career aspirations. The project’s curriculum will follow a research-based, culturally relevant multi-tiered mentoring approach. The curriculum is designed to develop participants’ STEM content knowledge, critical thinking, and logical reasoning capabilities through meaningful connections to real-life applications using hands-on coding and robotics. A mixed-method longitudinal study will examine the impact on participants’ STEM outcomes, emphasizing contributing new knowledge on the viability of multi-tiered, culturally relevant mentoring for increasing equity in informal STEM learning (ISL). The program's effectiveness will be evaluated using longitudinal assessments of mathematics standards, computer science and robotics conceptual knowledge, logical and critical thinking skills, STEM school achievements, interests and attitudes toward STEM subjects, advanced STEM course-taking, involvement in other ISL opportunities, and leadership in STEM in one’s school/university and community. The project will test a locally based informal learning model with projects hosted by other K-12 and college partners.
This Innovations in Development project is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to (a) advance new approaches to and evidence-based understanding of the design and development of STEM learning in informal environments; (b) provide multiple pathways for broadening access to and engagement in STEM learning experiences; (c) advance innovative research on and assessment of STEM learning in informal environments; and (d) engage the public of all ages in learning STEM in informal environments.
Black Girls Create (BGC) is a program that uses maker pedagogy, social history (i.e., Black women’s contributions in STEM), culturally responsive pedagogy, and mentoring to engage Black girls in STEM. For the project, culturally responsive making is operationally defined as the use of cultural knowledge and maker technologies to create, design, and produce artifacts that are related to a particular concept, theme, or person. The project will conduct a feasibility research study that investigates how engagement in BGC affects participants' STEM interest, STEM confidence, and racial and gender identities. In the project, participants will discover how Black women have made an impact in STEM fields and will learn how to digitally create cultural artifacts/wearable art related to their discoveries. Approximately 120 middle school girls in grades 6-8 (8 groups of 15 participants each) from underserved neighborhoods will be recruited for participation in this two-year study. Each group will meet for 3 hours per week over an 8-week period. Instructional materials from this research will be made freely available online so that they can be adapted and used at other formal and informal educational institutions that seek to garner interest and access to STEM learning for Black girls and women.
A pre- and post-test, quasi-experimental design will be used to research the program’s influence on participants’ self-efficacy, racial identity, and gender identity. Lagged regression models that control for students’ age, race/ethnicity, and pre-survey scores will be used to examine growth in each of the four outcomes. Once all of the participants in the groups have completed BGC, data from all groups will be combined to increase power and thereby detect statistically meaningful differences in pre- and post-survey scores. In addition, variables representing attendance and program engagement will be entered into the model to examine whether students who are highly engaged in BGC programming exhibit more growth in the four outcomes. Qualitative data will be derived from students' journal entries as well as focus group interviews. Weekly journals will be used to gather data about the context and experiences of participants as they unfold throughout the program. Semi-structured focus groups will center around the usefulness of skills and knowledge gained from the program activities, significant experiences with peers and program staff, the opportunity to learn about Black women STEM progenitors, and learning how to make digitally fabricated artifacts. NVIVO, a computer software program, will be used as a tool to support the analysis of the rich, text-based information resulting from the journals and focus group narratives.
This feasibilty research project is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to (a) advance new approaches to and evidence-based understanding of the design and development of STEM learning in informal environments; (b) provide multiple pathways for broadening access to and engagement in STEM learning experiences; (c) advance innovative research on and assessment of STEM learning in informal environments; and (d) engage the public of all ages in learning STEM in informal environments.
Science Hunters is an outreach project which employs the computer game Minecraft to engage children with scientific learning and research through school visits, events, and extracurricular clubs. We principally target children who may experience barriers to accessing Higher Education, including low socioeconomic status, being the first in their family to attend university, and disability (including Special Educational Needs). The Minecraft platform encourages teamwork and makes science learning accessible and entertaining for children, irrespective of background. We employ a flexible approach
DATE:
TEAM MEMBERS:
Laura HobbsCarly StevensJackie HartleyCalum Hartley
Program evaluators from the Education Development Center (EDC) used a mixed-methods, quasi-experimental design to evaluate the impact on girls’ awareness and interest in science, technology, engineering, and mathematics (STEM). After the final year of the project, EDC delivered a summative report to Techbridge Girls (TBG), which was based on data collected during the five-year grant period, with a particular focus on the final year that grant funds supported programming (2017-18). Data included pre- and post-surveys with TBG participants and comparison students, participant focus groups, and
The NIH Science Education Partnership Award (SEPA) program of Emory University endeavors to use an over-arching theme of citizen science principles to:
develop an innovative curriculum based on citizen science and experiential learning to evaluate the efficacy of informal science education in after-school settings;
promote biomedical scientific careers in under-represented groups targeting females for Girls for Science summer research experiences;
train teachers in Title I schools to implement this citizen science based curriculum; and
disseminate the citizen science principles through outreach.
This novel, experiential science and engineering program, termed Experiential Citizen Science Training for the Next Generation (ExCiTNG), encompasses community-identified topics reflecting NIH research priorities. The curriculum is mapped to Next Generation Science Standards.
A comprehensive evaluation plan accompanies each program component, composed of short- and/or longer-term outcome measures. We will use our existing outreach program (Students for Science) along with scientific community partnerships (Atlanta Science Festival) to implement key aspects of the program throughout the state of Georgia. These efforts will be overseen by a central Steering Committee composed of leadership of the Community Education Research Program of the Emory/Morehouse/Georgia Institute of Technology Atlanta Clinical Translational Science Institute (NIH CTSA), the Principal Investigators, representatives of each program component, and an independent K–12 STEM evaluator from the Georgia Department of Education.
The Community Advisory Board, including educators, parents, and community members, will help guide the program’s implementation and monitor progress. A committee of NIH-funded investigators, representing multiple NIH institutes along with experienced science writers, will lead the effort for dissemination and assure that on-going and new NIH research priorities are integrated into the program’s curriculum over time.
Techbridge Girls’ mission is to help girls discover a passion for science, engineering, and technology (SET). In August 2013, Techbridge Girls was awarded a five-year National Science Foundation grant to scale up its afterschool program from the San Francisco Bay Area to multiple new locations around the United States. Techbridge Girls began offering afterschool programming at elementary and middle schools in Greater Seattle in 2014, and in Washington, DC in 2015.
Education Development Center is conducting the formative and summative evaluation of the project. To assess the implementation
Techbridge Girls’ mission is to help girls discover a passion for science, engineering, and technology (SET). In August 2013, Techbridge Girls was awarded a five-year National Science Foundation grant to scale up its afterschool program from the San Francisco Bay Area to multiple new locations around the United States. Techbridge Girls began offering afterschool programming at elementary and middle schools in Greater Seattle in 2014, and in Washington, DC in 2015.
Education Development Center is conducting the formative and summative evaluation of the project. To assess the implementation
Techbridge Girls’ mission is to help girls discover a passion for science, engineering, and technology (SET). In August 2013, Techbridge Girls was awarded a five-year National Science Foundation grant to scale up its after-school program from the San Francisco Bay Area to multiple new locations around the United States. Techbridge Girls began offering after-school programming at elementary and middle schools in Greater Seattle in 2014, and in Washington, DC in 2015.
Education Development Center is conducting the formative and summative evaluation of the project. To assess the
Techbridge Girls’ mission is to help girls discover a passion for science, engineering, and technology (SET). In August 2013, Techbridge Girls was awarded a five-year National Science Foundation grant to scale up its after-school program from the San Francisco Bay Area to multiple new locations around the United States. Techbridge Girls began offering after-school programming at elementary and middle schools in Greater Seattle in 2014, and in Washington, DC in 2015.
Education Development Center is conducting the formative and summative evaluation of the project. To assess the