In fall 2019, the Bell Museum received funding via a NASA TEAM II grant to create Mars: The Ultimate Voyage, a full-dome planetarium show and accompanying hands-on activities that focus on the interdisciplinary roles that will be needed to send humans to Mars. This report from Catalyst Consulting Group presents the findings from the summative evaluation completed in March–May 2023.
We present the assets that collaboration across a land grant university brought to the table, and the Winterberry Citizen Science program design elements we have developed to engage our 1080+ volunteer berry citizen scientists ages three through elder across urban and rural, Indigenous and non-Indigenous, and formal and informal learning settings.
DATE:
TEAM MEMBERS:
Katie SpellmanJasmine ShawChristine VillanoChrista MulderElena SparrowDouglas Cost
We used a youth focused wild berry monitoring program that spanned urban and rural Alaska to test this method across diverse age levels and learning settings.
DATE:
TEAM MEMBERS:
Katie SpellmanDouglas CostChristine Villano
resourceevaluationMuseum and Science Center Exhibits
The Kaulele Kapa Exhibit was created to explore the effectiveness of a Hawaiian culture-based framework and approach in increasing learner engagement and depth of knowledge in STEM among Native Hawaiian/Pacific Islander (NHPI) learners. The exhibit utilized hands-on and interactive activities, coupled with scientific and cultural information, to create relevant learning experiences for these communities. To determine the effectiveness, exhibit attendees were invited to complete a survey that asked about how the exhibit influenced their interest and understanding of STEM and Hawaiian culture
The project team published a research synopsis article with Futurum Science Careers in Feb 2023 called “How Can Place Attachment Improve Scientific Literacy?”
The University of Montana spectrUM Discovery Area will implement “Making Across Montana” —a project to engage K–12 students and teachers in rural and tribal communities with making and tinkering. In collaboration with K–12 education partners in the rural Bitterroot Valley and on the Flathead Indian Reservation, the museum will develop a mobile making and tinkering exhibition and education program. The exhibition will be able to travel to K–12 schools statewide. The project team will develop a K–12 teacher professional development workshop, along with accompanying curriculum resources and supplies. The traveling program and related materials will build schools’ capacity to incorporate making and tinkering—and informal STEM experiences more broadly—into their teaching.
This project is expanding an effective mobile making program to achieve sustainable, widespread impact among underserved youth. Making is a design-based, participant-driven endeavor that is based on a learning by doing pedagogy. For nearly a decade, California State University San Marcos has operated out-of-school making programs for bringing both equipment and university student facilitators to the sites in under-served communities. In collaboration with four other CSU campuses, this project will expand along four dimensions: (a) adding community sites in addition to school sites (b) adding rural contexts in addition to urban/suburban, (c) adding hybrid and online options in addition to in-person), and (d) including future teachers as facilitators in addition to STEM undergraduates. The program uses design thinking as a framework to engage participants in addressing real-world problems that are personally and socially meaningful. Participants will use low- and high-tech tools, such as circuity, coding, and robotics to engage in activities that respond to design challenges. A diverse group of university students will lead weekly, 90-minute activities and serve as near-peer mentors, providing a connection to the university for the youth participants, many of whom will be first-generation college students. The project will significantly expand the Mobile Making program from 12 sites in North San Diego County to 48 sites across California, with nearly 2,000 university facilitators providing 12 hours of programming each year to over 10,000 underserved youth (grades 4th through 8th) during the five-year timeline.
The project research will examine whether the additional sites and program variations result in positive youth and university student outcomes. For youth in grades 4 through 8, the project will evaluate impacts including sustained interest in making and STEM, increased self-efficacy in making and STEM, and a greater sense that making and STEM are relevant to their lives. For university student facilitators, the project will investigate impacts including broadened technical skills, increased leadership and 21st century skills, and increased lifelong interest in STEM outreach/informal science education. Multiple sources of data will be used to research the expanded Mobile Making program's impact on youth and undergraduate participants, compare implementation sites, and understand the program's efficacy when across different communities with diverse learner populations. A mixed methods approach that leverages extant data (attendance numbers, student artifacts), surveys, focus groups, making session feedback forms, observations, and field notes will together be used to assess youth and university student participant outcomes. The project will disaggregate data based on gender, race/ethnicity, grade level, and site to understand the Mobile Making program's impact on youth participants at multiple levels across contexts. The project will further compare findings from different types of implementation sites (e.g., school vs. library), learner groups, (e.g., middle vs. upper elementary students), and facilitator groups (e.g., STEM majors vs. future teachers). This will enable the project to conduct cross-case comparisons between CSU campuses. Project research will also compare findings from urban and rural school sites as well as based on the modality of teaching and learning (e.g., in-person vs. online). The mobile making program activities, project research, and a toolkit for implementing a Mobile maker program will be widely disseminated to researchers, educators, and out-of-school programs.
The Harvard Museums of Science and Culture will improve the ability of middle school teachers to use museum-based digital resources to support classroom instruction aligned with state and national science standards. Working with advisory teachers from five collaborating school districts, the museum will co-create classroom activities, based on digital resources from its collections, along with associated teacher professional development programs at three sites across urban and rural Massachusetts. The project will provide schools with access to classroom-ready resources that successfully support student learning. Teachers will learn how to use these materials, integrate them into their teaching, and enhance their skills to teach science content and practice. External evaluators will assess the project's effectiveness by measuring teacher implementation of the digital resources in the classroom, requests for information and assistance, and changes in teachers' confidence and comfort levels.
For thousands of years, Native Hawaiian/Pacific Islander (NHPI) seafarers have successfully utilized systemic observation of their environment to traverse vast expanses of open ocean and thrive on the most remote islands on earth. Developing NHPI trust in the scientific enterprise requires building connections that bridge the values and concepts of 'ike kupuna (traditional knowledge) with scientific knowledge systems and contemporary technology. This project will develop and research a pop-up science exhibit that connects indigenous Hawaiian knowledge with contemporary Western science concepts. The exhibit will show how community knowledge (that is consistent with underlying scientific principles and natural laws) has informed innovation by indigenous peoples. This community-initiated and developed project will begin with a single pop-up exhibit designed to incorporate several hands-on culture-based STEM activities that integrate traditional and modern technologies. For example, the exhibit may cover indigenous systems of star navigation for ocean voyaging, systems of netting for food and water containers, or systems of home design with local and natural materials. This project seeks to develop preliminary evidence of the effectiveness of such an approach for supporting rural Hawaiian youths' STEM engagement, understanding, and personal connections to Native Hawaiian STEM knowledge. Findings from this pilot and feasibility study will inform the development of a larger pop-up science center grounded in indigenous Hawaiian STEM knowledge, and advance intellectual knowledge around culturally sustaining pedagogy by helping informal STEM education practitioners understand community initiated and developed STEM exhibits.
This pop-up science center pilot will be led by a local Hawaiian community organization, INPEACE, in collaboration with several local community members and other community-based organizations. The preliminary research will iteratively explore whether and how an existing Hawaiian culture-based framework can be used to design hands-on STEM exhibits to enhance rural learner engagement, depth of STEM knowledge, and connection to Native Hawaiian STEM knowledge. Research efforts led by Kamehameha Schools, which has a long history of conducting research from an indigenous worldview, will engage 120 learners from various rural communities across Hawaii, from which 40 will be pre-selected middle-school youth, and 80 individuals will be from public audiences of learners ages 12 and up. Through a series of observations, interviews, pre and post surveys with validated instruments, and focus groups, the research will probe: (1) The learners' thoughts on the science practice and its relevance to old and new Hawaii and modern society. (2) The level at which related STEM topics have been understood, and (3) The learners' perceptions about their connection to Native Hawaiian STEM knowledge. Results from this pilot study will inform a future pop-up science center development project, and add to the scarce literature on community-driven, culturally sustaining exhibition development.
This Pilots and Feasibility Studies project is funded by the NSF Advancing Informal STEM Learning program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
The employment demands in STEM fields grew twice as fast as employment in non-STEM fields in the last decade, making it a matter of national importance to educate the next generation about science, engineering and the scientific process. The need to educate students about STEM is particularly pronounced in low-income, rural communities where: i) students may perceive that STEM learning has little relevance to their lives; ii) there are little, if any, STEM-related resources and infrastructure available at their schools or in their immediate areas; and iii) STEM teachers, usually one per school, often teach out of their area expertise, and lack a network from which they can learn and with which they can share experiences. Through the proposed project, middle school teachers in low-income, rural communities will partner with Dartmouth faculty and graduate students and professional science educators at the Montshire Museum of Science to develop sustainable STEM curricular units for their schools. These crosscutting units will include a series of hands-on, investigative, active learning, and standards-aligned lessons based in part on engineering design principles that may be used annually for the betterment of student learning. Once developed and tested in a classroom setting in our four pilot schools, the units will be made available to other partner schools in NH and VT and finally to any school wishing to adopt them. In addition, A STEM rural educator network, through which crosscutting units may be disseminated and teachers may share and support each other, will be created to enhance the teachers’ ability to network, seek advice, share information, etc.
The NIH Science Education Partnership Award (SEPA) program of Emory University endeavors to use an over-arching theme of citizen science principles to:
develop an innovative curriculum based on citizen science and experiential learning to evaluate the efficacy of informal science education in after-school settings;
promote biomedical scientific careers in under-represented groups targeting females for Girls for Science summer research experiences;
train teachers in Title I schools to implement this citizen science based curriculum; and
disseminate the citizen science principles through outreach.
This novel, experiential science and engineering program, termed Experiential Citizen Science Training for the Next Generation (ExCiTNG), encompasses community-identified topics reflecting NIH research priorities. The curriculum is mapped to Next Generation Science Standards.
A comprehensive evaluation plan accompanies each program component, composed of short- and/or longer-term outcome measures. We will use our existing outreach program (Students for Science) along with scientific community partnerships (Atlanta Science Festival) to implement key aspects of the program throughout the state of Georgia. These efforts will be overseen by a central Steering Committee composed of leadership of the Community Education Research Program of the Emory/Morehouse/Georgia Institute of Technology Atlanta Clinical Translational Science Institute (NIH CTSA), the Principal Investigators, representatives of each program component, and an independent K–12 STEM evaluator from the Georgia Department of Education.
The Community Advisory Board, including educators, parents, and community members, will help guide the program’s implementation and monitor progress. A committee of NIH-funded investigators, representing multiple NIH institutes along with experienced science writers, will lead the effort for dissemination and assure that on-going and new NIH research priorities are integrated into the program’s curriculum over time.
This project will examine the characteristics and outcomes of a large sample of environmental education field trip programs for youth to elucidate program characteristics that most powerfully influence 21st century learning outcomes. Environmental education programs for youth, particularly day-long school trip programs, are popular and reside at the intersection of formal and informal STEM education. Such field trips provide opportunities for diverse audiences to participate in shared learning experiences, but current understanding of what leads to success in these programs is limited. This large-scale study will address this gap in knowledge by investigating the linkages between program characteristics and participant outcomes for at least 800 single-day environmental education field trip programs for youth in grades 5-8, particularly programs for diverse and underserved audiences. This study will result in the identification of evidence-based practices that will inform future program design for a wide variety of settings, including nature centers, national parks, zoos, museums, aquaria, and other locations providing informal environmental education programs.
This Research in Service to Practice study is guided by two research questions: 1) What program characteristics (context, design, and delivery) most powerfully influence learner self-determination and learner outcomes? And 2) Do the most influential program characteristics differ across diverse and underserved audiences (e.g. African American, Hispanic/Latino, economically disadvantaged) and contexts (e.g. rural versus urban)? This project will examine a wide range of program-related factors, including pedagogical approaches and contextual characteristics. A valid and reliable protocol for observing 78 program characteristics hypothesized to influence learner outcomes developed by a previous project will be used to systematically sample and observe 500 single-day environmental education field trip programs for youth in grades 5-8 distributed across at least 40 U.S. states and territories. Programs for diverse and underserved youth will be emphasized, and a diverse set of programs in terms of program type and context will be sought. Data from this sample will be combined with those of an existing sample of 334 programs provided by over 90 providers. The final combined sample of over 800 programs will provide sufficient statistical power to confidently identify which program components are most consistently linked with learning outcomes. This sample size will also enable stratification of the sample for examination of these relationships within relevant subpopulations. Principal component analyses will be used to reduce data in theoretically meaningful and statistically valid ways, and multilevel structural equation modeling will be employed to examine the influences of both participants' individual characteristics and program and context characteristics on participant outcomes. Since one research question focuses on whether program outcomes are the same across different audiences, the project will include at least 200 programs for each of three specific audiences to ensure sufficient statistical power for confidence in the results: primarily African American, primarily Hispanic/Latino, and primarily White.
This project is funded by the National Science Foundation's (NSF's) Advancing Informal STEM Learning (AISL) program, which supports innovative research, approaches, and resources for use in a variety of learning settings.
DATE:
-
TEAM MEMBERS:
Robert PowellMarc SternBrandon Frensley