The independent evaluation team subsequently undertook a formative evaluation to provide the production team with feedback on issues that arose from the front-end evaluation findings and from tpt’s early production work on the first Season Four episode and STEM role model videos.
As part of the development work of Latina SciGirls, the independent evaluation firm Knight Williams Inc. conducted a front-end evaluation focused on gathering input from the project’s primary public audiences (Latina girls and their parents/guardians) and professional audiences (the project’s advisers and partners).
Appendix includes logic model.
Amid calls from scientific leaders for their colleagues to become more effective public communicators, this study examines the objectives that scientists’ report drive their public engagement behaviors. We explore how scientists evaluate five specific communication objectives, which include informing the public about science, exciting the public about science, strengthening the public’s trust in science, tailoring messages about science, and defending science from misinformation. We use insights from extant research, the theory of planned behavior, and procedural justice theory to identify
It’s important to communicate the excitement and value of NSF-funded research. This tool (formatted as a Prezi presentation) helps you do that with assistance from NSF public affairs experts, exploring options for communicating your research and broader impacts.
DESIGN SQUAD GLOBAL (DSG) is a web-based, mobile-accessible digital hub and outreach initiative that creates new opportunities to empower middle school youth to solve real-world problems and understand the impact of engineering in a global context. The project builds on the Design Squad model for engineering education (including a television show and website). The ultimate goals of Design Squad Global are to: (1) develop innovative ways to incorporate effective engineering education into informal learning environments; (2) inform the field about promising practices in cross-cultural
To explore the role and impact of The Innovation Lab at Youth Radio, Rockman et al, an independent research and evaluation organization, conducted an external evaluation of the project. With funding from the National Science Foundation’s Advancing Informal STEM Learning (AISL) program, the Innovation Lab sought to develop and research a scalable, evidence-informed theory of action to engage underrepresented youth in Science, Technology, Engineering, and Mathematics (STEM) learning through the collaborative creation and dissemination of original journalistic media, technology, and curriculum
From October 2015 to July 2016, Smart Start Evaluation & Research conducted a formative evaluation of a full-length film festival cut of the film The Anthropologist in order to provide feedback that improves decision-making capacity for the film’s final edit, production design, and distribution strategies in line with the project’s goals (see p. 5).
SmartStart conducted focus group interviews with participants to 1) measure their initial impressions of the film and its content, and 2) provide feedback on content, edits, target audience, website, and ideas for distribution. SmartStart then
The Anthropologist examines climate change like no other film before. The fate of the planet is considered from the perspective of American teenager Katie Crate. Over the course of five years, she travels alongside her mother Susie, an anthropologist studying the impact of climate change on indigenous communities. Their journey parallels that of renowned anthropologist Margaret Mead, who for decades sought to understand how global change affects remote cultures.
From January 2012 to May 2012, SmartStart Educational Consulting Services conducted a front-end evaluation of the documentary
Co-led by the University of Washington and Science Gallery Dublin, this project aims to drive and transform the next generation of broadening participation efforts targeting teen-aged youth from communities historically underrepresented in STEM fields. This project investigates how out-of-school time (OST) programs that integrate epistemic practices of the arts, sciences, computer science, and other disciplines, in the context of consequential activities (such as creating radio segments, designing museum exhibitions, or building online games), can more broadly appeal to and engage youth who do not already identify as STEM learners. STEM-related skills and capacities (such as computational thinking, design, data visualizations, and digital storytelling) are key to productive and creative participation in many future civic and workplace activities, and are driving the 30 fastest-growing occupations in the US. But many new jobs will entail a hybrid blend of skills, such as programming and design skills that many students who have disengaged with academic STEM pathways may already have and would be eager to develop further. There is not currently a strong foundation of research-based evidence to guide the design, implementation, and evaluation transdisciplinary programs - in which STEM skills are embedded as tools for meaningful participation - or how such approaches relate to long-term outcomes. Hypothesizing that OST programs which effectively engage youth during their high-leverage teenage years can significantly impact youths' longer-term STEM learning trajectories, this project will involve: 1) Five 3-year studies documenting learning in different technology-rich contexts: Making Afterschool, Media Production, Museum Exhibition Design, Digital Arts Programs, and Pop-Up/Street Science Programs; 2) A 4-year longitudinal study, involving 100 youth from the above programs; 3) The creation of a number of practical measurement tools that can be used to monitor how programs are leveraging the intersections of the arts and sciences to support student engagement and learning; and 4) A Professional Development program conducted at informal science education conferences in the EU and US to engage the informal STEM field with emerging findings. This project is funded through Science Learning+, which is an international partnership between the National Science Foundation (NSF) and the Wellcome Trust with the UK Economic and Social Research Council. The goal of this joint funding effort is to make transformational steps toward improving the knowledge base and practices of informal STEM experiences to better understand, strengthen, and coordinate STEM engagement and learning. Within NSF, Science Learning+ is part of the Advancing Informal STEM Learning (AISL) program that seeks to enhance learning in informal environments.
Transdisciplinary, equity-oriented OST programs can provide supportive social contexts in which STEM concepts and practices are taken up as the means for meaningful participation in valued activities, building students' STEM skills in ways that can propel their future academic, career, and lifelong learning choices. This project will build the knowledge base about these emerging 21st century transdisciplinary approaches to broadening participation investigating: 1) The epistemic intersections across a range of disciplines (art, science, computation, design) that operate to broaden appeal and meaningful participation for underrepresented youth; 2) How transdisciplinary activities undertaken in the context of consequential learning (e.g., producing a radio segment, designing an exhibition for the general public) can illuminate the relevance of STEM to young people's lives, concerns, and futures; and 3) How participation in such programs can propel students' longer-term life choices and STEM learning trajectories. The project is a collaboration of the University of Washington, Science Gallery Dublin, Indiana University, Youth Radio in Oakland California, Guerilla Science in New York and London, and the London School of Economics.
Public Participation in Scientific Research (PPSR), often referred to as crowdsourcing or citizen science, engages participants in authentic research, which both advances science discovery as well as increases the potential for participants' understanding and use of science in their lives and careers. This four year research project examines youth participation in PPSR projects that are facilitated by Natural History Museums (NHMs). NHMs, like PPSR, have a dual focus on scientific research and science, technology, engineering, and mathematics (STEM) education. The NHMs in this project have established in-person and online PPSR programs and have close ties with local urban community-based organizations. Together, these traits make NHMs appropriate informal learning settings to study how young people participate in PPSR and what they learn. This study focuses on three types of PPSR experiences: short-term outdoor events like bioblitzes, long-term outdoor environmental monitoring projects, and online PPSR projects such as crowdsourcing the ID of field observations. The findings of this study will be shared through PPSR networks as well as throughout the field in informal STEM learning in order to strength youth programming in STEM, such that youth are empowered to engage in STEM research and activities in their communities. This project is funded through Science Learning+, which is an international partnership between the National Science Foundation (NSF) and the Wellcome Trust with the UK Economic and Social Research Council. The goal of this joint funding effort is to make transformational steps toward improving the knowledge base and practices of informal STEM experiences. Within NSF, Science Learning+ is part of the Advancing Informal STEM Learning (AISL) program that seeks to enhance learning in informal environments and to broaden access to and engagement in STEM learning experiences.
The study employs observations, surveys, interviews, and learning analytics to explore three overarching questions about youth learning: 1) What is the nature of the learning environments and what activities do youth engage in when participating in NHM-led PPSR? 2) To what extent do youth develop three science learning outcomes, through participation in NHM-led citizen science programs? The three are: a) An understanding of the science content, b) identification of roles for themselves in the practice of science, and c) a sense of agency for taking actions using science? 3) What program features and settings in NHM-led PPSR foster these three science learning outcomes among youth? Based on studies occurring at multiple NHMs in the US and the UK, the broader impact of this study includes providing research-based recommendations for NHM practitioners that will help make PPSR projects and learning science more accessible and productive for youth. This project is collaboration between education researchers at University of California, Davis and Open University (UK), and Oxford University (UK) and citizen science practitioners, educators, and environmental scientists at three NHMs in the US and UK: NHM London, California Academy of Sciences, and NHM Los Angeles.
Produced by the Climate Change Education Partnership (CCEP) Alliance, the "Climate Change Education: Effective Practices for Working with Educators, Scientists, Decision Makers and the Public" guide provides recommendations for effective education and communication practices when working with different types of audiences. While effective education has been traditionally defined as the acquisition of knowledge, Alliance programs maintain a broader definition of “effective” to include the acquisition and use of climate change knowledge to inform decision-making.
The CCEP Alliance is supported
The most important consideration in evaluating chemistry outreach efforts is how to best use the evaluation to serve project needs. Evaluation should be about making programs more effective—at communicating ideas, changing attitudes, inspiring action, or reaching wider audiences, for example. A well-conducted evaluation typically contributes to the quality of a project by helping its leaders better define their goals, identify important milestones and indicators of success, and use evidence to support ongoing improvements. At its best, evaluation is an integral part of project design and