Each year, the National Alliance for Broader Impacts (NABI) seeks to understand the current state of broader impacts (BI) in the national context. In 2017, NABI convened two forums to identify needs and solicit recommendations. The first event was a 90-minute town hall session with 120 participants facilitated by Jamie Bell at the April 2017 BI Summit. The second event was a two-day convening facilitated by Goose Creek Consulting at NSF headquarters in May 2017 of stakeholder groups including university administrators (e.g., provosts, associate provosts, vice-presidents of research)
In March of 2016, the Exploratorium transmitted a live webcast of a total solar eclipse from Woleai, a remote island in the southwestern Pacific. The webcast reached over 1 million viewers. Evaluation reveals effective use of digital media to engage learners in solar science and related STEM content.
Edu, Inc. conducted an external evaluation study that shows clear and consistent evidence of broad distribution of STEM content through multiple online channels, social media, pre-produced videos, and an app for mobile devices. IBM Watson did a deep analysis of tweets on eclipse topics that
As part of a grant from the National Science Foundation, the National Federation of the Blind (NFB) developed, implemented, and evaluated the National Center for Blind Youth in Science (NCBYS), a three-year full-scale development project to increase informal learning opportunities for blind youth in STEM. Through this grant, the NCBYS extended opportunities for informal science learning for the direct benefit of blind students by conducting six NFB STEM2U regional programs included programs for blind youth, their parents/caregivers, blind teen mentors (apprentices), and museum educators.
Staff facilitators in museums and science centers are a critical but often overlooked component of the visitor experience. Despite assertions about the important role they play in visitor learning, there continues to be almost no research to understand staff facilitation in these settings or identify effective practices. To address these gaps, we conducted a design-based research study to describe the work of experienced museum educators and iteratively refine a model of staff facilitation to support family learning at interactive math exhibits developed through a prior project. The resulting
This checklist identifies and describes the elements of an evaluation report. It is intended to serve as a flexible guide for determining an evaluation report’s content. It should not be treated as a rigid set of requirements. An evaluation client’s or sponsor’s reporting requirements should take precedence over the checklist’s recommendations. Decisions about the order of content and level of detail in a report should be made with consideration of the audience’s information needs and priorities.
This checklist is strictly focused on the content of long-form technical evaluation reports
EvaluATE is a national resource center dedicated to supporting and improving the evaluation practices of approximately 250 ATE grantees across the country. EvaluATE conducts webinars and workshops, publishes a quarterly newsletter, maintains a website with a digital resource library, develops materials to guide evaluation work, and conducts an annual survey of ATE grantees. EvaluATE's mission is to promote the goals of the ATE program by partnering with projects and centers to strengthen the program's evaluation knowledge base, expand the use of exemplary evaluation practices, and support the continuous improvement of technician education throughout the nation. EvaluATE's goals associated with this proposal are to: (1) Ensure that all ATE Principal Investigators and evaluators know the essential elements of a credible and useful evaluation; (2) Maintain a comprehensive collection of online resources for ATE evaluation; (3) Strengthen and expand the network of ATE evaluation stakeholders; and (4) Gather, synthesize, and disseminate data about the ATE program activities to advance knowledge about ATE/technician education. The Center plans to produce a comprehensive set of evaluation resources to complement other services, engaging several community college-based Principal Investigators and evaluators in that process.
EvaluATE's products are informed by current research on evaluation, the National Science Foundation's priorities for the evaluation of ATE grants, and the needs of ATE PIs and evaluators for sound guidance that is immediately relevant and usable in their contexts. The fundamental nature of EvaluATE's work is geared toward supporting ATE grantees to use evaluation regularly to improve their work and demonstrate their impacts. All of EvaluATE's products are available to the public. EvaluATE's findings from the annual survey of ATE grantees aid in advancing understanding of the status of technician education and illuminate areas for additional research. The new survey investigates ATE grantees' work to serve underrepresented and special populations, including women, people of color, and veterans. Survey data are available upon request for research and evaluation purposes.
DATE:
-
TEAM MEMBERS:
Lori WingateArlen GullicksonEmma PerkKelly RobertsonLyssa Becho
resourceprojectProfessional Development, Conferences, and Networks
As higher education institutions (HEIs) work to enhance Broader Impacts (BI) efforts, collaborations with informal science education institutions (ISEs) (e.g. science centers, aquaria, zoos) can help them strengthen their impact and reach broader audiences. This project builds on the successful Portal to the Public (PoP) framework, bringing together the expertise and resources of HEIs and ISEs around the shared mission of engaging public audiences in current STEM research. The project is designed to address several critical needs: (1) Public outreach BI activities are relatively uncommon compared to BI that is focused within the infrastructure of academia; (2) Because collaborations with ISEs are frequently tied to individual Principal Investigators (PIs), there is limited opportunity to build a body of knowledge around the practice of partnering for BI work; and (3) Collaborations are often transient, making it more difficult for universities to view BI on an institutional level in ways that leverage particular institutional assets or strategies and even link investigators from multiple projects. The specific areas of study are: a. Develop and test a structure for education/outreach BI experience design that addresses a public audience need and meets NSF's BI criterion: The project will create disseminatable tools around the activity design process (including evaluation of learning impacts). By convening cross-disciplinary teams, the project will ensure that experiences will reflect a wide range of expertise and will help meet the needs of multiple stakeholders. These established structures will lower the barrier to entry for PIs who want to do public outreach BI. b. Design, test, and study structures for long-term, mutually beneficial HEI-ISE partnerships: The project will build on the proven PoP model to create flexible, disseminatable tools around the development of institutional partnerships at three collaborating HEI-ISE site pairings that consider each institution's resources, constraints and strategic goals, including a cross-institutional and cross-disciplinary Broader Impacts Design (BID) Team structure. Sustained partnerships will support ongoing public engagement with current STEM research. c. Anchor the partnership at the HEI with a representative from an office of research support: Research support professionals will be a core part of the BID Team and will help support institutional strategies for aligning BI activities with broader goals around community engagement. d. Study the culture of HEI-ISE partnerships, building knowledge about how these institutions can form effective, sustained and mutually beneficial collaborations. Project partners include Pacific Science Center with the University of Washington, Bothell, WA; University of Wisconsin-Madison with the Wisconsin Institute for Discovery; and the Sciencenter with Cornell University, Ithaca, NY. In addition, the Center for Research in Lifelong Learning, Oregon State University will oversee the research aspects of the project. The project's primary benefit is the development of more effective mechanisms for HEIs and ISEs to collaborate, that will better enable them to engage their communities in experiences and conversations about current STEM research and innovation. This project is being funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments.
One way to encourage youth to pursue training in the STEM fields and enter the STEM workforce is to foster interest and engagement in STEM during adolescence. Informal STEM Learning Sites (ISLS) provide opportunities for building interest and engagement in the STEM fields through a multitude of avenues, including the programming that they provide for youth, particularly teens. Frequently, ISLS provide opportunities to participate in volunteer programs, internships or work, which allow teens both to learn relevant STEM knowledge as well as to share that knowledge with others through opportunities to serve as youth educators. While youth educator programs provide rich contexts for teens to engage as both learners and teachers in these informal STEM environments, research to date has not yet identified the relationship between serving as youth educators and STEM engagement. Thus, the goal of this project is to document the impact of youth educators on visitor learning in ISLS and to identify best practices for implementing youth educator programs. The project studies STEM interests and engagement in the youth participants and the visitors that they interact with at six different ISLS in the US and UK. This project is funded through Science Learning+, which is an international partnership between the National Science Foundation (NSF) and the Wellcome Trust with the UK Economic and Social Research Council. The goal of this joint funding effort is to make transformational steps toward improving the knowledge base and practices of informal STEM experiences. Within NSF, Science Learning+ is part of the Advancing Informal STEM Learning (AISL) program that seeks to enhance learning in informal environments and to broaden access to and engagement in STEM learning experiences.
This project examines youth educator experiences related to STEM identity, educational aspirations, and motivation. The project also identifies outcomes that the youth educators have on visitors to ISLS in terms of knowledge, interest, and engagement in STEM. The specific aims are: 1) Outcomes for Teens - To measure the longitudinal impact of participation in an extended youth educator experience in an ISLS; 2) Outcomes for Visitors - To compare visitor engagement with and learning from exhibits in ISLS when they interact with a youth educator, relative to outcomes of interacting with an adult educator or no educator; and 3) Outcomes Across Demographics and STEM Sites - To examine differences in visitor engagement based on participant characteristics such as socio-economic status (SES), age, gender, and ethnicity and to compare outcomes of youth educator experiences across different types of ISLS. This research, which draws on expectancy value theory and social cognitive theory, will follow youth participants longitudinally over the course of 5 years and use latent variable analyses to understand the impact on the youth educators as well as the visitors with whom they interact. Importantly, the results of this research will be used to develop best practices for implementing youth educator programs in ISLS and the results will be disseminated to both academic and practice-based communities.
This project has clear and measurable broader impacts in a variety of ways. First, the project provides guidance to improve programming for youth in ISLS, including both the sites involved directly in the research and to the larger community of ISLS through evaluation, development, and dissemination of best practices. Additionally, this project provides rigorous, research-based evidence to identify and describe the outcomes of youth educator programs. This study directly benefits the participants of the research, both the visiting public and the youth educators, through opportunities to engage with science. The findings speak to issues of access and inclusivity in ISLS, providing insight into how to design environments that are welcoming and accessible for diverse groups of learners. Finally, this project provides evidence for best practices for ISLS in developing programs for youth that will lead to interest in and pursuit of STEM careers by members of underrepresented groups.
Northern ecosystems are rapidly changing; so too are the learning and information needs of Arctic and sub-Arctic communities who depend on these ecosystems for wild harvested foods. Public Participation in Scientific Research (PPSR) presents a possible method to increase flow of scientific and local knowledge, enhance STEM-based problem solving skills, and co-create new knowledge about phenology at local and regional or larger scales. However, there remain some key challenges that the field of PPSR research must address to achieve this goal. The proposed research will make substantial contributions to two of these issues by: 1) advancing theory on the interactions between PPSR and resilience in social-ecological systems, and 2) advancing our understanding of strategies to increase the engagement of youth and adults historically underrepresented in STEM, including Alaska Native and indigenous youth and their families who play an essential role in the sustainability of environmental monitoring in the high latitudes and rural locations throughout the globe. In particular, our project results will assist practitioners in choosing and investing in design elements of PPSR projects to better navigate the trade-offs between large-scale scientific outcomes and local cultural relevance. The data collected across the citizen science network will also advance scientific knowledge on the effects of phenological changes on berry availability to people and other animals.
The Arctic Harvest research goals are to 1) critically examine the relationship between PPSR learning outcomes in informal science environments and attributes of social-ecological resilience and 2) assess the impact of two program design elements (level of support and interaction with mentors and scientists, and an innovative story-based delivery method) on the engagement of underserved audiences. In partnership with afterschool clubs in urban and rural Alaska, we will assess the impact of participation in Winterberry, a new PPSR project that investigates the effect of changes in the timing of the seasons on subsistence berry resources. We propose to investigate individual and community-level learning outcomes expected to influence the ability for communities to adapt to climate change impacts, including attributes of engagement, higher-order thinking skills, and their influence on the level of civic action and interest in berry resource stewardship by the youth groups. Using both quantitative and qualitative approaches, we compare these outcomes with the same citizen science program delivered through two alternate methods: 1) a highly supported delivery method with increased in-person interaction with program mentors and scientists, and 2) an innovative method that weaves in storytelling based on elder experiences, youth observations, and citizen science data at all stages of the program learning cycle. This project is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants. The project also has support from the Office of Polar Programs.
DATE:
-
TEAM MEMBERS:
Katie SpellmanElena SparrowChrista MulderDeb Jones
This 4-year project addresses fundamental equity issues in informal Science, Technology, Engineering and Mathematics (STEM) learning. Access to, and opportunities within informal STEM learning (ISL) remain limited for youth from historically underrepresented backgrounds in both the United States and the United Kingdom. However, there is evidence that ISL experiences can expand opportunities for youth learning and development in STEM, for instance, increase positive attitudes towards educational aspirations and future careers/pursuits, improve grades and test scores in school settings, and decrease disciplinary action and dropout rates. Through research and development, this project brings together researchers and practitioners to focus on the experiences, practices and tools that will support equitable youth pathways into STEM. Working across conceptual frameworks and ISL settings (e.g. science centers, community groups, zoos) and universities in four urban contexts in two different nations, the partnership will produce a coherent knowledge base that strengthens and expands research plus practice partnerships, builds capacity towards transformative research and development, and develops new models and tools in support of equitable pathways into STEM at a global level. This project is funded through Science Learning+, which is an international partnership between the National Science Foundation (NSF) and the Wellcome Trust with the UK Economic and Social Research Council. The goal of this joint funding effort is to make transformational steps toward improving the knowledge base and practices of informal STEM experiences. Within NSF, Science Learning+ is part of the Advancing Informal STEM Learning (AISL) program that seeks to enhance learning in informal environments and to broaden access to and engagement in STEM learning experiences.
This Equity Pathways project responds to three challenges at the intersections of ISL research and practice in the United States and the United Kingdom: 1) lack of shared understanding of how youth from historically underrepresented backgrounds perceive and experience ISL opportunities across national contexts, and the practices and tools needed to support empowered movement through ISL; 2) limited shared understanding and evidence of core high-leverage practices that support such youth in progressing within and across ISL, and 3) limited understanding of how ISL might be equitable and transformative for such youth seeking to develop their own pathways into STEM. The major goal of this Partnership is for practitioners and researchers, working with youth through design-based implementation research, survey and critical ethnography, to develop new understandings of how and under what conditions they participate in ISL over time and across settings, and how they may connect these experiences towards pathways into STEM. The project will result in: 1) New understandings of ISL pathways that are equitable and transformative for youth from historically underrepresented backgrounds; 2) A set of high leverage practices and tools that support equitable and transformative informal science learning pathways (and the agency youth need to make their way through them); and 3) Strengthened and increased professional capacity to broaden participation among youth from historically underrepresented backgrounds in STEM through informal science learning. The project will be carried out by research + practice partnerships in 4 cities: London & Bristol, UK and Lansing, MI & Portland, OR, US, involving university researchers (University College London, Michigan State University, Oregon State University/Institute for Learning Innovation) practitioners in science museums (@Bristol Science Centre, Brent Lodge Park Animal Centre, Impressions 5, Oregon Museum of Science & Industry) and community-based centers (STEMettes, Knowle West Media Centre, Boys & Girls Clubs of Lansing, and Girls, Inc. of the Pacific Northwest).
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program funds innovative research, approaches, and resources for use in a variety of settings. The proposed project broadens the utility of Public Participation in Scientific Research (PPSR) approaches, which include citizen science, to support new angles in informal learning. It also extends previous work on interactive data visualizations in museums to encompass an element of active contribution to scientific data. To achieve these goals, this project will develop and research U!Scientist (pronounced `You, Scientist!')--a novel approach to using citizen science and learning research-based technology to engage museum visitors in learning about the process of science, shaping attitudes towards science, and science identity development. Through the U!Scientist multi-touch tabletop exhibit, visitors will: (1) interact with scientific data, (2) provide interpretations of data for direct use by scientists, (3) make statements based on evidence, and (4) visualize how their data classifications contribute to globe-spanning research projects. Visitors will also get to experience the process of science, gaining efficacy and confidence through these carefully designed interactions. This project brings together Zooniverse, experts in interactive design and learning based on large data visualizations in museums, and leaders in visitor experience and learning in science museums. Over fifty thousand museum visitors are expected to interact annually with U!Scientist through this effort. This impact will be multiplied by packaging the open-source platform so that others can easily instantiate U!Scientist at their institution.
The U!Scientist exhibit development process will follow rapid iterations of design, implementation, and revision driven by evaluation of experiences with museum visitors. It will involve close collaboration between specialists in computer science, human-computer interaction and educational design, informal science learning experts, and museum practitioners. The summative evaluation will be based on shadowing observations, U!Scientist and Zooniverse.org logfiles (i.e., automated collection of user behavior metrics), and surveys. Three key questions will be addressed through this effort: Q1) Will visitors participate in PPSR activities (via the U!Scientist touch table exhibit) on the museum floor, despite all the distractions and other learning opportunities competing for their attention? If so, who engages, for how long, and in what group configurations? Q2) If visitors do participate, will they re-engage with the content after the museum visit (i.e., continue on to Zooniverse.org)? Q3) Does engaging in PPSR via the touch table exhibit--with or without continued engagement in Zooniverse.org after the museum visit--lead to learning gains, improved understanding of the nature of science, improved attitudes towards science, and/or science identity development?
Glaciers around the world are undergoing dramatic changes. Many people, however, have a limited understanding of the scope of these changes because they are geographically distant and difficult to visualize. Although both digital learning tools and online scientific data repositories have greatly expanded over the last decade, there is currently no interface that brings the two together in a way that allows the public to explore these rapidly changing glacial environments. Therefore, to both improve public understanding and provide greater access to already existing resources, the project team will develop the Virtual Ice Explorer to encourage informal learning about glacial environments. This web application will feature an immersive virtual environment and display a suite of environmental data for an array of Earth's glacial systems. An interactive globe will allow users to select from a collection of sites ranging from polar regions to tropical latitudes. Each featured site will offer users an opportunity to interact with (1) a 3D rendering of the landscape; (2) a local map of the site; (3) historical and contemporary photographs of the site; (4) background information text describing the location, past research, and climate impacts; and (5) available environmental data. One of the most original features of the application will be its realistic, immersive 3D rendering of glacial landscapes by combining very high-resolution digital elevation models and satellite imagery with the application's built-in capabilities for immersive virtual environments. Although immersive environments often require expensive equipment, we are maximizing accessibility by developing the Virtual Ice Explorer to run in a web browser and function across various devices. Thus, the application will be available to anyone with internet access, and they can explore at their own pace.
As part of the successful development of Virtual Ice Explorer, the project team will create a platform for digital elevation models to be visualized and explored in 3D by users within the web application; curate digital elevation models, maps, images, text, and environmental data for inclusion in the web application for up to 11 geographically diverse glaciers/glacial landscapes; iteratively user-test the web application with project partners; and design the architecture of the system to readily scale to a larger collection of glaciers/glacial landscapes. To extend dissemination of the final products, the team has partnered with the U.S. Geologic Survey to showcase four benchmark glaciers in their long-term Glaciers and Climate project. In addition to improving understanding of glacier systems in informal learning environments, the project team will explore applications for spatial learning, employment of 3D environments for educational interventions, and use of Virtual Ice Explorer in formal learning environments. This project is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants. The project also has support from the Office of Polar Programs.
DATE:
-
TEAM MEMBERS:
Jason CervenecJesse FoxJulien Nicolas