This paper examines the differences and challenges encountered when trying to create informal blended (virtual and hands-on) engineering design STEM activities. It contrasts the creation of STEM activities for formal and informal learning environments, stressing that the differences extend far beyond the length of the activity or depth of any learning goals. The discussion begins with an examination of differences between the two learning environments that need to be taken into consideration. These differences include the physical environments, organizational structures, and the goals or
This paper discusses findings from the use of the Wise Guys and Gals (WGG) Observation of WGG Youth Protocol in a blended learning environment. The protocol was used to assess youth engagement when completing blended engineering design challenges at two Boys and Girls Clubs. WGG is a project funded through a grant from the National Science Foundation and which brings blended learning design challenges to middle school aged learners in informal STEM (science, technology, engineering, and mathematics) settings. This paper explores the feasibility of using the observation protocol to collect data
Education Development Center, Inc. (EDC) received a grant to develop a STEM Evaluation Community from the National Science Foundation (NSF). The primary goal of this project is to increase the capacity of evaluators to produce high quality, conceptually sound, methodologically appropriate evaluations of NSF programs and projects, specifically in the area of STEM education and outreach.
In response to a need for support for evaluation capacity, articulated by those who evaluate NSF programs and projects, the STEM Evaluation Community planned to connect evaluators across directorates
Across the United States, individuals, organizations, and communities are wrestling with a wide array of challenging and persistent science, technology, engineering, and mathematics (STEM)-related problems. A few examples include ensuring more equitable access to STEM careers; building capacity for rural libraries to support STEM learning; and supporting greater cyber literacy among youth. The good news is that thousands of individuals, organizations, and communities are coming up with great ideas for how to confront these problems; many of them supported by the NSF. Unfortunately, most will encounter significant roadblocks to success along the way, but not because of bad ideas. Most change agents falter along the lengthy and often convoluted pathway between idea and successful execution because they bump up against barriers they do not expect or know how to overcome. This Pilot and Feasibility Study will create Learning Solutions, a multi-platform program designed to support those people and entities engaged in work that cultivates the public's understanding of, engagement with, and interest in STEM fields and STEM-related information. First, the project will systematically identify the real, but often unspoken issues that individuals, organizations, and communities run into as they work to bring about significant and impactful STEM-related change. Then, the project will assemble, curate, and make digitally available a collection of tools, resources and strategies designed to help someone understand and resolve these kinds of issues if and when they arise. By better understanding the experiences of change agents, the challenges they face, and the creative learning solutions they enact, this project will ensure that more change agents successfully access the learning know-how they need, when they need it, in curated, easy-to-digest formats. This award is funded by the Advanced Informal STEM Learning program which contributes to STEM engagement and literacy, workforce development, and educational success via supporting new approaches to and evidence-based understanding of STEM learning in informal environments. Learning Solutions will build capacity and will help more professionals successfully bring more good ideas to fruition.
The target audience for this Pilot & Feasibility phase of Learning Solutions will be STEM professionals working at the intersection of STEM and society across diverse sectors. It will focus on change agents -- individuals who want to be or who already are engaged in community-based, action-oriented STEM-related change projects, whether acting on their own, within an organization, or as part of a broader community of organizations. To achieve the goal of making STEM-related change easier to accomplish, Learning Solutions will implement a multi-step process. With input from five Critical Advisors, 20 Key Informants, and ultimately hundreds of change agents, project staff will: 1) Utilize an iterative process of in-depth interviews and broadly disseminated surveys to identify the major understandings, skills and processes that current and past STEM-related change agents have experienced as impediments to their success; 2) Determine how best to describe and categorize these issues across diverse problem spaces; 3) Select twelve issues, based on which are the most frequently mentioned and/or perceived to be the most critical or challenging, and research and curate the best and most authoritative resources responsive to these dozen issues; and finally, 4) Use a variety of platforms (e.g., social media, traditional media, digital and in print publications, podcasts, panels, and group presentations) and utilization metrics to ensure effective digital delivery of potential solutions to the selected issues. By the project's end, we will have identified some of the key challenges the STEM-related change agents who work in communities across America regularly encounter, as well as the feasibility of developing a mechanism for helping those change agents discover preexisting and readily accessible resources to assist them in resolving those challenges.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
DATE:
-
TEAM MEMBERS:
John H FalkElysa CorinStacey Sheehan
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants. The goal of this pilot and feasibility study is to increase participation in informal STEM learning in rural Idaho through Stories of Fire, a program based on personal narratives of wildland fire. Idaho is a rural state, with an average population of just 19 people per square mile, the fourth lowest population density in the United States. The state is experiencing increasingly severe wildfire, and effective responses to such environmental change require a better understanding of the underlying science. Contextualizing science learning, making connections between everyday lives and a sense of place can engage learners and bring about a better understanding of wildfire. This project will bring together a science communicator, a narratologist, a fire ecologist, and a specialist on emotions and public lands. They will work collaboratively with informal educators based in rural areas of Idaho underrepresented in STEM fields. Rural areas are rich in knowledge based on years of cumulative observations, cultural beliefs, and practices shared through community networks. This project builds on these rural assets while addressing the challenges rural populations face. The project addresses broadening participation in STEM through narrative practices that encourage more diverse ways of knowing, being, and representing science.
This research study will explore: 1) what mechanisms of narrative (storytelling) most effectively integrate individuals? personal experiences and accurate STEM content in fire science communication, and 2) what audience-centered approaches best facilitate narrative approaches to informal STEM learning. This project engages four levels of participants over four phases of research and programming: 1) The research team will interview and analyze the narratives of 40 Frontliners (e.g., wildland firefighters and evacuees) from the inland Northwest region with first-hand experience with wildfire. 2) They will conduct a narrative workshop to train 20 informal STEM Educators from across the state on audience-centered approaches that facilitate participant storytelling about fire. 3) Educators will pilot their own narrative-based informal science learning programs with program participants in their rural home communities across the state, 4) A professional podcaster will create two podcasts modeled on our research findings for public audiences reached through media.
This Pilots and Feasibility Studies award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
DATE:
-
TEAM MEMBERS:
Teresa CohnLeda KobziarJennifer LadinoErin James
In informal science contexts, the word tinkering describes a learning process that combines art, science, and technology through hands-on inquiry. With the growth in popularity of the making and tinkering movements nationwide, these practices are increasingly making their way into early childhood environments where they have great promise to positively impact the early STEM learning experiences of young children. This 2-day conference hosted at the Exploratorium in San Francisco will bring together stakeholders exploring applications of tinkering in informal early childhood environments. The conference will provide opportunities to explore the role, value, and challenges associated with implementing meaningful tinkering interventions in learning environments serving young children. The project seeks to 1) Convene stakeholders from the tinkering and early childhood programs; and 2) further the exploration and evolution of practitioner and researcher knowledge about tinkering in early childhood contexts. The long-term goal is to support more young children being introduced to STEM learning through tinkering's adaptable approaches to STEM-learning that align with the developmental needs of this young population.
This project will collaboratively analyze and document the state of the field of STEM-rich tinkering in informal early childhood contexts. Additionally, the project will deepen relationships across the early childhood tinkering ecosystem. Additional outcomes include an effort to provide tangible resources to the field highlighting current promising practices and future opportunities for development. The conference will also provide an understanding of how tinkering interventions may contribute to the development of STEM interest, identity and learning amongst early childhood audiences. Finally, the conference will bring together research and practitioners to explore how tinkering in early childhood settings can be used effectively to meet the needs of diverse learners including learners from underserved and underrepresented communities. The project will recruit a total of 75 participants with backgrounds in the field of tinkering and STEM learning, early childhood research, and professional development practices representing a diverse set of institutions and organizations. Research questions for the conference will focus on: 1) What types of supports and professional development do early childhood educators need to facilitate early STEM learning through tinkering? 2) What types of built environment and hands-on materials best support young children's ability to learn STEM content and practices through tinkering? 3) What types of strategies best support caregiver involvement in young children's learning? 4) What is the role of early childhood tinkering in young children?s STEM learning, interest, and identity development? 5) How can culturally and linguistically sustaining pedagogies be used to ensure equity across a diversity of young learners and their families? To answer these research questions the project will use qualitative methods before, during and post-conference. Research methods will include a landscape analysis identifying needs of participants, surveys, observations and informal interviews with participants.
This Conference award is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
The Advancing Informal STEM Learning (AISL) program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. An ongoing challenge to the design of effective STEM learning exhibits for diverse young children is the absence of reliable and evidence-based resources that designers can apply to the design of STEM exhibits that draw upon play as a child's primary pedagogy, while simultaneously engaging children with STEM content and processes that support development of STEM skills such as observation. To address these challenges, the project team will use a collaborative process in which learning researchers and informal STEM practitioners iteratively develop, design, and test the STEM for Play Framework that could then be applied to the design of STEM-focused exhibits that support play and STEM skill use among early learners.
This Research in Service to Practice project will address these questions: 1) What is a framework for play in early STEM learning that is inclusive of children's cultural influences?; 2) To what extent do interactions between early learners (ages 3-8) and caregivers or peers at exhibits influence the structure and effectiveness of play for supporting STEM skill development?; 3) How do practitioners link play to STEM skill development, and to what extent does a framework for play in early STEM learning assist in identifying types of play that supports early STEM skill development?; and 4) What do practitioners identify as best practices in exhibit design that support the development of STEM skills for early childhood audiences, and conversely, to what extent do practitioners perceive specific aspects of the design as influential to play? The project team will address these questions across four phases of study that will include (a) development of a critical research synthesis to inform the initial STEM for Play framework; (b) the use of surveys, focus groups, and interviews to solicit feedback from practitioners; (c) testing and revising the framework by conducting structured observations of STEM exhibits at multiple museums. The project team will use multiple analytic approaches including qualitative thematic analyses as well as inferential statistics. Results will be disseminated to children?s museums, science centers, and research communities.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
A makerspace is a place where participants explore their own interests and learn by creating, tinkering, and inventing artifacts through the use of a rich variety of tools and materials. This project will develop and research a flexible model for makerspaces that can be adapted to local settings to support informal STEM learning for hospitalized, chronically ill patients in pediatric environments who are predominantly youth of color from low-income backgrounds. These youth are subject to health disparities and healthcare inequities. Their frequent absence from school and other activities disrupt friendship formations, reduce their opportunities for social support, reduce their access to environments where they can feel a sense of self-agency through learning and creative activities. Through patient centered co-design, this project will build adaptable STEM makerspace environments conducive to STEM-rich learning, the exercise of self-agency, and development of STEM identity. Project design will focus on the sensitive nature of working with vulnerable populations (i.e., immunocompromised patients). The project will develop and disseminate several resources: (1) a flexible makerspace model that can be adapted to work in different pediatric settings; (2) research methods for conducting research in highly sensitive environments with and alongside young patients; and (3) professional development resources and a playbook including guidebook and facilitators guide that will articulate principles and processes for designing, implementing and sustaining makerspaces in pediatric settings. These resources will be widely disseminated through maker and other informal STEM networks.
The project will pursue two innovations. First, the project will develop the physical design of adaptable informal STEM makerspaces in pediatric settings. Second, the project will develop innovative patient-centered methodologies for studying approaches to physical design and the effects of makerspace installations for informal STEM-learning, self-agency, and STEM identity development. Using a design-based research approach, the project will investigate: (1) the extent to which physical makerspace designs support access to material, relational, and ideational resources for STEM-learning and well-being; (2) the extent to which makerspace installations, researchers, and medical care staff support patients in accessing and generating tools and other resources for personal learning and a sense of agency; and (3) the extent to which makerspace design with a focus on affording material, relational, and ideational resources provide rich opportunities for young patients to explore their own interests and cultivate STEM identities. One of the project's innovations, beyond development of adaptable makerspace model involves developing an innovative patient-centered methodology for conducting educational research toward broadening participation in STEM in highly sensitive medical care environments. The project will employ a mixed-methods research design and collect a variety of data to address these areas of research including documentation of makerspace design plans and renderings, observational data gathered through fieldnotes, video and audio recordings, informal interviews with patients, their families, and child-care staff, and patient generated artifacts. Articles for researchers and practitioners will be submitted for publication to appropriate professional journals and peer-reviewed publications.
As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program funds innovative research, approaches, and resources for use in a variety of settings.
This Innovations in Development award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
While there is increased interest in youth-centered maker programs in informal educational contexts, scarce research-informed professional development exist that focus on how informal educators do or should plan and handle ongoing, just-in-time support during moments of failure. Prior research supports the important role of failure in maker programming to increase learning, resilience and other noncognitive skills such as self-efficacy and independence. The objective of this project is to address this gap through adapting, implementing, and refining a professional development program for informal educators to productively attend, interpret, and respond to youths’ experiences with failure while engaged in maker programs in informal learning contexts. In the first two years of the project, the research team will work closely with six partners to implement and refine the professional development model: The Tech Museum of Innovation, The Bakken Museum, Montshire Museum of Science, The Minneapolis Institute of Art, Thinkery, and Amazeum Children’s Museum. In the last year of the project, the team will scale-up the professional development model through partnering with an additional nine institutions implementing maker programming for youth. The professional development consists of two models. In the first model, we support one to two lead facilitators at each partnering institution through an initial three-day workshop and ongoing support meetings. In the second model, the lead facilitators support other informal educators at their institution implementing making programs for youth. This project will enhance the infrastructure for research and education as collaborations and professional learning communities will be established among a variety of informal learning institutions. The project will also demonstrate a link between research and institutional and societal benefits through shifting the connotation and perceptions of failure to be valued for its educational potential and to empower informal educators to support discomfort and struggle throughout maker programs with youth.
The three goals of this collaborative project are to (a) advance the field of informal education through a research-based professional development program specific to youths’ failures during maker programs; (b) support shifts in informal educators’ facilitation practices and perspectives around youth’s failure experiences, and (c) investigate the effects of the professional development on youths’ resilience and failure mindset. The iterative nature of this project will be informed by the collection and analysis of video data of professional development sessions and informal educators facilitating maker programs, reflective journaling, surveys regarding the professional development, and pre-post surveys from youth engaged in the maker programs. Dissemination will address multiple stakeholders, including informal educators, program developers, evaluators, researchers, and public audiences.
This Innovations in Development project is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.
Over the past two decades the prevalence of Autism Spectrum Disorder (ASD) has nearly tripled and yet there is much to learn about serving this audience well. After high school exit, most are left to navigate the world without appropriate support or the requisite skills necessary for success. Educators working in informal science institutions (ISI) can better promote both social interaction and engagement in STEM education for individuals with ASD. A learning environment in which the learner chooses content aligned with their personal interests and where learning can be multifaceted (verbal, hands-on, fast or slow, social or solitary, directed or inquiry based, physical, etc.) is consistent with the central tenets of an evidence-based, outcome-driven approach for autism intervention. ISI educators have the desire but may not have sufficient and timely knowledge and skills to engage and support this audience. Currently, many are working at the local level to develop new programs and approaches for patrons with ASD, with little evaluation or research and not building on each other's work. The project will develop a rigorous customized professional learning experience designed to enhance capacity of ISIs broadly in ASD support techniques and strategies. The goal is to enable more inclusive opportunities for people with ASD based on current and emerging promising practices. The project's theory of action is that the ability of people with ASD to participate in traditional, mainstream experiences will improve their motivation to seek other similar opportunities, build interpersonal skills critical to successful interaction in society, formal education, and careers. This, in turn, will help individuals with ASD gain the skills and confidence needed to pursue STEM academically and professionally. The project is a collaboration between the Institute for Learning Innovation, the SciTech Institute, and the Southwest Autism Research & Resource Center (SARRC). This project is funded by the Advanced Informal STEM Learning program which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments.
The pilot's main research questions are: (1) To what degree does the professional learning program support the learning outcomes in knowledge, awareness, interest, skills, attitude, and behavior change in informal science education providers? and (2) What features of the program do educators consider most effective for improving their ability to serve this audience? Four Arizona ISIs will participate in a research-based design study; their staff will also comprise the founding members of a Community of Practice aimed at sharing promising practices and promoting broader engagement among the informal science education community. The professional development (PD) will be provided by SARRC. New formative evaluation skills will support ongoing innovation and build participant capacity. Leveraging this training, the ISIs will create and test new approaches and programs, apply new skills in formative evaluation, and develop internal workplace programs to create cultures of ASD understanding. A pilot research study will recruit 20 diverse individuals with ASD who will visit each institution prior to and after the PD for staff. The research will measure the degree to which the PD impacts attendee experience as well as assess the science learning that occurred because of their visit. This project will advance collaboration between ASD experts and ISI educators to iteratively develop effective museum learning strategies. Other goals of this work are to provide important insights into (a) the current state of accessibility programs in ISI venues nationally, (b) how PD can be leveraged to help institutions reach true inclusion, and (c) initial evidenced-based approaches for inclusion of individuals on the Autism spectrum in mainstream informal environments. In addition to the research findings, deliverables include an ASD PD model, national inventory of current practices and programs that support ASD learning and participation, and the establishment of a Community of Practice.
This Innovations in Development award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
The science museum field is only starting to look at ways of providing visitors with opportunities for the authentic observation of complex, real-time biological phenomenon. The project will develop and research a microscope-based exhibit with pedagogical scaffolding (i.e., helpful prompts) that responds to visitors' changing views as they explore live samples and biological processes. As part of its overall strategy to enhance learning in informal environments, the Advancing Informal STEM Learning (AISL) program funds innovative research, approaches, and resources for use in a variety of settings. Scientific observation is a systematic, complex practice, critical in the biological sciences where investigation is heavily reliant on visual data. Using techniques and equipment similar to what scientists themselves use, the exhibit will enable visitors to see and explore the complex, dynamic visual evidence that scientists themselves see. The exhibit will use new and more affordable high-resolution imaging technology and image analysis software to make microscopic images of living organisms visible. Armed with "smart" (i.e., computer-assisted) pedagogical scaffolding that supports inquiry, the project will develop exhibits that help informal learners bridge the gap between everyday observation and authentic scientific observation. The platform will incorporate strategies grounded in prior work on learning through observation that will be applicable to a range of biological content and live specimens. The project platform will be designed for use to a variety of informal science learning environments, including nature centers and mobile laboratories as well as interactive science centers. The project platform itself, including the microscope, related imaging, and learning technologies will be relatively inexpensive, bring it within reach of small science museum and schools. The exhibit will directly engage thousands of learners who visit the Exploratorium and will reach underserved audiences through partnerships with BioBus, a mobile unit that serves the New York City area, and the Noyo Center of Marine Science, a science museum that serves rural areas in Northern California.
The project will move beyond simulation and modeling of complex visual phenomena and provide learners with experiences using real visual evidence that can deeply engage them with the content and practice of biological science. By grounding the work in prior theoretical and empirical findings, project research will refine and broaden understanding of scaffolding strategies and their effect on informal science learning at exhibits. Project research will investigate how the project supports learners (1) asking productive questions (i.e., those answerable through observations) that are meaningful to them, (2) interpreting what they see, and (3) connecting their observations to biological concepts to build a more coherent understanding of the content and practice of biological disciplines. A series of comparative studies across and within venues, specimens, and content will assess engagement and scaffolding strategies, with a particular focus on appropriately integrating computational imaging techniques in a way that is responsive to the interests and needs of different venues' audiences. Project research will contribute important knowledge on ways to support informal learners who are engaged in authentic observation of biological phenomenon. Project research findings and technology resources will be widely shared with informal STEM researchers and practitioners concerned with engaging the public in current research in biology, as well as those interested in supporting observation in other disciplines (e.g., meteorology, ocean science, environmental science) that rely on an evidence base of live, dynamic, complex imagery.
This Innovations in Development award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
Various measures exist to assess learning outcomes from informal science, technology, engineering, or math (STEM) education programming, especially for youth audiences. Although there has been growing interest in the field among practitioners, researchers and evaluators for the development and implementation of shared, common STEM measures and frameworks, informal STEM learning (ISL) institutions (e.g., museums, zoos, aquaria, botanic gardens, and the like) face a unique challenge when they design and evaluate educational programs. By nature of the broad-ranging topic areas these institutions emphasize, programming seeks to reach audiences that vary in age, ability, cultural background, interest, and more. Participation also varies--for example, participants choose which in a series of exhibits they attend and how much time they spend at each. Given this planned-for variance, how do educators target appropriate learning outcomes? To what extent do outcomes map to activity complexity? How does programming success link to institutional goals? One solution may be a framework of outcome progressions. Progressions are within outcome categories (e.g., interest, attitude, knowledge, skills, behavior, science capital, etc.), the ordinal series of effects that can be mapped to programming that ranges from short duration with simple content (e.g. a hallway cart demonstration can trigger situational interest) to extended duration with complex content (e.g. a youth internship can generate well-developed personal interest). STEM educators, evaluators and researchers working in or with ISL institutions will come together to consider the utility and feasibility of an outcome progressions framework. With this framework STEM learning institution educators may be better equipped to understand the impact of their programming and how to design for it. This work will be undertaken by the COSI Center for Research and Evaluation in conjunction with the Office of Education, Outreach, and Visitor Services of the National Museum of Natural History and the PEAR Institute.
The primary aim of this multi-day virtual convening is to inform future development of the framework for outcome progressions. These progressions will reflect and accommodate outcomes targeted by existing and shared evaluation measures. The framework will provide informal educators and evaluators with (1) a way of aligning outcome expectations with programming depth and dosage; (2) outcome choices achievable with various depths of programming; (3) a strategic tool for planning institution-wide targeted outcomes across a full portfolio of institution educational programming; and (4) a tool for documenting an education department's varied outcomes across the full range of possible outcomes. To create rich and informed critical discussion, the conference will involve research consultant experts representing each outcome category; practitioners representing diverse programming; and evaluation theorists familiar with the broad needs of informal learning institutions. A conference subcommittee with expertise in culturally responsive and equitable program development and evaluation will ensure the agenda contains flexibility to generate emergent equitable and inclusive conversations. Conference participants will review and contribute to the soundness and utility of these categorizations as well as the progressions within them. These insights and inputs will inform future development of the framework as a valid and reliable tool. Deliverables include a paper summarizing the findings from the convenings and a related webinar that describes the framework and recommendations for its use. These will be shared broadly via the informalscience.org website and other professional venues.
This conference is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.