Various measures exist to assess learning outcomes from informal science, technology, engineering, or math (STEM) education programming, especially for youth audiences. Although there has been growing interest in the field among practitioners, researchers and evaluators for the development and implementation of shared, common STEM measures and frameworks, informal STEM learning (ISL) institutions (e.g., museums, zoos, aquaria, botanic gardens, and the like) face a unique challenge when they design and evaluate educational programs. By nature of the broad-ranging topic areas these institutions emphasize, programming seeks to reach audiences that vary in age, ability, cultural background, interest, and more. Participation also varies--for example, participants choose which in a series of exhibits they attend and how much time they spend at each. Given this planned-for variance, how do educators target appropriate learning outcomes? To what extent do outcomes map to activity complexity? How does programming success link to institutional goals? One solution may be a framework of outcome progressions. Progressions are within outcome categories (e.g., interest, attitude, knowledge, skills, behavior, science capital, etc.), the ordinal series of effects that can be mapped to programming that ranges from short duration with simple content (e.g. a hallway cart demonstration can trigger situational interest) to extended duration with complex content (e.g. a youth internship can generate well-developed personal interest). STEM educators, evaluators and researchers working in or with ISL institutions will come together to consider the utility and feasibility of an outcome progressions framework. With this framework STEM learning institution educators may be better equipped to understand the impact of their programming and how to design for it. This work will be undertaken by the COSI Center for Research and Evaluation in conjunction with the Office of Education, Outreach, and Visitor Services of the National Museum of Natural History and the PEAR Institute.
The primary aim of this multi-day virtual convening is to inform future development of the framework for outcome progressions. These progressions will reflect and accommodate outcomes targeted by existing and shared evaluation measures. The framework will provide informal educators and evaluators with (1) a way of aligning outcome expectations with programming depth and dosage; (2) outcome choices achievable with various depths of programming; (3) a strategic tool for planning institution-wide targeted outcomes across a full portfolio of institution educational programming; and (4) a tool for documenting an education department's varied outcomes across the full range of possible outcomes. To create rich and informed critical discussion, the conference will involve research consultant experts representing each outcome category; practitioners representing diverse programming; and evaluation theorists familiar with the broad needs of informal learning institutions. A conference subcommittee with expertise in culturally responsive and equitable program development and evaluation will ensure the agenda contains flexibility to generate emergent equitable and inclusive conversations. Conference participants will review and contribute to the soundness and utility of these categorizations as well as the progressions within them. These insights and inputs will inform future development of the framework as a valid and reliable tool. Deliverables include a paper summarizing the findings from the convenings and a related webinar that describes the framework and recommendations for its use. These will be shared broadly via the informalscience.org website and other professional venues.
This conference is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
Ideas and expectations about professions and about the process of becoming a professional are changing. Once defined largely by licenses or certificates, many fields of work are looking for more decentralized ways to determine what is a profession, and who is a professional. Many are turning to more decentralized ideas about professions and self-directed processes for lifelong professional learning. An increasing number of fields are using competency frameworks as one mechanism to guide professionalization without standardizing the preparation of those who work in the field. Research is needed
This poster was presented at the 2019 AISL Principal Investigators' Meeting.
This ISL Framework is a tool for professionals and informal science learning (ISL) institutions to understand, plan, and advance their professional capacity.
Individuals can identify their professional competencies and plan their professional development.
ISL institutions can plan, provide, and assess staff development.
ISL Institutions can identify staffing qualifications needed and develop appropriate job postings.
Professional organizations and academic programs can plan training and learning
This poster was presented at the 2020 American Alliance of Museums (AAM) Virutal Conference. The Informal STEM Learning (ISL) Professional Competency Framework was developed and validated through research with professionals across the field.
Informal STEM Learning (ISL) organizations such as science centers, aquariums, and natural history museums play a vital role in stimulating public interest and engagement in science, technology, engineering, and math (STEM). That work is largely dependent on the competencies of the individuals who work within the ISL field. The ISL Professional Competency Framework is a robust and flexible tool for understanding and advancing the skills, knowledge, and capabilities of those individuals. This guide was developed to support the use of the online version of the framework.
The competencies
In 2017, the Education Development Center (EDC) received a grant from the National Science Foundation (NSF) to bring together PIs of STEM Program Resource Centers (PRC) funded under NSF’s Education and Human Resources (EHR), evaluators of NSF STEM projects and programs, and evaluation advisors to address concerns about the quality and consistency of STEM evaluations. According to the grant proposal, the goal was “to increase the capacity of evaluators to produce high quality, conceptually sound, methodologically appropriate evaluations of NSF programs and projects, specifically in the area of
This CAREER proposal focuses on the development of teachers' identities, which are operationalized as beliefs and practices, behaviors, and pedagogical knowledge. The PI uses a qualitative approach, occurring over two phases, to investigate the impact of formal-informal collaborations on identity development over time. The study is grounded in an ecological theoretical approach that incorporates a view of informal learning settings as learner-driven and unique in providing opportunities for interaction with objects during meaning-making experiences among groups of learners. The longitudinal research design includes collection of an array of data, including observations of teaching and learning activities, interviews, survey responses, and archival documents such as student work and videos of classroom experiences. The PI uses a narrative analysis and a grounded theoretical approach to generate themes about beliefs and practices around behaviors and pedagogical knowledge informed by informal science education experiences.
Research findings and related educational activities inform the field's understanding of best practices of integrating informal science activities into science teacher education, including determining appropriate kinds of support for STEM teachers who learn to teach in informal learning environments (ILE). The PI is integrating research findings in the revision of existing courses and the development of new courses and experiences for both new and experienced teachers. The project addresses the need for empirical evidence of impacts of ILE experiences on professional development, and will build capacity of informal science institution and university professionals to provide effective teacher education experiences and new teacher support.
In this virtual conference, The PEAR Institute convened over 40 practitioners and researchers to explore the power of an integrated vision of STEM and social-emotional development (SED). This conference aimed to 1) create a collaboration of out-of-school time (OST) leaders, practitioners, and researchers to map the overlays of STEM and SED; 2) identify best practices for integrated STEM-SED programming; 3) explore common existing and potential measures; and 4) identify how data is measured, communicated, and used for skills that are important to both STEM and SED. Through this collaboration, we aimed to initiate the conversations, establish partnerships, and build the capacity for assessment of STEM and SED in informal STEM learning programs.
Rather than seeking consensus, this conference sought to unearth questions and generate new ideas to lay a foundation for future research and practice, a way to inform the field about high-quality research and practice to promote cross-disciplinary collaboration and synergy between STEM and SED.
Conference deliverables include a conference report, an informal key findings webinar, and conference website (https://www.thepearinstitute.org/stem-sed-conference-2020). The website includes links to a participant directory, conference at-a-glance overview, agendas, readings, and additional resources identified during the conference.
This webinar was presented by the NSF Education and Human Research (EHR) Department to describe a current funding opportunity, the EHR Core Research (ECR) program.
DATE:
TEAM MEMBERS:
Earnestine EasterGregg SolomonJolene Jesse
resourceevaluationProfessional Development, Conferences, and Networks
For the SciGirls Strategies supplemental activity, ten educators were trained to be SciGirls Strategies trainers during June 2019. During that time, they developed action plans for their local teacher training. The goal was for each Trainer to train ten or more teachers in their local schools/districts. Trainers could plan and schedule their workshops to fit their local context in order to accomplish the objectives of building teacher’s confidence and skills in using gender equitable and culturally responsive teaching strategies.
After the training workshop, the trainers met once a month
For the SciGirls Strategies project supplement, ten educators were trained to be SciGirls Strategies Trainers through a four-day in person workshop in June 2019. During the workshop, educators learned about the Gender Equitable Teaching and Advising Strategies (GETAS) course content, the research-based SciGirls Strategies framework and instructional strategies and began to develop plans for their localized professional development for STEM and CTE educators.
The goal of the Train-the-Trainer program was for each Trainer to train 10 teachers in their school and/or school district. Trainers
This RAPID award is made by the AISL program in the Division of Research on Learning in the Directorate for Education and Human Resources, using funds from the Coronavirus Aid, Relief, and Economic Security (CARES) Act. COVID-19 presents a national threat to the health of children and families, presenting serious implications for the mental and physical health of children. This project addresses two critical aspects of the impact on COVID-19 on families: (a) the large-scale shift to at-home learning based on nationwide school closures and (b) the critical need for families to understand the basic science of virus transmission and prevention. To address these needs, the project team will develop a series of STEM activities for families with children in grades K-6 that make use of items readily available in most households. The activities help children and their families learn about viruses, virus transmission, and virus prevention while also developing other STEM-skills, particularly related to engineering design. Importantly, the project team also considers the emotional well-being of children and families during the disruption of the COVID-19 pandemic. Led by researchers from Indiana University and Binghamton University, and experts in educational resource development from Science Friday (a non-profit organization dedicated to increasing the public's access to science and scientific information through podcasts, digital videos, original web articles, and educational resources for teachers and informal educators) the project is further supported by partnerships with the New York Hall of Science, Amazeum (AR), the Gulf of Maine Research Institute (ME), The Tech Museum of Innovation (CA), the Indiana State Museum, and Kopernik Observatory Science Center (NY). The activities will be shared with families through live-streamed web sessions that introduce the activity, give tips to adults for facilitation, share a bit on related STEM careers and engage the audience in dialog about the activity and their current experiences. Versions of the sessions that are recorded will be edited and include closed-captioning and subtitles in multiple languages before being posted on platforms such as YouTube.
This project uses a design-based research approach to investigate strategies for enabling families to actively engage with STEM while home and away from their traditional institutions during a period of crisis. The research components focus on:
Engagement: How do families engage in the activity tasks, in terms of processes, practices, and use of resources? Who participated, why did they choose to participate and how did they engage (including modification of activities)? What barriers prevented interested families from completing activities?
Impact: How did the activities change participants? feelings of: a) efficacy around STEM and b) connectedness/ isolation, during extended school closures?
The Activities: Which activities had the greatest uptake? How many activity ideas were submitted by those outside of the team? What was the age/content focus of each of these activities?
The researchers will analyze social media data (including data on resource downloads and use of tracked links, YouTube and Facebook views, comment threads during livestreams and Likes/Shares/Follows across social media sites) to refine and improve the activities and programming as well as learn about the ways families are engaging in the activities. The researchers will solicit survey responses from website visitors to gather more information on participants, why they participated, how they engaged and how the activities impacted participants? efficacy around STEM and their feelings of connectedness or isolation. The researchers will also ask participants to submit images, videos and text that describes what they are making and their process along the way. Analysis of this data would lead to insights on how children and families use STEM language and practices; how children and families ask questions and use COVID-19-related and other information as part of their design work; and how ideas are formed, shaped and refined as families engage in design and making. While the project focuses on a unique opportunity to collect data on family STEM engagement as families respond to disruptions from the COVID-19 pandemic, this project and its findings will provide a knowledge base that can be utilized to inform future responses to national emergencies, other work aimed at promoting family learning at home, and approaches to supporting children in open-ended problem solving.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.