EvaluATE is a national resource center dedicated to supporting and improving the evaluation practices of approximately 250 ATE grantees across the country. EvaluATE conducts webinars and workshops, publishes a quarterly newsletter, maintains a website with a digital resource library, develops materials to guide evaluation work, and conducts an annual survey of ATE grantees. EvaluATE's mission is to promote the goals of the ATE program by partnering with projects and centers to strengthen the program's evaluation knowledge base, expand the use of exemplary evaluation practices, and support the continuous improvement of technician education throughout the nation. EvaluATE's goals associated with this proposal are to: (1) Ensure that all ATE Principal Investigators and evaluators know the essential elements of a credible and useful evaluation; (2) Maintain a comprehensive collection of online resources for ATE evaluation; (3) Strengthen and expand the network of ATE evaluation stakeholders; and (4) Gather, synthesize, and disseminate data about the ATE program activities to advance knowledge about ATE/technician education. The Center plans to produce a comprehensive set of evaluation resources to complement other services, engaging several community college-based Principal Investigators and evaluators in that process.
EvaluATE's products are informed by current research on evaluation, the National Science Foundation's priorities for the evaluation of ATE grants, and the needs of ATE PIs and evaluators for sound guidance that is immediately relevant and usable in their contexts. The fundamental nature of EvaluATE's work is geared toward supporting ATE grantees to use evaluation regularly to improve their work and demonstrate their impacts. All of EvaluATE's products are available to the public. EvaluATE's findings from the annual survey of ATE grantees aid in advancing understanding of the status of technician education and illuminate areas for additional research. The new survey investigates ATE grantees' work to serve underrepresented and special populations, including women, people of color, and veterans. Survey data are available upon request for research and evaluation purposes.
DATE:
-
TEAM MEMBERS:
Lori WingateArlen GullicksonEmma PerkKelly RobertsonLyssa Becho
resourceprojectProfessional Development, Conferences, and Networks
The Complex Adaptive Systems as a Model for Network Evaluations (CASNET) study was a four-year research project investigating evaluation capacity building (ECB) within a network using a complexity theory lens. The study used a case study approach to examine and understand evaluation capacity building within the Nanoscale Informal Science Education Network (NISE Net). NISE Net is a national community of researchers and informal science educators dedicated to fostering public awareness, engagement, and understanding of nanoscale science, engineering, and technology. Instituted in 2005 through NSF funding (DRL-0532536 and 0940143), NISE Net has continuously expanded and is currently comprised of close to 600 science museum and university partners. The intent of the CASNET project was to provide insights on (1) the implications of complexity theory for promoting widespread and systemic use of evaluation within a network, and (2) complex system conditions that foster or impede ECB within a network, i.e., in this case, within the NISE Net.
The Center for Advancement of Informal Science Education (CAISE), a cooperative agreement with the National Science Foundation Advancing Informal STEM Learning (AISL) program, is a partnership of the Association of Science-Technology Centers with faculty and professionals from the University of Pittsburgh Center for Learning in Out-of-School Environments (UPCLOSE), Oregon State University (OSU), the Great Lakes Science Center, KQED Public Media, advisors and other collaborators. CAISE works to support and resource ongoing improvement of, and NSF investments in, the national infrastructure for informal Science Technology Engineering and Mathematics (STEM) education. CAISE's roles are to build capacity and support continued professionalization for the field by fostering a community that bridges the many varied forms in which informal STEM learning experiences are developed and delivered for learners of all ages. To that end, CAISE activities also include: creating field-driven evidence databases about the impacts of informal STEM education; facilitating federated searches of those databases; furthering dialogue and knowledge transfer between learning research and practice; working to enhance the quality and diversity of evaluation knowledge and processes; and helping STEM researchers improve their efforts in informal STEM education, outreach and communication. For Principal Investigators (PIs) and potential PIs, CAISE provides resources that can assist in the development of evidence-based proposals. It also facilitates and strengthens networks through PI meetings, communications, and other methods that encourage sharing of deliverables, practices, outcomes and findings across projects. For the AISL Program at NSF, CAISE is assisting program officers in understanding the portfolio of awards, identifying the portfolio's impacts in key areas, and integrating the program's investments in education infrastructure.
The National Writing Project (NWP) is collaborating with the Association of Science-Technology Centers (ASTC) on a four-year, full-scale development project that is designed to integrate science and literacy. Partnerships will be formed between NWP sites and ASTC member science centers and museums to develop, test, and refine innovative programs for educators and youth, resulting in the creation of a unique learning network. The project highlights the critical need for the integration of science and literacy and builds on recommendations in the Common Core State Standards and the National Research Council's publication, "A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas." The content focus includes current topics in science and technology such as environmental science, sustainability, synthetic biology, geoengineering, and other subjects which align with science center research and exhibits. The project design is supported by a framework that incorporates a constructivist/inquiry-based approach that capitalizes on the synergy between rigorous science learning and robust literacy practices. Project deliverables include a set of 10 local partnership sites, professional development for network members, a project website, and an evaluation report highlighting lessons learned. Partnership sites will be selected based on interest, proximity, history, and expertise. Two geographically and demographically diverse cohorts, consisting of five partnerships each will be identified in Years 2 and 3. Each set of partners will be charged with creating a comprehensive two-year plan for science literacy activities and products to be implemented at local sites. It is anticipated that the pilot programs may result in the creation of new programs that merge science and writing, integrate writing into existing museum science programs, or integrate science activities into existing NWP programs. Interest-driven youth projects such as citizen science and science journalism activities are examples of programmatic approaches that may be adopted. The partners will convene periodically for planning and professional development focused on the integration of science and literacy for public and professional audiences, provided in part by national practitioners and research experts. A network Design Team that includes leadership representatives from NWP, ASTC, and the project evaluator, Inverness Research, Inc., will oversee project efforts in conjunction with a national advisory board, while a Partnership Coordinator will provide support for the local sites. Inverness Research will conduct a multi-level evaluation to address the following questions: -What is the nature and quality of the local partner arrangements, and the larger network as a whole? -What is the nature and quality of the local science literacy programs that local partners initiate, and how do they engage local participants, and develop their sense of inquiry and communication skills? First, a Designed-Based Implementation Research approach will be used for the developmental evaluation to assess the implementation process. Next, the documentation and portrayal phase will assess the benefits to youth, educators, institutions, and the field using surveys, interviews, observations of educators, and reviews of science communication efforts created by youth. Finally, the summative evaluation includes a comprehensive portfolio of evidence to document the audience impacts and an independent assessment of the project model by an Evaluation Review Board. This project will result in the creation of a robust learning community while contributing knowledge and lessons learned to the field about networks and innovative partnerships. It is anticipated that formal and informal educators will gain increased knowledge about science and literacy programs and develop skills to provide effective programs, while youth will demonstrate increased understanding of key science concepts and the ability to communicate science. Programs created by the local partnerships will serve approximately 650 educators (450 informal educators and 200 K-12 teachers) and 500 youth ages 9-18. Plans for dissemination, expansion, and sustainability will be undertaken by the sub-networks of the collaborating national organizations drawing on the 350 ASTC member institutions and nearly 200 NWP sites at colleges and universities.
The Virginia Institute of Marine Science (VIMS) and The Watermen's Museum, Yorktown, VA, will produce an underwater robotics research and discovery education program in conjunction with time-sensitive, underwater archeological research exploring recently discovered shipwrecks of General Cornwallis's lost fleet in the York River. The urgency of the scientific research is based upon the dynamic environment of the York River with its strong tidal currents, low visibility, and seasonal hypoxia that can rapidly deteriorate the ships, which have been underwater since 1781. Geophysical experts believe that further erosion is likely once the wrecks are exposed. Given the unknown deterioration rate of the shipwrecks coupled with the constraints of implementing the project during the 2011-2012 school-year, any delays would put the scientific research back at least 18 months - a potentially devastating delay for documenting the ships. The monitoring and studying of the historic ships will be conducted by elementary through high school-aged participants and their teachers who will collect the data underwater through robotic missions using VideoRay Remotely Operated Vehicles (ROVs) and a Fetch Automated Underwater Vehicle (AUV) from a command station at The Watermen's Museum. Students and teachers will be introduced to the science, mathematics, and integrated technologies associated with robotic underwater research and will experience events that occur on a real expedition, including mission planning, execution, monitoring, and data analysis. Robotic missions will be conducted within the unique, underwater setting of the historical shipwrecks. Such research experiences and professional development are intended to serve as a key to stimulating student interest in underwater archeological research, the marine environment and ocean science, advanced research using new technologies, and the array of opportunities presented for scientific and creative problem solving associated with underwater research. A comprehensive, outcomes-based formative and summative, external evaluation of the project will be conducted by Dr. L. Art Safer, Loyola University. The evaluation will inform the project's implementation efforts and investigate the project's impact. The newly formed partnership between the Waterman's Museum and VIMS will expand the ISE Program's objectives to forge new partnerships among informal venues, and to expand the use of advanced technologies for informal STEM learning. Extensive public dissemination during and after the project duration, includes but is not limited to, hosting an "Expedition to the Wrecks" web portal on the VIMS BRIDGE site for K-12 educators providing real-time results of the project and live webcasts. The website will be linked to the education portal at the Association for Unmanned Vehicle Systems International, the world's largest organization devoted to promoting unmanned systems and to the FIRST Robotics community through the Virginia portal. The website will be promoted through scientific societies, the National Marine Educators Association, National Science Teachers Association, and ASTC. Links will be provided to the Center for Archeological Research at the College of William and Mary and the Immersion Presents web portal--consultants to Dr. Bob Ballard's K-12 projects and JASON explorations. The NPS Colonial National Historic Park and the Riverwalk Landing will create public exhibits about the shipwreck's archeological and scientific significance, and will provide live observation of the research and the exploration technologies employed in this effort.
The Balboa Park Cultural Partnership, in collaboration with several informal science education and other cultural and business organizations in San Diego, Chicago, and Worcester, MA are implementing a research and development project that investigates a range of possible approaches for stimulating the development of 21st Century creativity skills and innovative processes at the interface between informal STEM learning and methods for creative thinking. The goal of the research is to advance understanding of the potential impacts of creative thinking methods on the public's understanding of and engagement with STEM, with a focus on 21st Century workforce skills of teens and adults. The goal of the project's development activities is to experiment with a variety of "innovation incubator" models in cities around the country. Modeled on business "incubators" or "accelerators" that are designed to foster and accelerate innovation and creativity, these STEM incubators generate collaborations of different professionals and the public around STEM education and other STEM-related topics of local interest that can be explored with the help of creative learning methodologies such as innovative methods to generate creative ideas, ideas for transforming one STEM idea to others, drawing on visual and graphical ideas, improvisation, narrative writing, and the process of using innovative visual displays of information for creating visual roadmaps. Hosting the project's incubators are the Balboa Park Cultural Partnership (San Diego), the Museum of Science and Industry (Chicago) and the EcoTarium (Worcester, MA). National partners are the Association of Science-Technology Centers, the American Association for the Advancement of Science, and the Americans for the Arts. Activities will include: the formation and collaborative processes of three incubator sites, a research study, the development of a creative thinking curriculum infused into science education, professional development based on the curriculum, public engagement events and exhibits, a project website and tools for social networking, and project evaluation. A national advisory council includes professionals in education, science, creativity, and business.
Iridescent is a not-for-profit company that develops and implements informal science and engineering experiences for students by facilitating the translation of the work that scientists and engineers do in a way that makes that work accessible to families. The proposal expands the Iridescent outreach activities funded by the Office of Naval Research, to provide a blended combination of in-person and online support to the families of underrepresented populations. The project is producing twenty videos of scientists and engineers presenting their research that are closely aligned with one hundred scientific inquiry and engineering design-based experiments and lesson plans. These digital resources, collectively called the Curiosity Machine, provide opportunities for parents and children to engage in scientific inquiry and engineering design in multiple face-to-face and online environments, including mobile technologies. The evaluation findings from this project provide a model of how to engage STEM education practitioners, teachers and online communities, to substantively connect underserved communities, in both informal and more formal learning environments to develop experiences with engineering design and to improve students' perspectives about and motivations to prepare for STEM careers. The Curiosity Machine portal is designed to present scientists and engineers explaining the work that they do in a way that makes it accessible to parents and students. Iridescent is working at three sites across the country in South Los Angeles, the South Bronx in New York City, and San Francisco. Students and their families have multiple access points to the science and engineering videos and materials through after school activities, Family Science Nights and summer camps. The project is piloting the use of electronic badges, similar to those offered in the Boy and Girl Scouts as a mechanism to enhance the engagement and persistence of students in the online activities. The project is developing ways to evaluate student engagement and performance through the analysis of the products that students submit online in response to particular science and engineering challenges. Students can also gain extra credit at school for their participation in the Curiosity Machine activities. The materials that the Curiosity Machine activities and challenges use are those that are commonly available to families, and the project provides access to mobile technology to facilitate participation by families. Student access to out of school science and engineering experiences is limited by the resources in terms of time and availability science centers have available. This project develops the resources and tools to bridge the in-school and out of school activities for students through the use of videos and online participation in ways that expand the opportunity of students from underserved populations to continue to engage in substantive science and engineering experiences beyond what they might get during an intermittent visit to a science center. The research and evaluation that is part of this study provides information about how new forms of extrinsic motivation might be used to support student engagement and persistence in learning about science and engineering.
'Be a Scientist!' is a full-scale development project that examines the impact of a scalable, STEM afterschool program which trains engineers to develop and teach inquiry-based Family Science Workshops (FSWs) in underserved communities. This project builds on three years of FSWs which demonstrate improvements in participants' science interest, knowledge, and self-efficacy and tests the model for scale, breadth, and depth. The project partners include the Viterbi School of Engineering at the University of Southern California, the Albert Nerken Engineering Department at the Cooper Union, the Los Angeles Museum of Natural History, and the New York Hall of Science. The content emphasis is physics and engineering and includes topics such as aerodynamics, animal locomotion, automotive engineering, biomechanics, computer architecture, optics, sensors, and transformers. The project targets underserved youth in grades 1-5 in Los Angeles and New York, their parents, and engineering professionals. The design is grounded in motivation theory and is intended to foster participants' intrinsic motivation and self-direction while the comprehensive design takes into account the cultural, social, and intellectual needs of diverse families. The science activities are provided in a series of Family Science Workshops which take place in afterschool programs in eight partner schools in Los Angeles and at the New York Hall of Science in New York City. The FSWs are taught by undergraduate and graduate engineering students with support from practicing engineers who serve as mentors. The primary project deliverable is a five-year longitudinal evaluation designed to assess (1) the impact of intensive training for engineering professionals who deliver family science activities in community settings and (2) families' interest in and understanding of science. Additional project deliverables include a 16-week training program for engineering professionals, 20 physics-based workshops and lesson plans, Family Science Workshops (40 in LA and 5 in NY), a Parent Leadership Program and social networking site, and 5 science training videos. This project will reach nearly one thousand students, parents, and student engineers. The multi-method evaluation will be conducted by the Center for Children and Technology at the Education Development Center. The evaluation questions are as follows: Are activities such as recruitment, training, and FSWs aligned with the project's goals? What is the impact on families' interest in and understanding of science? What is the impact on engineers' communication skills and perspectives about their work? Is the project scalable and able to produce effective technology tools and develop long-term partnerships with schools? Stage 1 begins with the creation of a logic model by stakeholders and the collection of baseline data on families' STEM experiences and knowledge. Stage 2 includes the collection of formative evaluation data over four years on recruitment, training, co-teaching by informal educators, curriculum development, FSWs, and Parent Leadership Program implementation. Finally, a summative evaluation addresses how well the project met the goals associated with improving families' understanding of science, family involvement, social networking, longitudinal impact, and scalability. A comprehensive dissemination plan extends the project's broader impacts in the museum, engineering, evaluation, and education professional communities through publications, conference presentations, as well as web 2.0 tools such as blogs, YouTube, an online social networking forum for parents, and websites. 'Be a Scientist!' advances the field through the development and evaluation of a model for sustained STEM learning experiences that helps informal science education organizations broaden participation, foster collaborations between universities and informal science education organizations, increase STEM-based social capital in underserved communities, identify factors that develop sustained interest in STEM, and empower parents to co-invest and sustain a STEM program in their communities.
Expanding on the encouraging outcomes of an NSF-funded conference, this three-year project led by the National Center for Science and Civic Engagement at Harrisburg University of Science and Technology, in collaboration with the Koshland Science Museum of the National Academy of Sciences, will explore and evaluate ways to support new collaborations between professionals in institutions of higher education and informal STEM education around areas of common interest. The primary goal is to develop the educational infrastructure to grow and efficiently sustain multiple cross-organizational partnership activities at the intersection of learning about science, society and civic engagement around such possible topics as energy, environment, genetics, earth resources, computers and ethics, nanotechnology, etc. The initiative is: 1) creating a joint organizing "secretariat" to provide communications and support through low-cost shared services for at least six partnerships around the country; 2) providing partnership support and technical assistance to seed the six national partnerships, and 3) sharing evaluation and analysis services across all the partnerships. The outcomes of the work pertain to improvements in professional knowledge and practice in higher education and informal science education, as well as the improvement of learning by undergraduates and by the general public.
Portal to the Public: Expanding the National Network (PoP: ENN) is implementing around the county the successful NSF-funded Portal to the Public model in which researchers are trained to communicate and interact with the general public at informal science education (ISE) institutions about the research that they are conducting. The project, which follows on a thorough evaluation of the model at eight sites and current implementation at an additional fifteen sites, will incorporate twenty new ISE sites into the growing network, provide training and mentorship to ISE professionals on the use and adaptation of the PoP implementation manual and toolkits, and develop an enhanced network website that will serve as a communication and innovation hub. The work is responsive to the needs and activities of ISE organizations which continue to expand their missions beyond presenting to the public established science, technology, engineering and math (STEM) and are working to become places where visitors can also experience the process and promise of current research via face-to-face interactions with researchers. The project is expanding both the kind and number of institutions involved around the country and is facilitating their capacity to develop a knowledge base, share experiences and best practices.
The NRC Framework for K – 12 Science Education (2012) lists five major ideas that are essential to the design of assessments and learning environments: 1) limited number of core ideas of science, 2) cross-cutting concepts, 3) engaging students in scientific and engineering practices, 4) building integrated understanding as a developmental process, and 5) the coupling of scientific ideas and scientific and engineering practices to develop integrated understanding. What implications do these major ideas have for assessment in informal science setting? This paper will discuss each of these ideas
The Universally Designed Museum Programming project was envisioned as a way to create public programs that are more inclusive of people with disabilities. We used the concepts of universal design and Universal Design for Learning as well as our prior experiences with these topics in exhibition design and nanotechnology programming as a foundation for our work. Through this project, we gained insight into building a community of interest, facilitating a charrette in an inclusive way, using universal design guidelines to develop programs, and measuring the effectiveness of our process.