The starting point of the Evaluating Evaluation project was our impression that despite the substantial resources that are spent on the summative evaluation of museums and galleries the research has little impact and largely remains ineffectual.
With support from the Wellcome Trust and the Heritage Lottery Fund, we set out to see why this seems to be the case and to explore whether there are ways in which the impact of summative evaluations on knowledge and practice might be enhanced. To this end, we reviewed a substantial number of reports, undertook a range of interviews and held two
The most important consideration in evaluating chemistry outreach efforts is how to best use the evaluation to serve project needs. Evaluation should be about making programs more effective—at communicating ideas, changing attitudes, inspiring action, or reaching wider audiences, for example. A well-conducted evaluation typically contributes to the quality of a project by helping its leaders better define their goals, identify important milestones and indicators of success, and use evidence to support ongoing improvements. At its best, evaluation is an integral part of project design and
The Summative Study of the Nano Mini-exhibition took place during the spring and summer of 2012. After being observed during their Mini-exhibition experience, 455 visitors across six different partner institutions participated in surveys and interviews with NISE Net evaluation team members. This report begins by describing the key findings of the study in detail, with additional information about study methods, instruments, and two exploratory sub-studies found in the Appendices.
This project will be conducted by a team of investigators from North Carolina State University. The principal investigator proposes to examine the characteristics, motivations, in and out-of-school experiences, informal science activities, and career trajectories of 1000 science hobbyists and "master hobbyists." Master hobbyists are individuals who have developed science expertise and spend considerable free time engaging in science as a leisure activity. Master science hobbyists are found across most areas of science (e.g. birdwatchers, amateur astronomers). This research will determine who these individuals are, their career pathways, how they engage in science activities and what motivates, sustains, and defines their science interests. One of the particular goals of this research is to develop new understandings of how science hobby interests develop for women and underserved minorities. In the proposed research investigators will use the results of interviews and surveys to identify contextual factors that influence the motivational processes that, in turn, influenced choices of careers and contribute to ongoing choices in hobby and citizen science activities. Of interest in this study is how citizen scientists who are also serious hobbyists differ from master science hobbyists. Research on citizen scientists has shown that this group is highly motivated by collective motives (such as a desire to help others and further science), whereas this may not be the case with the master science hobbyist. Two groups will be sampled: a) birdwatchers and b) amateur astronomers. This sampling model will allow investigators to contrast their findings by: 1) those who have selected a science career versus those that did not select a science career, 2) those who participate in citizen science activities and those that do not, and 3) those who are birdwatchers (greater mathematical components) and those who are amateur astronomers (lesser mathematical components). Additional coding and analyses will examine any differences in the evolution of bird watching and astronomy hobbies. The results of this research will be examined in light of existing motivational and sociocultural models of career selection. This research will document differences in the perceived motivational elements that influenced master science hobbyists/citizen scientists to choose a science career or not. The results can inform federal, state, and local policies for supporting youth and adults engaged in free choice learning. Results of this research will inform the design of intervention/recruitment programs and ISE outreach initiatives. Potential audiences include ISE institutions (e.g. museums and science centers), organizations with links to STEM (e.g. scouts, boys/girls clubs) and pre- and college initiatives that seek to influence career choices and life-long science interests. The proposed cross-disciplinary approach will promote new understandings of complex issues related to motivation, retention, career selection, leisure activities, engagement with formal and informal educational environments, gender and ethnicity, communities of practice and changes in interests over time. Members of the advisory board have expertise in assessment and measurement and will work closely with the project team to conduct a detailed examination of methodologies and analyses at all phases of the project.
This pathways project will study how audiences in public spaces, in this case those in a museum setting, relate to and make sense of large data displays. The project is preliminary to development of a traveling, hands-on exhibition enabling users to create and utilize representations of big data displays such as maps and charts. As the test case, the project will use science maps that provide an overview of science generally and specific areas of STEM, charting and exploring the history and future of science and technology. The data collection portion of the project will take place at the New York Hall of Science, the Marian Koshland Science Museum, COSI in Columbus, Ohio, and WonderLab Museum in Bloomington, Indiana. The project will create a foundation for the design of museum exhibits and educational programs that teach museum visitors how to explore, engage and make better sense of big data. The project is potentially transformative because big data is becoming ubiquitous and making sense out of large data displays is necessary in order to understand big data sets.
The Adler Planetarium, Johns Hopkins University, and Southern Illinois University-Edwardsville are investigating the potential of online citizen science projects to broaden the pool of volunteers who participate in analysis and investigation of digital data and to deepen volunteers' engagement in scientific inquiry. The Investigating Audience Engagement with Citizen Science project is administering surveys and conducting case studies to identify factors that lead volunteers to engage in the astronomy-focused Galaxy Zoo project and its Zooniverse extensions. The project is (1) identifying volunteers' motivations for joining and staying involved, (2) determining factors that influence volunteers' movement from lower to higher levels of involvement, and (3) designing features that influence volunteer involvement. The project's research findings will help informal science educators and scientists refine existing citizen science programs and develop new ones that maximize volunteer engagement, improve the user experience, and build a more scientifically literate public.
DATE:
-
TEAM MEMBERS:
Karen CarneyMichael RaddickPamela Gay
The Oregon Museum of Science and Industry is implementing a Pathways project that will test and refine a model to promote an appreciation that science is everywhere and personally relevant by engaging transit riders in Portland, OR with location-relevant STEM content through unfacilitated, interactive science exhibits in everyday places. The study will employ a "design-based research" approach that both iteratively tests exhibit effectiveness and develops and refines an underlyting theoretical model that can contribute knowledge to the field. The "Science on the Move" model will be developed and tested using an exhibit prototype that includes 1) an easily transportable prototype core with a familiar touch-screen interface, 2) multiple sets of interactive digital content, and 3) a variety of accompanying outer skins designed to attract the public. The exhibit prototype will be placed at bus transit nodes to reach adults, specifically targeting those without college degrees who are underrepresented in science centers. A range of possible STEM content domains will be selected and tested based on topics of interest to the public. If successful, given the several challenges involved, the feasibility study will be applied more fully around Portland and be a model for other cities to consider.
Experts in informal science education, citizen science, bats, and computer science are developing and evaluating, at Mammoth Cave National Park, a hybrid onsite/online proof-of-concept program for citizen scientists to observe and record (tag) roosting behaviors of Rafinesque's big-eared bats via streaming video. Participants are helping bat researchers document bat behaviors that provide insights into the benefits of communal roosting and interactions that serve to create or maintain social bonds and spread disease. The project is also developing a proof-of-concept mobile kiosk that links the citizen science bat study program to other exhibits at Mammoth Cave. Project personnel are conducting evaluation studies to identify best practices for engaging underserved populations and professional scientists in the study of bat behavior, and developing plans for scaling up the program to include video and educational resources from other bat sites around the country. Lessons learned from the project will inform development of a new, nationwide, online citizen science program focused on bat behavior.
This Pathways proposal seeks to find the most effective means for communicating research, research processes and ocean sciences to the public from the ocean ship Joides Resolution (JR). The JR is an NSF-supported research vessel that assists scientists and educators in their quest for understanding the science of oceans and ocean floors worldwide. The goal of this project is to find the best mechanism for communicating science through the numerous platforms of informal science education. Through a series of discussions by stakeholders that include scientists, educators, end-users in the informal science education community and evaluators, 3-5 pilot projects will be selected for further elaboration, testing and evaluation for communication methodologies. From these studies and refinements, models for communication will be produced for further implementation. Partners in this venture include: Consortium for Ocean Leadership\'s Deep Earth Academy (DEA)and the Education Division for the Integrated Ocean Drilling Program (IODP). This project will enhance the public\'s understanding of oceans, ocean floors, and related research in these elements, and inspire younger individuals who may consider ocean science as a career. Models for how best to communicate the world of ocean science will be tested and disseminated through informal science education networks and platforms. Further, models for evaluation of this multidisciplinary science endeavor will be helpful to advancing the informal science education field.
The Balboa Park Cultural Partnership, in collaboration with several informal science education and other cultural and business organizations in San Diego, Chicago, and Worcester, MA are implementing a research and development project that investigates a range of possible approaches for stimulating the development of 21st Century creativity skills and innovative processes at the interface between informal STEM learning and methods for creative thinking. The goal of the research is to advance understanding of the potential impacts of creative thinking methods on the public's understanding of and engagement with STEM, with a focus on 21st Century workforce skills of teens and adults. The goal of the project's development activities is to experiment with a variety of "innovation incubator" models in cities around the country. Modeled on business "incubators" or "accelerators" that are designed to foster and accelerate innovation and creativity, these STEM incubators generate collaborations of different professionals and the public around STEM education and other STEM-related topics of local interest that can be explored with the help of creative learning methodologies such as innovative methods to generate creative ideas, ideas for transforming one STEM idea to others, drawing on visual and graphical ideas, improvisation, narrative writing, and the process of using innovative visual displays of information for creating visual roadmaps. Hosting the project's incubators are the Balboa Park Cultural Partnership (San Diego), the Museum of Science and Industry (Chicago) and the EcoTarium (Worcester, MA). National partners are the Association of Science-Technology Centers, the American Association for the Advancement of Science, and the Americans for the Arts. Activities will include: the formation and collaborative processes of three incubator sites, a research study, the development of a creative thinking curriculum infused into science education, professional development based on the curriculum, public engagement events and exhibits, a project website and tools for social networking, and project evaluation. A national advisory council includes professionals in education, science, creativity, and business.
The University of California, Davis Tahoe Environmental Research Center (TERC), UC Davis W.M. Keck Center for Active Visualization in the Earth Sciences (KeckCAVES), ECHO Lake Aquarium and Science Center (ECHO), UC Berkeley Lawrence Hall of Science (LHS), and the Institute for Learning Innovation (ILI) will study how 3-D visualizations can most effectively be used to improve general public understanding of freshwater lake ecosystems and Earth science processes through the use of immersive three-dimensional (3-D) visualizations of lake and watershed processes, supplemented by tabletop science activity stations. Two iconic lakes will be the focus of this study: Lake Tahoe in California and Nevada, and Lake Champlain in Vermont and New York, with products readily transferable to other freshwater systems and education venues. The PI will aggregate and share knowledge about how to effectively utilize 3-D technologies and scientific data to support learning from immersive 3-D visualizations, and how other hands-on materials can be combined to most effectively support visitor learning about physical, biological and geochemical processes and systems. The project will be structured to iteratively test, design, and implement 3-D visualizations in both concurrent and staggered development. The public will be engaged in the science behind water quality and ecosystem health; lake formation; lake foodwebs; weather and climate; and the role and impact of people on the ecosystem. A suite of publicly available learning resources will be designed and developed on freshwater ecosystems, including immersive 3-D visualizations; portable science stations with multimedia; a facilitator's guide for docent training; and a Developer's Manual to allow future informal science education venues. Project partners are organized into five teams: 1) Content Preparation and Review: prepare and author content including writing of storyboards, narratives, and activities; 2) 3-D Scientific Visualizations: create visualization products using spatial data; 3) Science Station: plan, design, and produce hands-on materials; 4) Website and Multimedia: produce a dissemination strategy for professional and public audiences; 4) Evaluation: conduct front-end, formative, and summative evaluation of both the 3-D visualizations and science activity stations. The summative evaluation will utilize a mixed methods approach, using both qualitative and quantitative methods, and will include focus groups, semi-structured interviews, web surveys, and in-depth interviews. Leveraging 3-D tools, high-quality visual displays, hands-on activities, and multimedia resources, university-based scientists will work collaboratively with informal science education professionals to extend the project's reach and impact to an audience of 400,000 visitors, including families, youth, school field trip groups, and tourists. The project will implement, evaluate, and disseminate knowledge of how 3-D visualizations and technologies can be designed and configured to effectively support visitor engagement and learning about physical, biological and geochemical processes and systems, and will evaluate how these technologies can be transferred more broadly to other informal science venues and schools for future career and workforce development in these critical STEM areas.