Pushing the Limits: Summative Evaluation

Pushing the Limits: Making Sense of Science (PTL) is an NSF-funded program designed to build the capacity of rural and small libraries to provide programming to enhance public understanding of science and math. PTL provides professional support, technical assistance, specially produced video segments, and funding for library professionals and their local science partners to co-facilitate a series of science café-style public discussions with adult patrons. In Phase I of the PTL project (September 2012-August 2013), 20 rural and small libraries piloted the program. In Phase II (September 2013-August 2014), the PTL program was scaled up to 74 additional libraries. External evaluator Goodman Research Group, Inc. conducted the external evaluation of the PTL project, including the second formative evaluation during Phase I and the summative evaluation during Phase II, described in this report. The summative evaluation focused on the effectiveness of the PTL project in meeting its goals for the library professionals, their patrons, and their science partners. GRG’s summative evaluation design included background and post-programming surveys for the Phase II library professionals, post-programming surveys for their science partners, patron surveys after the first and fourth PTL event at 19 libraries, and observations at 10 libraries. Responses rates were excellent for the library professionals (93-94%) and patrons (84-88%) and very good for science partners (73%). Key Findings -The Phase II PTL libraries have moderately low staffing and STEM resources. The library professionals are very experienced with adult programming, but only a quarter had experience with adult science programming. -Fewer than half of science partners had experience with adult public science programming. Before PTL, they rated themselves fairly high in interest, ability, and comfort in public science programming. -Library professionals found the PTL-provided PD resources — particularly the website, listserv, and webinars — very helpful in preparing them to plan, market, and implement their PTL programming. They gave high marks to the PD’s organization and clarity and felt it was especially strong in preparing them to plan and facilitate events. -The majority of science partners felt the PTL-provided materials prepared them to co-facilitate the program; most also drew on their own professional background in science. -Library professionals and science partners customized their events to the topics, audience interests, and their local community and culture in interesting ways, including with food, activities, demonstrations, and examples. -Library professionals and science partners reported great success in planning the series, engaging audiences, and facilitating lively discussions. On average, across all objectives, 79% of library professionals and 85% of science partners rated themselves as very or extremely successful in accomplishing them. -The PTL programming successfully drew a crowd that was not necessarily already interested in STEM topics; 61% were only somewhat or less interested. At most libraries, the PTL events drew people who don’t usually attend the programming. More than half said PTL events drew more men than usual, and a third had bigger crowds than usual. -PTL events were very successful in engaging patrons. The most engaging component was the science partners, followed by the audience discussion. Library professionals also gave high ratings to the PTL programming in terms of broader goals, particularly advancing library goals and serving their communities. -Patrons at the 19 surveyed libraries were extremely engaged, interested, and curious to learn more about the topics. Fully 59% were more interested in the science, technology, and engineering aspects of the event than they had expected. In describing what they had learned, 41% mentioned something involving science or technology. -Library professionals made large gains in self-efficacy, especially in science knowledge, comfort finding science resources, and comfort facilitating science programming. They also felt much better prepared to continue developing and planning science programming. This translates into concrete plans to continue science programming for about two-thirds of the library professionals. -Most science partners plan to continue their involvement with public science programming. Despite their initially high ratings in interest, ability, and comfort with public science, there were significant gains after PTL, particularly in comfort in informal learning environments and in interest in facilitating science programs. Key Recommendations -GRG recommends further enhancing the customizability of the PTL programming by offering several book choices per topic, along with links to existing author videos that could be used with the alternate book selections. Discussion questions at varying levels of scientific sophistication would also help them tailor the program to their audiences. -Library professionals and science partners wanted some additional materials. GRG suggests providing science partners with a brief written orientation to the PTL program, its goals, and the science partner’s role along with a very brief video of the program in action. For library professionals, PTL should supply more customizable marketing templates and expanded lists of related books, videos, and resources. GRG also suggests expanding the discussion questions and providing a set of programming suggestions, activities, and display ideas. -Because there is a fair amount of turnover in library personnel over the course of a programming year, GRG urges the PTL team to consider placing all of the PTL material, including all media, on the website to make it easier and more convenient for libraries to share materials with new staff and with science partners. -Because Pushing the Limits is a highly successful program, GRG recommends providing libraries with more than four thematic units and expanding the program to additional small and rural libraries, as well as to other libraries.

If you would like to edit a resource, please email us to submit your request.