Citizen Science Project: Summative Evaluation

As part of its National Science Foundation (NSF) grant, the Conservation Trust of Puerto Rico (the Trust) hired Randi Korn & Associates, Inc. (RK&A) to evaluate its Citizen Science Project. The goal of the study was to measure the project’s success against an Impact Framework, the organizing framework NSF requires for evaluation. During the life of the project, an Impact Framework serves several functions, including to (1) articulate the project goals for the NSF proposal; (2) act as a roadmap for the Trust to align project practices with the impact it intends to achieve on specific audiences; and (3) serve as the gauge for the evaluation. How did we approach this study? Trust staff, with RK&A, created the Impact Framework, which RK&A used to develop all data collection instruments, conduct analysis, and report findings. RK&A employed a mixed-methods approach, including control and treatment group interviews, case studies, and core participant interviews whereby each method served a specific purpose. Control and treatment interviews measured impact among a broad group of participants using a rubric; qualitative interview data from the two groups were scored along a continuum and their scores were compared to measure impact. Case studies sought to identify how different roles (individual staff and project participants) may have contributed to the successes and challenges of the project. Case studies are not generalizable and not meant to demonstrate project impact. Interviews with core participants (those who had a high level of participation, including data collection, analysis, and reporting, usually for a single research project) explored how the project may have affected surrounding communities. What did we learn? Analysis of data indicate that the project did not have a measurable impact on participants. Participants had low to moderate achievement for most impacts, with the exception of their comfort with science-process skills, which they rated highly, although there was not a statistical difference between control and treatment groups. However, findings demonstrate potential for the program to deepen learning among participants. For example, participants scored low on their ability to identify relationships among living things in or along the Rio Grande of Manati River, indicating an opportunity to increase their knowledge of these relationships through reiteration of connections during programming. Core participant interviews revealed that participants are in the beginning stages of exploring their own research questions and sharing their experiences with the community, all of which suggests that through repeated exposure to the program, deeper engagement and learning gains are possible. What are the implications of the findings? While the overall findings may seem discouraging, the notion of citizen science, and specifically its manifestation in this project, is relatively new, and informal science organizations, including the Trust, are still perfecting the implementation of the “contributor, collaborator, and co-creator” citizen-science model employed in this project. Achieving measurable impact takes time (on the scale of years not months), and it is unusual for the initial phases of a project to detect change in participants on a broad scale, unless participants are coming to the project as blank slates, which we know from the current literature on citizen science is not the case. Rather, the majority of people who select to participate in citizen science activities are already inclined towards nature and science, which creates a ceiling effect. Further, because the majority of participants have only some degree of project exposure such as collecting data, achieving impact is difficult. However, there may be opportunities to achieve greater impact with core participants. They have repeated exposure and a more holistic experience than casual participants. The Trust experimented with a project model that had a discernible impact on core participants. Applying lessons learned from this evaluation to the next iteration of the project model could increase impact over time.

If you would like to edit a resource, please email us to submit your request.