An Informal STEM Outcomes Framework to Drive Creative and Culturally Inclusive Planning, Design, and Evaluation

Informal STEM learning institutions, educators, and evaluators face a unique challenge. Each institution sponsors a wide array of programming for all sorts of learners. Educational interactions can range from two- or three-minute interactions to multi-year intensive programming. An educator with a hands-on activity cart in a museum hallway may be interacting with a group of unrelated visitors who span ages from toddlers to seniors. In the next room there may be a group of high school seniors who have been engaged since seventh grade in a career ladder program. Across this programming spectrum, contact time, and learners, how do we plan and evaluate these programs in a way that is useful to the program providers while contributing to a single understanding of institutional impact?

In a review of informal STEM education evaluation, Allen and Peterman (2019) challenged the field to “design methods that characterize the impact of a STEM resource or program not just as a stand-alone offering, but in terms of its capacity to support and connect to other experiences, resources, and programs in the ecosystem” (p. 29–-30). The Center of Science and Industry (COSI) Center for Research and Evaluation and the National Museum of Natural History (NMNH) have been developing a Strategic Outcomes Framework that addresses these challenges. After incorporating recommendations and feedback from an NSF-funded conference organized in April 2021 by COSI, NMNH, and PEAR (NSF #2039209) involving educators, practitioners, and content knowledge scholars, the framework is becoming a useful tool. Over a 10-day period, participants explored the framework’s potential and challenged its limits. They reviewed and contributed to the soundness and utility of the outcome categorizations as well as the outcome types within them.

The Strategic Outcomes Framework builds on informal learning categories and strands (Bell et al., 2009; Friedman, 2008) by delineating realistic outcome expectations based on dosage and complexity of programming. In its simplest form (Figure 1), each category of outcome divides into two scopes.

Figure 1. Simplified Strategic Outcomes Framework

 

The more refined framework (Figure 2) divides each scope into outcome types. These depth levels and outcome types were based on research within each of the disciplines (see resources below) and feedback from conference scholars representing each discipline.

Figure 2. Refined Strategic Outcomes Framework

Specifically, the Framework’s outcomes exist within eight categories: interest, attitude, knowledge, STEM skills, 21st century skills  & social emotional development and learning (SED/L), behavior, STEM capital, and career activation and preparation. Each category includes roughly six outcome types that can be mapped to programming that ranges from short duration with simple content to extended duration with complex content. Once the category has been selected, a program designer uses the outcome type’s suggested indicator statement to specify an outcome for the program’s participants. For example, a program aimed at having participants learn and apply a STEM skill (21st Century Skills 2) might specify that outcome as “Participants in this program actively used critical thinking skills to select resources for making decisions about their health.”

Application of these outcomes involves some important principles and guidelines. First, a program rarely targets a single outcome. Instead, program design derives from outcome combinations, each of which leads to different educational strategies. These combinations can occur both between and within categories. Programs can use the Framework for primary and supporting outcomes. For example, a program with the primary outcome of creating a behavior change may have supporting attitude, knowledge, and skill outcomes that participants need to be able to make that change. A program also may be defined in terms of expected and “reach” outcomes. For example, all participants in a workshop may be expected to report that they experienced situational interest, and the workshop also has the goal that some of those participants will develop personal interest. 

Moreover, outcome types and even depth levels are value free. A program that targets a simple, less complex outcome can be equally valuable as one that targets deeper, more complex outcomes. Second, outcome depth does not necessarily correlate with developmental age or learner academic standing. Conceptually, given the time and creative programming strategy, some level six outcomes can be targeted even in preschool settings.

Another principle is that the Framework must be nested within a system that acknowledges the voices of value holders (aka stakeholders) while selecting, defining, and strategizing toward achieving these outcomes. We have nested the Framework within what we call “Voices and Choices Planning for Self Determination.” This process, based on Peter Checkland’s soft systems approach (Checkland and Poulter, 2006), recognizes that each value holder brings to the planning process an acknowledged worldview and set of environmental constraints or context. Each value holder also assumes the roles of customer (who benefits), owner (who has the power to start and stop the program), and change agent (who selects and implements the strategies that lead to outcomes). Discussions of these varied conditions and perspectives lead to Framework use that honors and empowers value holder voices which are often overlooked in the program planning process.

Both the Framework and the self-determination process can be employed at various program stages. At the institution level, they can help determine how all the programs align with each other and the institutional mission. At the individual program level, they can be used during design and planning, promotion and distribution, implementation, and evaluation.

Another principle is that the Strategic Outcomes Framework functions alongside a similar educational environment objectives framework that addresses cognitive, affective, and physical learning conditions as separate from outcomes.

With these assumptions, the Framework has the potential to provide informal education institutions with a tool for both program-level and institution-level strategic educational planning and evaluation. As an example of how it informs institution-level planning, consider a report the Framework can produce (Figure 3). In this figure, donut charts illustrate the range of outcomes addressed by an institution’s programs. The darker shading represents more successful outcome achievement. The number to the upper right of each donut shows how many programs contributed to the donut; the number to the lower left represents the number of responses. In this example, note that, across the outcome categories, levels 1 and 2 are well targeted in this institution—though the more complex levels have not been as well achieved.

Figure 3. Institution-wide Report 

Benefits of the Strategic Outcomes Framework

Early users have reported that the Framework:

• Promotes important dialogue at each program phase
• Aids in strategic planning
• Helps to articulate an overview of institutional educational intentions
• Parses learning outcomes to realistically reflect contact time and experience
• Focuses program planning
• Builds practitioner capacity
• Builds evaluation capacity
• Aids in communication to and collaboration with value holders (aka stakeholders)
• Creates a full context for learning
• Provides language and data for communicating impact

Some caveats

On the other hand, the Framework comes with some important limitations and challenges to solve. Among these are:

• Responsivity to cultural diversity, equity, accessibility, inclusiveness and voice
• Reinforcing the “authority” of science as constructed by exclusive science community membership
• Accounting for outcomes not in the framework: e.g., social justice and system change
• Focus on acquisition and assimilation; what about transformation?
• How to account for constructivist, serendipitous outcomes

Future work

We continue to develop and pilot test this Framework and the broader system, and are learning more with each collaborator. We are very interested in having others use the Framework and provide feedback to us, knowing that future use will include validity testing. Contact us and be part of a conversation about how to use the Framework.

 

References

Allen, S., & Peterman, K. (2019). Evaluating Informal STEM Education: Issues and Challenges in Context. New Directions for Evaluation, 2019(161), 17–33. https://doi.org/10.1002/EV.20354

Bell, P., Lewenstein, B., Shouse, A. W., & Feder, M. A. (2009). Learning science in informal environments: People, places, and pursuits (P. Bell, B. Lewenstein, A. W. Shouse, & M. A. Feder (eds.)). The National Academies Press. http://books.google.com/books?hl=en&lr=&id=vdOIlY7WdJUC&oi=fnd&pg=PT1&dq=informal+education+exhibit+influence+on+behavior&ots=PQ_InA5U8t&sig=mj1aviIrKNkc1cIRd4g8TCdgRG0

Checkland, P., & Poulter, J. (2006). Learning For Action: A Short Definitive Account of Soft Learning For Action. John Wiley & Sons, Ltd., West Sussex England. 

Friedman, A. J. (Ed.). (2008). Framework for Evaluating Impacts of Informal Science Education Projects | InformalScience.org. https://www.informalscience.org/framework-evaluating- impacts-informal-science-education-projects

References by outcome category

Interest

Hidi, S., & Renninger, K. A. (2006). The Four-Phase Model of Interest Development. Educational Psychologist, 41(2), 111–127. 

Conference scholar: Anne Renninger, Swarthmore College

Attitude

Bohner, G., & Dickel, N. (2011). Attitudes and attitude change. Annual Review of Psychology, 62, 391–417. https://doi.org/10.1146/annurev.psych.121208.131609 

Sinatra, G. M., Kienhues, D., & Hofer, B. K. (2014). Addressing Challenges to Public Understanding of Science: Epistemic Cognition, Motivated Reasoning, and Conceptual Change. https://Doi.Org/10.1080/00461520.2014.916216, 49(2), 123–138. https://doi.org/10.1080/00461520.2014.916216 

Conference scholar: Cameron Denson, North Carolina State University; Mwenda Kudumu, North Carolina State University

Knowledge

Berland, L. K., Schwarz, C. V., Krist, C., Kenyon, L., Lo, A. S., & Reiser, B. J. (2016). Epistemologies in practice: Making scientific practices meaningful for students. Journal of Research in Science Teaching, 53(7), 1082–1112. https://doi.org/10.1002/tea.21257

Carlone, H. B., Mercier, A. K., & Metzger, S. R. (2021). The Production of Epistemic Culture and Agency during a First-Grade Engineering Design Unit in an Urban Emergent School. Journal of Pre-College Engineering Education Research (J-PEER), 11(1), 10. https://doi.org/10.7771/2157-9288.1295

Duschl, R. (2008). Science Education in Three-Part Harmony: Balancing Conceptual, Epistemic, and Social Learning Goals: http://Dx.Doi.Org/10.3102/0091732X07309371, 32, 268–291. https://doi.org/10.3102/0091732X07309371

Lau, M., & Sikorski, T. R. (2018). Dimensions of Science Promoted in Museum Experiences for Teachers. Journal of Science Teacher Education, 29(7), 578–599. https://doi.org/10.1080/1046560X.2018.1483688

McGill, T., & Volet, S. (1997). A conceptual framework for analyzing students’ knowledge of programming. Journal of Research on Computing in Education, 29(3), 276. https://www.tandfonline.com/doi/abs/10.1080/08886504.1997.10782199

Pierson, A. E., & Clark, D. B. (2018). Engaging students in computational modeling: The role of an external audience in shaping conceptual learning, model quality, and classroom discourse. Science Education, 102(6), 1336–1362. https://doi.org/10.1002/SCE.21476

Sinatra, G. M., Kienhues, D., & Hofer, B. K. (2014). Addressing Challenges to Public Understanding of Science: Epistemic Cognition, Motivated Reasoning, and Conceptual Change. https://Doi.Org/10.1080/00461520.2014.916216, 49(2), 123–138. https://doi.org/10.1080/00461520.2014.916216

Yildirim, Z., Ozden, M. Y., & Aksu, M. (2001). Comparison of hypermedia learning and traditional instruction on knowledge acquisition and retention. Journal of Educational Research, 94(4), 207–214. https://doi.org/10.1080/00220670109598754

Conference scholar: Tiffany Sikorski, George Washington University

STEM Skills

Anderson, L., & Sosniak, L. (1994). Bloom’s Taxonomy. University of Chicago Press. 

Landry, J. P., Longnecker, H. E., Haigood, B., & Feinstein, D. L. (2000). Comparing Entry-Level Skill Depths Across Information Systems Job Types: Perceptions of IS Faculty: Americas Conference on Information Systems, 1968–1972.

Conference scholars: Angela Calabrese Barton, Michigan State University; Edna Tan, University of North Carolina, Greensboro

21^st Century Skills

Anderson, L., & Sosniak, L. (1994). Bloom’s Taxonomy. University of Chicago Press.

Armstrong, P. (2016). Bloom’s Taxonomy Background Information. Center for Teaching, Vanderbilt University. https://programs.caringsafely.org/wp-content/uploads/2019/05/Caring-Safely-Professional-Program-Course-Development.pdf

Landry, J. P., Longnecker, H. E., Haigood, B., & Feinstein, D. L. (2000). Comparing Entry-Level Skill Depths Across Information Systems Job Types: Perceptions of IS Faculty: Americas Conference on Information Systems, 1968–1972.

Zeidler, D. L. (2016). STEM education: A deficit framework for the twenty-first century? A sociocultural socioscientific response. Cultural Studies of Science Education, 11(1), 11–26. https://doi.org/10.1007/s11422-014-9578-z

Conference scholar: Dana Zeidler, University of South Florida

Behavior 

Allen, S., Campbell, P. B., Dierking, L. D., Flagg, B. N., Alan, J. F., Garibay, C., Korn, R., Silverstein, G., & Ucko, D. A. (2008). Framework for evaluating impacts of informal science education projects: report from a national science foundation workshop (A. J. Friedman (Ed.)).

Dierking, L. D. (2008). Evidence and categories of ISE impacts. In J. Friedman, Alan (Ed.), Framework for evaluating impacts of informal science education projects. National Science Foundation Directorate for Education and Human Resources DRL. 

Heimlich, J. E., & Ardoin, N. M. (2008). Understanding behavior to understand behavior change: A literature review. Environmental Education Research, 14(3), 215–237. 

Heimlich, J. E., Mony, P., & Yocco, V. (2013). Belief to Behavior: A Vital Link. In International Handbook of Research on Environmental Education (pp. 262–274). Routledge.

Conference Scholar: Joe Heimlich, COSI Center for Research & Evaluation

STEM Capital

Archer, L., Dawson, E., DeWitt, J., Seakins, A., & Wong, B. (2015). “Science capital”: A conceptual, methodological, and empirical argument for extending bourdieusian notions of capital beyond the arts. Journal of Research in Science Teaching, 52, 922–948.

Basu, S. J., Calabrese-Barton, A., & Tan, E. (Eds.). (2011). Democratic Science Teaching: Building the Expertise to Empower Low-income Minority Youth in Science (Vol. 3). Sense Publishers.

Carlone, H. B., & Johnson, A. (2007). Understanding the science experiences of successful women of color: Science identity as an analytic lens. Journal of Research in Science Teaching, 44(8), 1187–1218.

DeWitt, J., Archer, L., & Mau, A. (2016). Dimensions of science capital: exploring its potential for understanding students’ science participation. International Journal of Science Education, 38(16), 2431–2449. https://doi.org/10.1080/09500693.2016.1248520 

Godec, S., King, H., & Archer, L. (2017). The Science Capital Teaching Approach: engaging students with science, promoting social justice. London: University College London.

Conference scholar: Spela Godec, University College London 

Career Building

Byars-Winston, A., Gutierrez, B., Topp, S., & Carnes, M. (2011). Integrating theory and practice to increase scientific workforce diversity: a framework for career development in graduate research training. CBE—Life Sciences Education, 10(4), 357–367. https://doi.org/10.1187/cbe.10-12-0145

Hernandez, P. R., Woodcock, A., Estrada, M., & Schultz, P. W. (2018). Undergraduate research experiences broaden diversity in the scientific workforce. BioScience, 68(3), 204–211. https://doi.org/10.1093/biosci/bix163

Tai, R. H., Kong, X., Mitchell, C. E., Dabney, K. P., Read, D. M., Jeffe, D. B., Andriole, D. A., & Wathington, H. D. (2017). Examining summer laboratory research apprenticeships for high school students as a factor in entry to MD/PhD programs at matriculation. CBE—Life Sciences Education, 16(2), ar37. https://doi.org/10.1187/cbe.15-07-0161 

Conference scholar: Robert Tai, University of Virginia