Combining STEM and Character Development in an Out-of-School Time Program: Participatory Practices for Developing and Validating a Theory of Change

Abstract

Background: The goal of the study was to develop and verify a pathway model (or theory of change) for STEM Scouts, an out-of-school time (OST), positive youth development program for elementary through high school-aged youth. Pathway models include connections between hypothesized activities to short-, mid-, and long-term outcomes. STEM Scouts is offered across the United States and a secondary goal of the study was to understand the STEM ecosystem in the communities where STEM Scouts is currently implemented.

Results: Evidence mapping was conducted to determine the extent to which the theory of change was consistent with the extant literature; 69% of the connections on the key pathways were supported by research. For connections where supporting evidence was not found, the connections were either removed or retained because the connection between constructs was understudied. The literature review also led to the addition of outcomes and language revisions to more closely align with terminology used in the literature. Focus groups were conducted to determine the extent to which the pathway model was consistent with youth and adult participant’s lived experience and to better understand STEM ecosystems. The pathway model resonated with most participants’ experiences with STEM Scouts. Participants across multiple focus groups suggested adding outcomes related to learning and leadership. Parents and other family members, the availability of OST STEM programs, and schools were frequently noted as critical elements of the STEM ecosystem.  

Conclusions: Careful evaluation planning, including the development and validation of a detailed theory of change, is a critical step in STEM education program evaluation. This study revealed the value-added when such steps are taken. Critical outcomes and connections would have been missed without the development and validation of the theory of change which will form the basis of a future evaluation. OST STEM programs should strive to engage in careful and detailed evaluation planning in advance of engaging in program evaluation.