Assessing implementation fidelity for a longitudinal intervention: Strengths and challenges of a responsive evaluation framework

Background: School-based interventions play a crucial role in ameliorating childhood obesity. Grounded in the coordinated school health model (CSHM), the HEROES (Healthy, Energetic, Ready, Outstanding, Enthusiastic Schools) Initiative aims to promote wellness in schools in order to mitigate rising rates of childhood obesity in Indiana, Illinois, and Kentucky. Purpose: Given the multi-year design of the HEROES Initiative, the program has evolved over time to allow for continual enhancement. This presentation will discuss the strengths and challenges of evaluating implementation fidelity in a responsive manner for a longitudinal intervention. Significance: HEROES serves as a model CSHM program. This presentation provides essential information for individuals interested in developing a rigorous process evaluation framework and presents issues that must be considered to ensure program growth. Methodology: A comprehensive school site visit protocol was first conducted in Spring 2010 to examine each facet of the implementation process. Site visits are repeated for all participating schools each spring. Quantitative scores are awarded based on several interviews and observational indices at each site. Findings/Results: Important evaluative analyses have been successfully conducted, including annual individual and aggregated domain and overall implementation scores, which have also been compared by various school characteristics. However, year-by-year comparisons across schools and by implementation year have proven challenging. Conclusions/Recommendations: Responsive evaluation is a necessary component of any intervention that stakeholders wish to improve over time. Yet it can also limit specific evaluative analyses. Lessons learned and strategies used to attempt to overcome these challenges will be provided.