Quality Improvement in Public Health: “Real World” evaluation to meet local program needs

Background/issue: Public health quality improvement suffers from evaluations that are often poorly conceived, have sub-optimal study designs, and lack good quality data for program outcomes.

Description: A program evaluation was designed for a federally grant funded MCH home visiting program targeted at families affected by substance abuse. The program to be evaluated was already in progress and had been designed without adequate attention to effective study design, data for outcomes evaluation, or analyzable data for program implementation. Federal technical assistance was inflexible and demanding beyond the means of local program capacity. The new program evaluation plan made ready use of Bamberger’s “Real World” evaluation approach incorporating constraints associated with budget, stakeholder expectations, data limitations, and political considerations.

Lessons learned: Beyond the obvious lesson—that program evaluations must be planned prior to implementation, program managers and evaluators learned to use Bamberger’s constraints to evaluate the program for local program improvement. Failing to consistently satisfy the federal funder, the local program was able to expand capacity and illuminate which programs were most likely to be effective in this small rural setting. 

Implications/recommendations: Federally funded public health programs often come with the expectation that public health departments have evaluation capacities far beyond what is realistic for small agencies in rural settings with limited resources. Contracting with “expert” program evaluators is often not possible given local resources, and may consume much of program budgets. To be successful, health departments may need to lay realistic expectations of program evaluability and retain a focus on local needs.