Evaluation of a School Nurse Leadership and Public Health Program: The Role of a Retrospective Pre-Test in Accurately Assessing Growth in Skills

BACKGROUND: The Johnson and Johnson School Health Leadership Program (JJSHLP) aims to improve school and population health through the development of leadership and public health skills among school nurses. The 18-month fellowship program includes a 6-day leadership institute (LI) and implementation of community health initiatives that target an identified need. The JJSHLP impact on participants' skills and behaviors is evaluated through a pre-test self-assessment of nurses’ skills and behaviors with 9- and 18-month post-assessments. Pre-test ratings may be overinflated due to nurses' lack of knowledge on assessed topics, which presents a challenge for capturing an accurate assessment of baseline leadership skills, and thus growth.

DESCRIPTION: The 2014 JJSHLP employed a 17-item retrospective pre-test survey immediately following the LI (N=39).  Participants were asked to reflect on their skills prior to the LI and to retrospectively rate their competency prior to participation. Nurses provided significantly lower retrospective ratings on all skills compared to their initial pre-test ratings (ps < .05).

LESSONS LEARNED: Findings suggest evidence of response shift bias; the nurses’ over-estimated their skill level prior to LI attendance.  The lower retrospective ratings suggest the LI was successful in building nurses' understanding of leadership and management skills, leading to more accurate ratings of their initial skill level.

IMPLICATIONS: A retrospective pre-test survey is a critical part of program evaluation for public health training models. A traditional pre-post design that excludes this assessment may not be an accurate measure of growth in skills gained through participation. Other programs should consider this methodological approach.