David L Buckeridge, MD, MSc¹, Paul Switzer, PhD², and Mark A Musen, MD, PhD¹. (1) Stanford Medical Informatics, Stanford University, MSOB X-215, 251 Campus Drive, Stanford, CA 94305-5479, 650 723 6979, david.buckeridge@stanford.edu, (2) Statistics, Stanford University, Sequoia 136, Stanford, CA 94305

Background: Existing evaluation metrics for outbreak detection are not well suited to evaluating timeliness, which is important for many outbreaks including those resulting from bioterrorism.

Methods: We develop a Bayesian approach to evaluating the timeliness of outbreak detection. We first estimate empirical probability density functions for detector output under outbreak and non-outbreak conditions. We then use these functions together with Bayes’ theorem to calculate sequentially the posterior probability of an outbreak given the detector output and a specified prior probability. We plot the posterior probability during outbreak conditions against days following the onset of the outbreak to summarize timeliness and sensitivity. From this plot, it is possible to calculate timeliness and sensitivity at different thresholds. We plot traces of posterior probability during non-outbreak conditions in a similar manner to summarize specificity.

Results: We applied our evaluation method to the output of outbreak detectors run against outpatient clinician visit billing data from five metropolitan areas. Detectors attempted to identify 8 respiratory and 5 gastrointestinal outbreaks in diagnostic syndrome sets as defined by a panel of experts. Our results illustrate that detectors with reasonable sensitivity (0.77, 95% CI 0.54 – 1.00) and false alarm rate (1 per 4 weeks) can exhibit a considerable range in timeliness (1 to 14 days following onset of outbreak).

Conclusion: A Bayesian approach to evaluation of outbreak detector performance allows assessment of timeliness of outbreak detection in conjunction with sensitivity and specificity.

Learning Objectives: At the conclusion of the session, the participant (learner) in this session will be able to

• List methods for evaluating outbreak detection algorithms in surveillance systems
• Describe shortcomings of current methods for evaluating timeliness of detection
• Articulate the benefits and shortcomings of a Bayesian approach for evaluating timeliness

Keywords: Surveillance, Bioterrorism

Presenting author's disclosure statement:
I do not have any significant financial interest/arrangement or affiliation with any organization/institution whose products or services are being discussed in this session.