Improving Biosurveillance System Performance. Ronald D. Fricker, Jr. Virginia Tech July 7, 2015

Size: px

Start display at page:

Download "Improving Biosurveillance System Performance. Ronald D. Fricker, Jr. Virginia Tech July 7, 2015"

Dwight Skinner
5 years ago
Views:

1 Improving Biosurveillance System Performance Ronald D. Fricker, Jr. Virginia Tech July 7, 2015

2 What is Biosurveillance? The term biosurveillance means the process of active datagathering of biosphere data in order to achieve early warning of health threats, early detection of health events, and overall situational awareness of disease activity. [1] [1] Homeland Security Presidential Directive HSPD-21, October 18,

3 Biosurveillance is (near) real-time monitoring of diseaserelated data to (quickly) detect outbreaks It s part of public health surveillance: Biosurveillance of people 3

4 Purposes of Biosurveillance Early Event Detection (EED) is the ability to detect at the earliest possible time events that may signal a public health emergency. EED is comprised of case and suspect case reporting along with statistical analysis of health-related data. [1] Health Situational Awareness is the ability to utilize detailed, real-time health data to confirm, refute and to provide an effective response to the existence of an outbreak. It also is used to monitor an outbreak s magnitude, geography, rate of change and life cycle. [1] [1] CDC ( 4

5 Existing Biosurveillance Systems BioSense developed by the CDC Early Aberration Reporting System (EARS) developed by the CDC Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE) developed by the Department of Defense Real-time Outbreak Detection System (RODS) developed by the University of Pittsburgh Google Flu Trends developed by Google 5

6 One Application: BioSense 6

7 Challenges in Biosurveillance There are many, including: Designing and implementing systems for data collection and analysis Characterizing outbreak manifestations Modeling systematic effects and data trends Developing and evaluating detection algorithms I come at the problem from an SPC/SPM perspective, but there are critical differences For more detail, see: Fricker, R.D., Jr. (2011). Some Methodological Issues in Biosurveillance (with commentaries and rejoinder), Statistics in Medicine, 30,

8 Data Characterization Challenges Classical 8

Even More Fundamental Data Issues Deriving events from chief

definitions For more detail, see: Hagen, K.S., R.D. Fricker, Jr.

Assessing the Early Aberration Reporting System's Ability to

9 Even More Fundamental Data Issues Deriving events from chief complaint text data Daily counts vary depending on key word definitions For more detail, see: Hagen, K.S., R.D. Fricker, Jr., K. Hanni, S. Barnes, and K. Michie (2011). Assessing the Early Aberration Reporting System's Ability to Locally Detect the 2009 Influenza Pandemic, Statistics, Politics, and Policy, 2, issue 1, article 1. 9

10 Challenge: Using the Right Metrics In SPC, we characterize performance in terms of in-control and out-of-control ARLs In biosurveillance, terms are at best misnomers What does it mean to be in control? More importantly, with transient outbreaks, ARL 0 and ARL 1 are insufficient measures Requires three metrics to characterize performance; e.g., Average time to first signal or perhaps ATBSE Conditional expected delay Probability of successful detection 10

11 Metrics Example Performance comparison of EARS vs. CUSUM Simulated data CUSUM run on residuals from an adaptive regression model CUSUM outperformed EARS Though slightly higher CED, misses many fewer outbreaks (higher PSD) CED CED CED 1-PSD 1-PSD 1-PSD For more detail, see: Fricker, R.D., Jr., Hegler, B.L., and D.A. Dunfee (2008). Comparing Syndromic Surveillance Detection Methods: EARS' Versus a CUSUM-Based Methodology, Statistics in Medicine, 27,

12 Challenge: Controlling False Positives Perhaps most importantly, public health officials are simultaneously monitoring multiple time series Explicitly controlling false positive rate is critical From Galit Shmueli s blog: most health monitors learned to ignore alarms triggered by their system. This is due to the excessive false alarm rate that is typical of most systems - there is nearly an alarm every day! [1] [1] See 12

13 Tunable Biosurveillance Can t watch for everything, everywhere, all the time and maintain a reasonable false positive signal rate Instead, design system to be tunable I.e., use it to focus on the most likely threats One approach: set detection thresholds so most likely events most detectable As threats change, can change thresholds Also, set thresholds so that false positive error rate constrained at tolerable level 13

14 System Description Let X it denote residual from a model forecasting counts from region i on day t If no outbreak anywhere X it ~ F 0 for all i and t If outbreak occurs on day t, X it ~ F 1, t =t, t+1,... Use one-sided Shewhart chart to look for outbreaks in each of n regions Signal on day t* if Xit* hi for one or more regions Each location has a (conditional) probability of outbreak p,...,, 1 1 p n p i = Probability defined by system user i 14

15 Performance Metrics of Interest Pr(detect outbreak at i) = [1 F 1 (h i )] p i q i (h) where q i (h) = Y j6=i F 0 (h j )(1 p j ) + X j6=i + + X j6=i 0 F 0 (h j )](1 p j ) Y F 0 (h k )(1 p k ) A k6=i,j 0 0 (h j )(1 p j ) Y [1 F 0 (h k )](1 p k ) A k6=i,j + Y j6=i[1 F 0 (h j )](1 p j ) E(# false positive signals) = X i [1 F 0 (h i )] 15

16 About the Metrics For a single Shewhart control chart: ARL 0 =[1 ARL 1 =[1 F 0 (h)] F 1 (h)] 1 1 For multiple charts with different thresholds, probability of detection and expected number of false positive signals are (to me) the logical summary performance metrics Particularly for biosurveillance, where explicit monitoring of false positive rate critical 16

17 It s an Optimization Problem max h s.t. X [1 F 1 (h i )] p i q i (h) i X [1 F 0 (h i )] apple apple i with q i (h) = Y j6=i F 0 (h j )(1 p j ) + X j6=i + + X j6=i 0 F 0 (h j )](1 p j ) Y F 0 (h k )(1 p k ) A k6=i,j k6=i,j 0 0 (h j )(1 p j ) Y [1 F 0 (h k )](1 p k ) A For n = 10, there are 512 terms ugh! + Y j6=i[1 F 0 (h j )](1 p j ) 17

18 Simplifying the Problem Monitoring n regions means optimization has n free parameters Hard for to solve for large systems So, consider simplifying the objective function from X max [1 F 1 (h i )] p i q i (h) h to max h i X [1 F 1 (h i )] p i i 18

19 Simplification Works Well Scenario p 1 p 2 p 3 E(#FA) P(detect) for Full Objective Function P(detect) for Reduced Obj Function Percent Change 1 1/3 1/3 1/ Detecting a mean shift of 2 standard deviations Detection only occurs if signal occurs in region with the outbreak 19

20 A Benefit of Simplifying Simplified problem has a one-parameter solution [1] Theorem: For the constrained optimization problem X max [1 F 1 (h i )] p i h i X s.t. [1 F 0 (h i )] apple apple µ where F 0 =N(0,1) and F 1 =N(,1), the optimal solution is to find that satisfies nx 1 µ ln(p i) = n apple, i=1 and where the thresholds are i h i = 1 µ ln(p i). [1] Fricker, R.D., Jr., and D. Banschbach (2012). Optimizing Biosurveillance Systems that Use Threshold-based Event Detection Methods, Information Fusion, 13,

21 Hospital Monitoring Illustration We re monitoring standardized residuals from n = 10 hospital forecasts Using a model to remove systematic effects in the data, can assume X it iid F 0 =N(0,1) An outbreak will result in a 2 increase in the mean of the residuals so that F 1 =N(2,1) One hospital more likely to have an outbreak: p = {.797,.064,.056,.048,.013,.006,.006,.005,.003,.002} Then, the thresholds are h i = ln(p i )/2. 21

Results Hospital i p i Common Threshold #1 Variable Thresholds Common Threshold #2 1 0.797 2.189 1.162 1.780 2 0.064 2.189 2.422 1.780 3 0.056 2.189 2.490 1.780 4 0.048 2.189 2.567 1.780 5 0.013 2.

22 Results Hospital i p i Common Threshold #1 Variable Thresholds Common Threshold # P(detect) = E(#FA) =

23 Other Results Hospital i p i Common Threshold #1 Variable Thresholds Common Threshold # P(detect) = E(#FA) =

24 Optimizing a County-level System 24

25 Thresholds as a Function of Probability of Outbreak / Attack Counties with low probability of attack à high thresholds Unlikely to detect attack Few false signals Counties with high probability of attack à lower thresholds Better chance to detect attack Higher number of false signals 25

26 Application to Industrial SPC Method also applies directly to an industrial setting Monitoring n production lines using -charts Goal is to maximize the factory-wide probability of detecting an out-of-control condition subject to a constraint on the expected number of false signals Most relevant when there are differences between production lines in terms of probability of going out-of-control Perhaps machines on one line are much older (or less reliable or less precise) than on the others x For more detail, see: Fricker, R.D., Jr. (2009). Optimizing Shewhart Charts in Parallel Production Lines, International Journal of Quality Engineering and Technology, 1,

27 Future Research (1) How to improve on current approach? E.g., for some : max h X i [1 F 1 (h i )] [(1 )p i + /n] P(detect outbreak at lcoation i) α 27

28 Future Research (2) Apply to CUSUM and EWMA Deriving an equivalently simple threshold expression seems unlikely Perhaps use EWMA & CUSUM thresholds proportional to Shewhart thresholds? Perhaps apply Shewhart thresholds as probability limits for CUSUM and EWMA? How do the methods perform with a transient out-of-control period? 28

29 Selected References Fricker, R.D., Jr. (2013). Introduction to Statistical Methods for Biosurveillance: With an Emphasis on Syndromic Surveillance, Cambridge University Press. Fricker, R.D., Jr., and D. Banschbach (2012). Optimizing Biosurveillance Systems that Use Threshold-based Event Detection Methods, Information Fusion, 13, Hagen, K.S., R.D. Fricker, Jr., K. Hanni, S. Barnes, and K. Michie (2011). Assessing the Early Aberration Reporting System's Ability to Locally Detect the 2009 Influenza Pandemic, Statistics, Politics, and Policy, 2, issue 1, article 1. Fricker, R.D., Jr. (2011). Some Methodological Issues in Biosurveillance (with commentaries and rejoinder), Statistics in Medicine, 30, Fricker, R.D., Jr., Hegler, B.L., and D.A. Dunfee (2008). Comparing Syndromic Surveillance Detection Methods: EARS' Versus a CUSUM-Based Methodology, Statistics in Medicine, 27, Fricker, R.D., Jr., and H.R. Rolka (2006). Protecting Against Biological Terrorism: Statistical Issues in Electronic Biosurveillance, Chance, 19,

Take-aways SPC/SPM literature has a lot to offer Public health community would benefit from further exposure Must be adapted to the problem Data Terminology Metrics Lots of

30 Take-aways SPC/SPM literature has a lot to offer Public health community would benefit from further exposure Must be adapted to the problem Data Terminology Metrics Lots of challenges in biosurveillance, including: Developing methods that appropriately allow for controlling false positive rate Developing methods that satisfy both EED and SA needs 30

31 Questions? 31

Quality Control & Statistical Process Control (SPC)

Quality Control & Statistical Process Control (SPC) DR. RON FRICKER PROFESSOR & HEAD, DEPARTMENT OF STATISTICS DATAWORKS CONFERENCE, MARCH 22, 2018 Agenda Some Terminology & Background SPC Methods & Philosophy