Forecasting a Moving Target: Ensemble Models for ILI Case Count Predictions

Transcription

1 Forecasting a Moving Target: Ensemble Models for ILI Case Count Predictions Prithwish Chakraborty 1,2, Pejman Khadivi 1,2, Bryan Lewis 3, Aravindan Mahendiran 1,2, Jiangzhuo Chen 3, Patrick Butler 1,2, Elaine O. Nsoesie 3,4,5, Sumiko R. Mekaru 4,5, John S. Brownstein 4,5, Madhav V. Marathe 3, Naren Ramakrishnan 1,2 1 Dept. of Computer Science, Virginia Tech, USA 2 Discovery Analytics Center, Virginia Tech, USA 3 NDSSL, Virginia Bioinformatics Institute, USA 4 Children s Hopsital Informatics Program, Boston Children s Hopsital, USA 5 Dept. of Pediatrics, Harvard Medical School, USA. October 26, / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

2 2 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

3 Problem Overview 3 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

4 Problem Overview Predicting weekly Influenza-like-illness (ILI) case counts for 15 Latin American countries Investigating different open source data-streams as possible surrogate indicators of ILI 3 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

5 Motivation Traditional methods are often not enough!! ILI surveillance is not real-time - often lags several weeks Estimates are unstable - often revised over several months 4 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

6 Motivation Traditional methods are often not enough!! ILI surveillance is not real-time - often lags several weeks Estimates are unstable - often revised over several months Can surrogate information be used to provide more stable and real time estimates? Either non-physical indicators or physical indicators investigated How to handle the instability associated with ILI surveillance. 4 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

7 Key Contributions 1 Real-time prospective study - most studies till now have been retrospective 5 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

8 Key Contributions 1 Real-time prospective study - most studies till now have been retrospective 2 Integrates both social and physical indicators 5 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

9 Key Contributions 1 Real-time prospective study - most studies till now have been retrospective 2 Integrates both social and physical indicators 3 Data level fusion vs Model level fusion? 5 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

10 Key Contributions 1 Real-time prospective study - most studies till now have been retrospective 2 Integrates both social and physical indicators 3 Data level fusion vs Model level fusion? 4 Accounting for uncertainties in the official surveillance estimates 5 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

11 Key Contributions 1 Real-time prospective study - most studies till now have been retrospective 2 Integrates both social and physical indicators 3 Data level fusion vs Model level fusion? 4 Accounting for uncertainties in the official surveillance estimates 5 Investigate importance of different sources - Ablation test 5 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

12 Overall Framework Healthmap6Data 7P6MB6Historical P:56MBI6per6week Twitter6Data 5LLGB6Historical PL6GBI6per6week Weather6Data P6GB6historical P86MBI6week Google6Trends 6LL6MB6historical 86MBI6week OpenTable6Res6Data PP6MB6historical P766KBI6week Google6Flu6Trends 46MB6historical PLL6KBI6week Data6Enrichment Healthmap6 Data P4L6MB6hist: b6mbi6week Twitter6Data PTB6hist: OL6GBI6week Healthmap6Data66:666POMB6 Weather6Data666666:66665L6MB Twitter6Data :666676GB66 Filtering6for6 Flu6Related6Content Time6series6Surrogate Extraction Healthmap6Data666:666PL6KB Weather6Data :666P56KB Twitter6Data :666PL6KB 6 ILI6Prediction 6/1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

13 Data Sources Non-physical indicators 7 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

14 Data Sources Non-physical indicators 1 Google Flu Trends - uses unpublished set of keywords 7 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

15 Data Sources Non-physical indicators 1 Google Flu Trends - uses unpublished set of keywords 2 Custom User Keywords 1 Google Search Trends 2 Healthmap News Feed 3 Twitter Feed 7 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

16 Data Sources Non-physical indicators 1 Google Flu Trends - uses unpublished set of keywords 2 Custom User Keywords 1 Google Search Trends 2 Healthmap News Feed 3 Twitter Feed Physical indicators Misc. Indicators 1 Opentable reservations 7 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

17 Google Flu Trends 8 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

18 Finding Custom user keyword dictionary A multiple step process : Started with a seed set of keywords from experts. Seed set contains words in Spanish, Portuguese, and English. example : gripe (flu in Spanish) 9 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

19 Finding Custom user keyword dictionary A multiple step process : Started with a seed set of keywords from experts. Seed set contains words in Spanish, Portuguese, and English. example : gripe (flu in Spanish) Pseudo-query expansion Crawled top 20 web-sites for each seed word. Crawled expert web-sites e.g. CDC. Crawled few other hand-picked sites. Top 500 frequently occurring words selected. 9 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

20 Finding Custom user keyword dictionary A multiple step process : Started with a seed set of keywords from experts. Seed set contains words in Spanish, Portuguese, and English. example : gripe (flu in Spanish) Pseudo-query expansion Crawled top 20 web-sites for each seed word. Crawled expert web-sites e.g. CDC. Crawled few other hand-picked sites. Top 500 frequently occurring words selected. Time series correlation analysis Used Google Correlate to find words with search history correlated with ILI incidence curve. Interesting words such as ginger and Acemuk found. 9 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

21 Finding Custom user keyword dictionary A multiple step process : Started with a seed set of keywords from experts. Seed set contains words in Spanish, Portuguese, and English. example : gripe (flu in Spanish) Pseudo-query expansion Crawled top 20 web-sites for each seed word. Crawled expert web-sites e.g. CDC. Crawled few other hand-picked sites. Top 500 frequently occurring words selected. Time series correlation analysis Used Google Correlate to find words with search history correlated with ILI incidence curve. Interesting words such as ginger and Acemuk found. Final filtering : 114 words 9 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

22 GFT vs other non-physical indicators using custom keyword set Google Search Trends (GST) Healthmap Twitter 10 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

23 Physical Indicators Meteorological data for every lat-long, worldwide, every 8 hours Humidity, Temperature, Rainfall Analyzing grid cells covering PAHO sites. 11 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

24 System framework once again!! Healthmap6Data 7P6MB6Historical P:56MBI6per6week Twitter6Data 5LLGB6Historical PL6GBI6per6week Weather6Data P6GB6historical P86MBI6week Google6Trends 6LL6MB6historical 86MBI6week OpenTable6Res6Data PP6MB6historical P766KBI6week Google6Flu6Trends 46MB6historical PLL6KBI6week Data6Enrichment Healthmap6 Data P4L6MB6hist: b6mbi6week Twitter6Data PTB6hist: OL6GBI6week Healthmap6Data66:666POMB6 Weather6Data666666:66665L6MB Twitter6Data :666676GB66 Filtering6for6 Flu6Related6Content Time6series6Surrogate Extraction Healthmap6Data666:666PL6KB Weather6Data :666P56KB Twitter6Data :666PL6KB 6 ILI6Prediction 12 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

25 Preliminaries To find predictive modelf Variable Setup f : P t = f (P, X ) V t P t β α, X t β α, P t+1 β α, X t+1 β α,..., P t α, X t α L t P t Parameters α : the lookahead window length β : the lookback window length 13 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

26 Matrix Factorization (MF) Can find latent factors in the dataset. 14 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

27 Matrix Factorization (MF) Can find latent factors in the dataset. Model M i,j = b u,i + U T i F j b i,j = M + b j 14 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

28 Matrix Factorization (MF) Can find latent factors in the dataset. Model Fitting M i,j = b u,i + U T i F j b i,j = M + b j b, F, U = argmin( m 1 i=1 (M i,n M i,n ) 2 +λ 1 ( n b 2 j + m 1 U i 2 + n F j 2 )) j=1 i=1 j=1 (1) 14 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

29 Nearest Neighbor model (NN) Impose non-linearity. 15 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

30 Nearest Neighbor model (NN) Impose non-linearity. N (i) = {k : V k is one of the top K nearest neighbors of V i } 15 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

31 Nearest Neighbor model (NN) Impose non-linearity. N (i) = {k : V k is one of the top K nearest neighbors of V i } Fitting P T = ( k N (T ) θ k L k,t α )/ K θ k (2) k=1 15 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

32 Matrix Factorization using Nearest Neighborhood (MFN) Inspired from Koren et al. s work in Recommender systems. 16 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

33 Matrix Factorization using Nearest Neighborhood (MFN) Inspired from Koren et al. s work in Recommender systems. M i,j = b i,j + Ui T F j +F j N (i) 1 2 k N(i) (M (3) i,k b i,k )x k 16 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

34 Matrix Factorization using Nearest Neighborhood (MFN) Inspired from Koren et al. s work in Recommender systems. M i,j = b i,j + Ui T F j +F j N (i) 1 2 k N(i) (M (3) i,k b i,k )x k Fitting b, F, U, x = argmin( m 1 i=1 (M i,n M i,n ) 2 +λ 2 ( n b 2 j + m 1 U i 2 + n F j 2 + x k 2 )) j=1 i=1 j=1 k (4) koren2008factor 16 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

35 Accuracy comparison Quality Metric ( A = 4 t e P t 1 ˆP ) t N p max(p t, ˆP t, 10) t=t s (5) 17 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

36 Accuracy comparison 18 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

37 Accuracy comparison On average, MFN has better performance over MF and NN In Mexico, MF has the best accuracy - possibly because the 2013 ILI season in Mexico was late by a few weeks than in usual. 18 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

38 Model level fusion Output from models combined based on historical accuracy. 19 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

39 Model level fusion Output from models combined based on historical accuracy. Model [ CM t = 1 P t... C P t P t ] (6) 19 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

40 Model level fusion Output from models combined based on historical accuracy. Model [ CM t = 1 P t... C P t P t ] (6) Fitting C M i,j = µ i + C b j + C Ui T CF j + C F j C N (i) 1 2 k C N(i) ( (7) CM i,k µ i + C b k ) C x k 19 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

41 Data level fusion Feature vector is a tuple over all data set features. Use MFN to fit the value X t = T t, W t 20 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

42 Accuracy comparison 21 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

43 Accuracy comparison On average, model level fusion produces better accuracy than data level fusion. Interesting deviations like Chile and El Salvador indicates that a possible strategy could be a mix of data level and model fusion - however complexity of training will increase manifold. 21 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

44 Uncertainty in official estimates Can take up to several months to stabilize. Average relative error of PAHO count values with respect to stable values. (a) Comparison between Argentina and Colombia (b) Comparison between different seasons for Argentina. 22 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

45 Correcting uncertainty Recognize high, low and mid-season months for countries. Variable setup { P S A = (1, P (1) i, P i, N (1) i ),..., (m, P (m) i, P } i, N (m) i ),... Correction Model P ˆ (m) i = a 0 + a 1 m + a 2 P (m) i + a 3 N (m) i (8) 23 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

46 Correcting uncertainty Recognize high, low and mid-season months for countries. Variable setup { P S A = (1, P (1) i, P i, N (1) i ),..., (m, P (m) i, P } i, N (m) i ),... Correction Model P ˆ (m) i = a 0 + a 1 m + a 2 P (m) i + a 3 N (m) i (8) 23 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target

47 Investigating importance of each source : Ablation Test 24 / 1 Prithwish Chakraborty (prithwi@vt.edu) Forecasting a Moving Target