Rob J Hyndman. Visualizing and forecasting. big time series data. Victoria: scaled

Transcription

1 Rob J Hyndman Visualizing and forecasting big time series data Victoria: scaled

2 Outline 1 Examples of biggish time series 2 Time series visualisation 3 BLUF: Best Linear Unbiased Forecasts 4 Application: Australian tourism 5 Fast computation tricks 6 hts package for R 7 References Visualising and forecasting big time series data Examples of biggish time series 2

3 1. Australian tourism demand Visualising and forecasting big time series data Examples of biggish time series 3

4 1. Australian tourism demand Quarterly data on visitor night from 1998:Q1 2013:Q4 From: National Visitor Survey, based on annual interviews of 120,000 Australians aged 15+, collected by Tourism Research Australia. Split by 7 states, 27 zones and 76 regions (a geographical hierarchy) Also split by purpose of travel Holiday Visiting friends and relatives (VFR) Business Other 304 bottom-level series Visualising and forecasting big time series data Examples of biggish time series 3

5 2. Labour market participation Australia and New Zealand Standard Classification of Occupations 8 major groups 43 sub-major groups 97 minor groups 359 unit groups * 1023 occupations Example: statistician 2 Professionals 22 Business, Human Resource and Marketing Professionals 224 Information and Organisation Professionals 2241 Actuaries, Mathematicians and Statisticians Statistician Visualising and forecasting big time series data Examples of biggish time series 4

6 2. Labour market participation Australia and New Zealand Standard Classification of Occupations 8 major groups 43 sub-major groups 97 minor groups 359 unit groups * 1023 occupations Example: statistician 2 Professionals 22 Business, Human Resource and Marketing Professionals 224 Information and Organisation Professionals 2241 Actuaries, Mathematicians and Statisticians Statistician Visualising and forecasting big time series data Examples of biggish time series 4

7 3. Spectacle sales Monthly UK sales data from Provided by a large spectacle manufacturer Split by brand (26), gender (3), price range (6), materials (4), and stores (600) About 1 million bottom-level series Visualising and forecasting big time series data Examples of biggish time series 5

8 3. Spectacle sales Monthly UK sales data from Provided by a large spectacle manufacturer Split by brand (26), gender (3), price range (6), materials (4), and stores (600) About 1 million bottom-level series Visualising and forecasting big time series data Examples of biggish time series 5

9 3. Spectacle sales Monthly UK sales data from Provided by a large spectacle manufacturer Split by brand (26), gender (3), price range (6), materials (4), and stores (600) About 1 million bottom-level series Visualising and forecasting big time series data Examples of biggish time series 5

10 3. Spectacle sales Monthly UK sales data from Provided by a large spectacle manufacturer Split by brand (26), gender (3), price range (6), materials (4), and stores (600) About 1 million bottom-level series Visualising and forecasting big time series data Examples of biggish time series 5

11 Outline 1 Examples of biggish time series 2 Time series visualisation 3 BLUF: Best Linear Unbiased Forecasts 4 Application: Australian tourism 5 Fast computation tricks 6 hts package for R 7 References Visualising and forecasting big time series data Time series visualisation 6

12 Kite diagrams Line graph profile 0 0 Duplicate & flip around the horizontal axis 0 Fill the colour Visualising and forecasting big time series data Time series visualisation 7

13 Kite diagrams: Victorian tourism Victoria BAA BAB BAC BBA BCA BCB BCC BDA BDB BDC BDD BDE BDF BEA BEB BEC BED BEE BEF BEG Other Business VFR Holiday Visualising and forecasting big time series data Time series visualisation 8

14 Kite diagrams: Victorian tourism Visualising and forecasting big time series data Time series visualisation 8

15 Kite diagrams: Victorian tourism Visualising and forecasting big time series data Time series visualisation 8

16 Kite diagrams: Victorian tourism Victoria: scaled BAA BAB BAC BBA BCA BCB BCC BDA BDB BDC BDD BDE BDF BEA BEB BEC BED BEE BEF BEG Other Business VFR Holiday Visualising and forecasting big time series data Time series visualisation 8

17 An STL decomposition STL decomposition of tourism demand for holidays in Peninsula remainder trend seasonal data Visualising and forecasting big time series data Time series visualisation 9

18 Seasonal stacked bar chart Place positive values above the origin while negative values below the origin Map the bar length to the magnitude Encode quarters by colours Visualising and forecasting big time series data Time series visualisation 10

19 Seasonal stacked bar chart Place positive values above the origin while negative values below the origin Map the bar length to the magnitude Encode quarters by colours 1.0 Seasonal Component Holiday Qtr Q1 Q2 Q3 Q4 BAA BABBACBBABCABCBBCCBDABDBBDCBDDBDEBDF BEA BEBBECBEDBEE BEFBEG Regions Visualising and forecasting big time series data Time series visualisation 10

20 Seasonal stacked bar chart: VIC Visualising and forecasting big time series data Time series visualisation 11

21 Corrgram of remainder Compute the correlations among the remainder components Render both the sign and magnitude using a colour mapping of two hues Order variables according to the first principal component of the correlations. Visualising and forecasting big time series data Time series visualisation 12

22 Corrgram of remainder: VIC Visualising and forecasting big time series data Time series visualisation BEEHol BEFOth BEEOth BDEOth BEBOth BEABus BEFBus BDCOth BACHol BEBBus BEAVis BBAHol BDEHol BABOth BAAVis BAAHol BDCHol BBABus BCBHol BEGBus BDDVis BABVis BDAVis BEAOth BDFHol BEEBus BAAOth BACOth BDAOth BDEBus BCBOth BACBus BEBVis BACVis BCAOth BEFVis BCBVis BEDHol BEGOth BDBHol BABBus BEBHol BDFBus BECHol BCAHol BDBOth BEAHol BDCBus BECVis BDBVis BCCHol BBAVis BABHol BBAOth BCCOth BCBBus BCCVis BEGVis BDDHol BECOth BDCVis BAABus BCCBus BECBus BCAVis BDFVis BEGHol BDDOth BEDOth BEDVis BDDBus BDEVis BEFHol BEEVis BDBBus BDABus BDAHol BCABus BDFOth BEDBus BEEHol BEFOth BEEOth BDEOth BEBOth BEABus BEFBus BDCOth BACHol BEBBus BEAVis BBAHol BDEHol BABOth BAAVis BAAHol BDCHol BBABus BCBHol BEGBus BDDVis BABVis BDAVis BEAOth BDFHol BEEBus BAAOth BACOth BDAOth BDEBus BCBOth BACBus BEBVis BACVis BCAOth BEFVis BCBVis BEDHol BEGOth BDBHol BABBus BEBHol BDFBus BECHol BCAHol BDBOth BEAHol BDCBus BECVis BDBVis BCCHol BBAVis BABHol BBAOth BCCOth BCBBus BCCVis BEGVis BDDHol BECOth BDCVis BAABus BCCBus BECBus BCAVis BDFVis BEGHol BDDOth BEDOth BEDVis BDDBus BDEVis BEFHol BEEVis BDBBus BDABus BDAHol BCABus BDFOth BEDBus

23 Corrgram of remainder: TAS FCAHol FBBHol FBAHol FAAHol FCBHol FCAVis FBBVis FAAVis FCBBus FAAOth FCAOth FBBOth FBABus FBAOth FCBVis FCABus FBAVis FCBOth FBBBus FCAHol FBBHol FBAHol FAAHol FCBHol FCAVis FBBVis FAAVis FCBBus FAAOth FCAOth FBBOth FBABus FBAOth FCBVis FCABus FBAVis FCBOth FBBBus FAABus FAABus 1 Visualising and forecasting big time series data Time series visualisation 14

24 Feature analysis Summarize each time series with a feature vector: strength of trend lumpiness (variance of annual variances of remainder) strength of seasonality size of seasonal peak size of seasonal trough ACF1 linearity of trend curvature of trend spectral entropy Do PCA on feature matrix Visualising and forecasting big time series data Time series visualisation 15

25 Feature analysis standardized PC2 (18.6% explained var.) season trough peak fo.acf linearity curvature lump spike entropy standardized PC1 (43.2% explained var.) Visualising and forecasting big time series data Time series visualisation 16

26 Feature analysis 800 value BEGBus BCCBus CCAOth CACOth Time Visualising and forecasting big time series data Time series visualisation 16

27 Feature analysis 1600 value ADAHol BDCHol BBAHol ACAHol Time Visualising and forecasting big time series data Time series visualisation 16

28 Feature analysis Visualising and forecasting big time series data Time series visualisation 16 NSW NT QLD SA TAS VIC WA PC1 PC2

29 Feature analysis Visualising and forecasting big time series data Time series visualisation 16 Bus Hol Oth Vis PC1 PC2

30 Outline 1 Examples of biggish time series 2 Time series visualisation 3 BLUF: Best Linear Unbiased Forecasts 4 Application: Australian tourism 5 Fast computation tricks 6 hts package for R 7 References Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 17

31 Hierarchical time series A hierarchical time series is a collection of several time series that are linked together in a hierarchical structure. Total A AA AB AC B BA BB BC C CA CB CC Examples Net labour turnover Tourism by state and region Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 18

32 Hierarchical time series A hierarchical time series is a collection of several time series that are linked together in a hierarchical structure. Total A AA AB AC B BA BB BC C CA CB CC Examples Net labour turnover Tourism by state and region Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 18

33 Hierarchical time series A hierarchical time series is a collection of several time series that are linked together in a hierarchical structure. Total A AA AB AC B BA BB BC C CA CB CC Examples Net labour turnover Tourism by state and region Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 18

34 Hierarchical time series Total A B C Y t : observed aggregate of all series at time t. Y X,t : observation on series X at time t. B t : vector of all series at bottom level in time t. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 19

35 Hierarchical time series Total A B C Y t : observed aggregate of all series at time t. Y X,t : observation on series X at time t. B t : vector of all series at bottom level in time t. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 19

36 Hierarchical time series Total A B C y t = [Y t, Y A,t, Y B,t, Y C,t ] = Y t : observed aggregate of all series at time t. Y X,t : observation on series X at time t. B t : vector of all series at bottom level in time t Y A,t Y B,t Y C,t Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 19

37 Hierarchical time series Total A B C y t = [Y t, Y A,t, Y B,t, Y C,t ] = Y t : observed aggregate of all series at time t. Y X,t : observation on series X at time t. B t : vector of all series at bottom level in time t Y A,t Y B,t Y C,t }{{} S Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 19

38 Hierarchical time series Total A B C y t = [Y t, Y A,t, Y B,t, Y C,t ] = Y t : observed aggregate of all series at time t. Y X,t : observation on series X at time t. B t : vector of all series at bottom level in time t Y A,t Y B,t Y C,t }{{}}{{} B t S Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 19

39 Hierarchical time series Total A B C y t = [Y t, Y A,t, Y B,t, Y C,t ] = y t = SB t Y t : observed aggregate of all series at time t. Y X,t : observation on series X at time t. B t : vector of all series at bottom level in time t Y A,t Y B,t Y C,t }{{}}{{} B t S Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 19

40 Hierarchical time series Total A B C y t = AX AY AZ Y t Y A,t Y B,t Y C,t Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t = BX BY BZ } {{ } S Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t }{{} B t CX CY CZ Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 20

41 Hierarchical time series Total A B C y t = AX AY AZ Y t Y A,t Y B,t Y C,t Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t = BX BY BZ } {{ } S Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t }{{} B t CX CY CZ Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 20

42 Hierarchical time series Total A B C y t = AX AY AZ Y t Y A,t Y B,t Y C,t Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t = BX BY BZ } {{ } S Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t }{{} B t CX CY CZ y t = SB t Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 20

43 Forecasting notation Let ŷ n (h) be vector of initial h-step forecasts, made at time n, stacked in same order as y t. (They may not add up.) Reconciled forecasts are of the form: ỹ n (h) = SPŷ n (h) for some matrix P. P extracts and combines base forecasts ŷ n (h) to get bottom-level forecasts. S adds them up Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 21

44 Forecasting notation Let ŷ n (h) be vector of initial h-step forecasts, made at time n, stacked in same order as y t. (They may not add up.) Reconciled forecasts are of the form: ỹ n (h) = SPŷ n (h) for some matrix P. P extracts and combines base forecasts ŷ n (h) to get bottom-level forecasts. S adds them up Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 21

45 Forecasting notation Let ŷ n (h) be vector of initial h-step forecasts, made at time n, stacked in same order as y t. (They may not add up.) Reconciled forecasts are of the form: ỹ n (h) = SPŷ n (h) for some matrix P. P extracts and combines base forecasts ŷ n (h) to get bottom-level forecasts. S adds them up Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 21

46 Forecasting notation Let ŷ n (h) be vector of initial h-step forecasts, made at time n, stacked in same order as y t. (They may not add up.) Reconciled forecasts are of the form: ỹ n (h) = SPŷ n (h) for some matrix P. P extracts and combines base forecasts ŷ n (h) to get bottom-level forecasts. S adds them up Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 21

47 Forecasting notation Let ŷ n (h) be vector of initial h-step forecasts, made at time n, stacked in same order as y t. (They may not add up.) Reconciled forecasts are of the form: ỹ n (h) = SPŷ n (h) for some matrix P. P extracts and combines base forecasts ŷ n (h) to get bottom-level forecasts. S adds them up Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 21

48 General properties: bias ỹ n (h) = SPŷ n (h) Assume: base forecasts ŷ n (h) are unbiased: E[ŷ n (h) y 1,..., y n ] = E[y n+h y 1,..., y n ] Let ˆBn (h) be bottom level base forecasts with β n (h) = E[ˆBn (h) y 1,..., y n ]. Then E[ŷ n (h)] = Sβ n (h). We want the revised forecasts to be unbiased: E[ỹ n (h)] = SPSβ n (h) = Sβ n (h). Revised forecasts are unbiased iff SPS = S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 22

49 General properties: bias ỹ n (h) = SPŷ n (h) Assume: base forecasts ŷ n (h) are unbiased: E[ŷ n (h) y 1,..., y n ] = E[y n+h y 1,..., y n ] Let ˆBn (h) be bottom level base forecasts with β n (h) = E[ˆBn (h) y 1,..., y n ]. Then E[ŷ n (h)] = Sβ n (h). We want the revised forecasts to be unbiased: E[ỹ n (h)] = SPSβ n (h) = Sβ n (h). Revised forecasts are unbiased iff SPS = S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 22

50 General properties: bias ỹ n (h) = SPŷ n (h) Assume: base forecasts ŷ n (h) are unbiased: E[ŷ n (h) y 1,..., y n ] = E[y n+h y 1,..., y n ] Let ˆBn (h) be bottom level base forecasts with β n (h) = E[ˆBn (h) y 1,..., y n ]. Then E[ŷ n (h)] = Sβ n (h). We want the revised forecasts to be unbiased: E[ỹ n (h)] = SPSβ n (h) = Sβ n (h). Revised forecasts are unbiased iff SPS = S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 22

51 General properties: bias ỹ n (h) = SPŷ n (h) Assume: base forecasts ŷ n (h) are unbiased: E[ŷ n (h) y 1,..., y n ] = E[y n+h y 1,..., y n ] Let ˆBn (h) be bottom level base forecasts with β n (h) = E[ˆBn (h) y 1,..., y n ]. Then E[ŷ n (h)] = Sβ n (h). We want the revised forecasts to be unbiased: E[ỹ n (h)] = SPSβ n (h) = Sβ n (h). Revised forecasts are unbiased iff SPS = S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 22

52 General properties: bias ỹ n (h) = SPŷ n (h) Assume: base forecasts ŷ n (h) are unbiased: E[ŷ n (h) y 1,..., y n ] = E[y n+h y 1,..., y n ] Let ˆBn (h) be bottom level base forecasts with β n (h) = E[ˆBn (h) y 1,..., y n ]. Then E[ŷ n (h)] = Sβ n (h). We want the revised forecasts to be unbiased: E[ỹ n (h)] = SPSβ n (h) = Sβ n (h). Revised forecasts are unbiased iff SPS = S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 22

53 General properties: bias ỹ n (h) = SPŷ n (h) Assume: base forecasts ŷ n (h) are unbiased: E[ŷ n (h) y 1,..., y n ] = E[y n+h y 1,..., y n ] Let ˆBn (h) be bottom level base forecasts with β n (h) = E[ˆBn (h) y 1,..., y n ]. Then E[ŷ n (h)] = Sβ n (h). We want the revised forecasts to be unbiased: E[ỹ n (h)] = SPSβ n (h) = Sβ n (h). Revised forecasts are unbiased iff SPS = S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 22

54 General properties: bias ỹ n (h) = SPŷ n (h) Assume: base forecasts ŷ n (h) are unbiased: E[ŷ n (h) y 1,..., y n ] = E[y n+h y 1,..., y n ] Let ˆBn (h) be bottom level base forecasts with β n (h) = E[ˆBn (h) y 1,..., y n ]. Then E[ŷ n (h)] = Sβ n (h). We want the revised forecasts to be unbiased: E[ỹ n (h)] = SPSβ n (h) = Sβ n (h). Revised forecasts are unbiased iff SPS = S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 22

55 General properties: variance ỹ n (h) = SPŷ n (h) Let variance of base forecasts ŷ n (h) be given by Σ h = Var[ŷ n (h) y 1,..., y n ] Then the variance of the revised forecasts is given by Var[ỹ n (h) y 1,..., y n ] = SPΣ h P S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 23

56 General properties: variance ỹ n (h) = SPŷ n (h) Let variance of base forecasts ŷ n (h) be given by Σ h = Var[ŷ n (h) y 1,..., y n ] Then the variance of the revised forecasts is given by Var[ỹ n (h) y 1,..., y n ] = SPΣ h P S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 23

57 General properties: variance ỹ n (h) = SPŷ n (h) Let variance of base forecasts ŷ n (h) be given by Σ h = Var[ŷ n (h) y 1,..., y n ] Then the variance of the revised forecasts is given by Var[ỹ n (h) y 1,..., y n ] = SPΣ h P S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 23

58 BLUF via trace minimization Theorem For any P satisfying SPS = S, then min P = trace[spσ h P S ] has solution P = (S Σ h S) 1 S Σ h. Σ h is generalized inverse of Σ h. Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Equivalent to GLS estimate of regression ŷ n (h) = Sβ n (h) + ε h where ε N(0, Σ h ). Problem: Σ h hard to estimate. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 24

59 BLUF via trace minimization Theorem For any P satisfying SPS = S, then min P = trace[spσ h P S ] has solution P = (S Σ h S) 1 S Σ h. Σ h is generalized inverse of Σ h. Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Equivalent to GLS estimate of regression ŷ n (h) = Sβ n (h) + ε h where ε N(0, Σ h ). Problem: Σ h hard to estimate. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 24

60 BLUF via trace minimization Theorem For any P satisfying SPS = S, then min P = trace[spσ h P S ] has solution P = (S Σ h S) 1 S Σ h. Σ h is generalized inverse of Σ h. Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Equivalent to GLS estimate of regression ŷ n (h) = Sβ n (h) + ε h where ε N(0, Σ h ). Problem: Σ h hard to estimate. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 24

61 BLUF via trace minimization Theorem For any P satisfying SPS = S, then min P = trace[spσ h P S ] has solution P = (S Σ h S) 1 S Σ h. Σ h is generalized inverse of Σ h. Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Equivalent to GLS estimate of regression ŷ n (h) = Sβ n (h) + ε h where ε N(0, Σ h ). Problem: Σ h hard to estimate. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 24

62 BLUF via trace minimization Theorem For any P satisfying SPS = S, then min P = trace[spσ h P S ] has solution P = (S Σ h S) 1 S Σ h. Σ h is generalized inverse of Σ h. Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Equivalent to GLS estimate of regression ŷ n (h) = Sβ n (h) + ε h where ε N(0, Σ h ). Problem: Σ h hard to estimate. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 24

63 BLUF via trace minimization Theorem For any P satisfying SPS = S, then min P = trace[spσ h P S ] has solution P = (S Σ h S) 1 S Σ h. Σ h is generalized inverse of Σ h. Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Equivalent to GLS estimate of regression ŷ n (h) = Sβ n (h) + ε h where ε N(0, Σ h ). Problem: Σ h hard to estimate. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 24

64 Optimal combination forecasts Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Solution 1: OLS Assume ε h Sε B,h where ε B,h is the forecast error at bottom level. Then Σ h SΩ h S where Ω h = Var(ε B,h ). If Moore-Penrose generalized inverse used, then (S Σ h S) 1 S Σ h = (S S) 1 S. ỹ n (h) = S(S S) 1 S ŷ n (h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 25

65 Optimal combination forecasts Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Solution 1: OLS Assume ε h Sε B,h where ε B,h is the forecast error at bottom level. Then Σ h SΩ h S where Ω h = Var(ε B,h ). If Moore-Penrose generalized inverse used, then (S Σ h S) 1 S Σ h = (S S) 1 S. ỹ n (h) = S(S S) 1 S ŷ n (h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 25

66 Optimal combination forecasts Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Solution 1: OLS Assume ε h Sε B,h where ε B,h is the forecast error at bottom level. Then Σ h SΩ h S where Ω h = Var(ε B,h ). If Moore-Penrose generalized inverse used, then (S Σ h S) 1 S Σ h = (S S) 1 S. ỹ n (h) = S(S S) 1 S ŷ n (h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 25

67 Optimal combination forecasts Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Solution 1: OLS Assume ε h Sε B,h where ε B,h is the forecast error at bottom level. Then Σ h SΩ h S where Ω h = Var(ε B,h ). If Moore-Penrose generalized inverse used, then (S Σ h S) 1 S Σ h = (S S) 1 S. ỹ n (h) = S(S S) 1 S ŷ n (h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 25

68 Optimal combination forecasts Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Solution 1: OLS Assume ε h Sε B,h where ε B,h is the forecast error at bottom level. Then Σ h SΩ h S where Ω h = Var(ε B,h ). If Moore-Penrose generalized inverse used, then (S Σ h S) 1 S Σ h = (S S) 1 S. ỹ n (h) = S(S S) 1 S ŷ n (h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 25

69 Optimal combination forecasts Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Solution 1: OLS Assume ε h Sε B,h where ε B,h is the forecast error at bottom level. Then Σ h SΩ h S where Ω h = Var(ε B,h ). If Moore-Penrose generalized inverse used, then (S Σ h S) 1 S Σ h = (S S) 1 S. ỹ n (h) = S(S S) 1 S ŷ n (h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 25

70 Optimal combination forecasts Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Solution 2: WLS Suppose we approximate Σ 1 by its diagonal. Easy to estimate, and places weight where we have best forecasts. Empirically, it gives better forecasts than other available methods. ỹ n (h) = S(S ΛS) 1 S Λŷ n (h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26

71 Optimal combination forecasts Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Solution 2: WLS Suppose we approximate Σ 1 by its diagonal. Easy to estimate, and places weight where we have best forecasts. Empirically, it gives better forecasts than other available methods. ỹ n (h) = S(S ΛS) 1 S Λŷ n (h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26

72 Optimal combination forecasts Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Solution 2: WLS Suppose we approximate Σ 1 by its diagonal. Easy to estimate, and places weight where we have best forecasts. Empirically, it gives better forecasts than other available methods. ỹ n (h) = S(S ΛS) 1 S Λŷ n (h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26

73 Optimal combination forecasts Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Solution 2: WLS Suppose we approximate Σ 1 by its diagonal. Easy to estimate, and places weight where we have best forecasts. Empirically, it gives better forecasts than other available methods. ỹ n (h) = S(S ΛS) 1 S Λŷ n (h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26

74 Optimal combination forecasts Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Solution 2: WLS Suppose we approximate Σ 1 by its diagonal. Easy to estimate, and places weight where we have best forecasts. Empirically, it gives better forecasts than other available methods. ỹ n (h) = S(S ΛS) 1 S Λŷ n (h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26

75 Optimal combination forecasts Revised forecasts ỹ n (h) = S(S Σ h S) 1 S Σ hŷn(h) Base forecasts Solution 2: WLS Suppose we approximate Σ 1 by its diagonal. Easy to estimate, and places weight where we have best forecasts. Empirically, it gives better forecasts than other available methods. ỹ n (h) = S(S ΛS) 1 S Λŷ n (h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26

76 Challenges ỹ n (h) = S(S ΛS) 1 S Λŷ n (h) Computational difficulties in big hierarchies due to size of the S matrix and singular behavior of (S ΛS). Loss of information in ignoring covariance matrix in computing point forecasts. Still need to estimate covariance matrix to produce prediction intervals. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 27

77 Challenges ỹ n (h) = S(S ΛS) 1 S Λŷ n (h) Computational difficulties in big hierarchies due to size of the S matrix and singular behavior of (S ΛS). Loss of information in ignoring covariance matrix in computing point forecasts. Still need to estimate covariance matrix to produce prediction intervals. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 27

78 Challenges ỹ n (h) = S(S ΛS) 1 S Λŷ n (h) Computational difficulties in big hierarchies due to size of the S matrix and singular behavior of (S ΛS). Loss of information in ignoring covariance matrix in computing point forecasts. Still need to estimate covariance matrix to produce prediction intervals. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 27

79 Outline 1 Examples of biggish time series 2 Time series visualisation 3 BLUF: Best Linear Unbiased Forecasts 4 Application: Australian tourism 5 Fast computation tricks 6 hts package for R 7 References Visualising and forecasting big time series data Application: Australian tourism 28

80 Australian tourism Visualising and forecasting big time series data Application: Australian tourism 29

81 Australian tourism Hierarchy: States (7) Zones (27) Regions (82) Visualising and forecasting big time series data Application: Australian tourism 29

82 Australian tourism Hierarchy: States (7) Zones (27) Regions (82) Base forecasts ETS (exponential smoothing) models Visualising and forecasting big time series data Application: Australian tourism 29

83 Base forecasts Domestic tourism forecasts: Total Visitor nights Year Visualising and forecasting big time series data Application: Australian tourism 30

84 Base forecasts Domestic tourism forecasts: NSW Visitor nights Year Visualising and forecasting big time series data Application: Australian tourism 30

85 Base forecasts Domestic tourism forecasts: VIC Visitor nights Year Visualising and forecasting big time series data Application: Australian tourism 30

86 Base forecasts Domestic tourism forecasts: Nth.Coast.NSW Visitor nights Year Visualising and forecasting big time series data Application: Australian tourism 30

87 Base forecasts Domestic tourism forecasts: Metro.QLD Visitor nights Year Visualising and forecasting big time series data Application: Australian tourism 30

88 Base forecasts Domestic tourism forecasts: Sth.WA Visitor nights Year Visualising and forecasting big time series data Application: Australian tourism 30

89 Base forecasts Domestic tourism forecasts: X201.Melbourne Visitor nights Year Visualising and forecasting big time series data Application: Australian tourism 30

90 Base forecasts Domestic tourism forecasts: X402.Murraylands Visitor nights Year Visualising and forecasting big time series data Application: Australian tourism 30

91 Base forecasts Domestic tourism forecasts: X809.Daly Visitor nights Year Visualising and forecasting big time series data Application: Australian tourism 30

92 Reconciled forecasts Total Visualising and forecasting big time series data Application: Australian tourism 31

93 Reconciled forecasts NSW VIC QLD Other Visualising and forecasting big time series data Application: Australian tourism 31

94 Reconciled forecasts Sydney Other NSW Melbourne Other VIC GC and Brisbane Other QLD Capital cities Other Visualising and forecasting big time series data Application: Australian tourism 31

95 Forecast evaluation Select models using all observations; Re-estimate models using first 12 observations and generate 1- to 8-step-ahead forecasts; Increase sample size one observation at a time, re-estimate models, generate forecasts until the end of the sample; In total 24 1-step-ahead, 23 2-steps-ahead, up to 17 8-steps-ahead for forecast evaluation. Visualising and forecasting big time series data Application: Australian tourism 32

96 Forecast evaluation Select models using all observations; Re-estimate models using first 12 observations and generate 1- to 8-step-ahead forecasts; Increase sample size one observation at a time, re-estimate models, generate forecasts until the end of the sample; In total 24 1-step-ahead, 23 2-steps-ahead, up to 17 8-steps-ahead for forecast evaluation. Visualising and forecasting big time series data Application: Australian tourism 32

97 Forecast evaluation Select models using all observations; Re-estimate models using first 12 observations and generate 1- to 8-step-ahead forecasts; Increase sample size one observation at a time, re-estimate models, generate forecasts until the end of the sample; In total 24 1-step-ahead, 23 2-steps-ahead, up to 17 8-steps-ahead for forecast evaluation. Visualising and forecasting big time series data Application: Australian tourism 32

98 Forecast evaluation Select models using all observations; Re-estimate models using first 12 observations and generate 1- to 8-step-ahead forecasts; Increase sample size one observation at a time, re-estimate models, generate forecasts until the end of the sample; In total 24 1-step-ahead, 23 2-steps-ahead, up to 17 8-steps-ahead for forecast evaluation. Visualising and forecasting big time series data Application: Australian tourism 32

99 Hierarchy: states, zones, regions MAPE h = 1 h = 2 h = 4 h = 6 h = 8 Average Top Level: Australia Bottom-up OLS WLS Level: States Bottom-up OLS WLS Level: Zones Bottom-up OLS WLS Bottom Level: Regions Bottom-up OLS WLS Visualising and forecasting big time series data Application: Australian tourism 33

100 Outline 1 Examples of biggish time series 2 Time series visualisation 3 BLUF: Best Linear Unbiased Forecasts 4 Application: Australian tourism 5 Fast computation tricks 6 hts package for R 7 References Visualising and forecasting big time series data Fast computation tricks 34

101 Fast computation: hierarchical data Total A B C y t = AX AY AZ Y t Y A,t Y B,t Y C,t Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t = BX BY BZ } {{ } S Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t }{{} B t CX CY CZ y t = SB t Visualising and forecasting big time series data Fast computation tricks 35

102 Fast computation: hierarchical data Total A B C y t = AX AY AZ Y t Y A,t Y AX,t Y AY,t Y AZ,t Y B,t Y BX,t Y BY,t Y BZ,t Y C,t Y CX,t Y CY,t Y CZ,t = BX BY BZ } {{ } S Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t }{{} B t CX CY CZ y t = SB t Visualising and forecasting big time series data Fast computation tricks 36

103 Fast computation: hierarchies Think of the hierarchy as a tree of trees: Total T 1 T 2... T K Then the summing matrix contains k smaller summing matrices: S = 1 n 1 1 n 2 1 n K S S S K where 1 n is an n-vector of ones and tree T i has n i terminal nodes. Visualising and forecasting big time series data Fast computation tricks 37.

104 Fast computation: hierarchies Think of the hierarchy as a tree of trees: Total T 1 T 2... T K Then the summing matrix contains k smaller summing matrices: S = 1 n 1 1 n 2 1 n K S S S K where 1 n is an n-vector of ones and tree T i has n i terminal nodes. Visualising and forecasting big time series data Fast computation tricks 37.

105 Fast computation: hierarchies S ΛS = S 1Λ 1 S S 2Λ 2 S S KΛ K S K. +λ 0 J n λ 0 is the top left element of Λ; Λ k is a block of Λ, corresponding to tree T k ; J n is a matrix of ones; n = k n k. Now apply the Sherman-Morrison formula... Visualising and forecasting big time series data Fast computation tricks 38

106 Fast computation: hierarchies S ΛS = S 1Λ 1 S S 2Λ 2 S S KΛ K S K. +λ 0 J n λ 0 is the top left element of Λ; Λ k is a block of Λ, corresponding to tree T k ; J n is a matrix of ones; n = k n k. Now apply the Sherman-Morrison formula... Visualising and forecasting big time series data Fast computation tricks 38

107 Fast computation: hierarchies (S 1 Λ 1S 1 ) (S ΛS) 1 0 (S 2 = Λ 2S 2 ) cs (S K Λ KS K ) 1 S 0 can be partitioned into K 2 blocks, with the (k, l) block (of dimension n k n l ) being J nk,n l (S k Λ ks k ) 1 J nk,n l (S l Λ ls l ) 1 is a n k n l matrix of ones. c 1 = λ n k (S k Λ ks k ) 1 1 nk. k Each S k Λ ks k can be inverted similarly. S Λy can also be computed recursively. Visualising and forecasting big time series data Fast computation tricks 39

108 Fast computation: hierarchies (S 1 Λ 1S 1 ) (S ΛS) 1 0 (S 2 = Λ 2S 2 ) cs (S K Λ KS K ) 1 S 0 can be partitioned into K 2 blocks, with the (k, l) block (of dimension n k n l ) being The recursive (S k Λ calculations ks k ) 1 J nk,n l (S l Λ can ls l ) 1 be done in such a way that we never J nk,n l store is a n k any n l matrix of ones. of the large matrices c 1 = involved. λ n k (S k Λ ks k ) 1 1 nk. k Each S k Λ ks k can be inverted similarly. S Λy can also be computed recursively. Visualising and forecasting big time series data Fast computation tricks 39

109 Fast computation A similar algorithm has been developed for grouped time series with two groups. When the time series are not strictly hierarchical and have more than two grouping variables: Use sparse matrix storage and arithmetic. Use iterative approximation for inverting large sparse matrices. Paige & Saunders (1982) ACM Trans. Math. Software Visualising and forecasting big time series data Fast computation tricks 40

110 Fast computation A similar algorithm has been developed for grouped time series with two groups. When the time series are not strictly hierarchical and have more than two grouping variables: Use sparse matrix storage and arithmetic. Use iterative approximation for inverting large sparse matrices. Paige & Saunders (1982) ACM Trans. Math. Software Visualising and forecasting big time series data Fast computation tricks 40

111 Fast computation A similar algorithm has been developed for grouped time series with two groups. When the time series are not strictly hierarchical and have more than two grouping variables: Use sparse matrix storage and arithmetic. Use iterative approximation for inverting large sparse matrices. Paige & Saunders (1982) ACM Trans. Math. Software Visualising and forecasting big time series data Fast computation tricks 40

112 Outline 1 Examples of biggish time series 2 Time series visualisation 3 BLUF: Best Linear Unbiased Forecasts 4 Application: Australian tourism 5 Fast computation tricks 6 hts package for R 7 References Visualising and forecasting big time series data hts package for R 41

113 hts package for R hts: Hierarchical and grouped time series Methods for analysing and forecasting hierarchical and grouped time series Version: 4.5 Depends: forecast ( 5.0), SparseM Imports: parallel, utils Published: Author: Rob J Hyndman, Earo Wang and Alan Lee Maintainer: Rob J Hyndman <Rob.Hyndman at monash.edu> BugReports: License: GPL ( 2) Visualising and forecasting big time series data hts package for R 42

114 Example using R library(hts) # bts is a matrix containing the bottom level time series # nodes describes the hierarchical structure y <- hts(bts, nodes=list(2, c(3,2))) Visualising and forecasting big time series data hts package for R 43

115 Example using R library(hts) # bts is a matrix containing the bottom level time series # nodes describes the hierarchical structure y <- hts(bts, nodes=list(2, c(3,2))) Total A B AX AY AZ BX BY Visualising and forecasting big time series data hts package for R 43

116 Example using R library(hts) # bts is a matrix containing the bottom level time series # nodes describes the hierarchical structure y <- hts(bts, nodes=list(2, c(3,2))) # Forecast 10-step-ahead using WLS combination method # ETS used for each series by default fc <- forecast(y, h=10) Visualising and forecasting big time series data hts package for R 44

117 forecast.gts function Usage forecast(object, h, method = c("comb", "bu", "mo", "tdgsf", "tdgsa", "tdfp"), fmethod = c("ets", "rw", "arima"), weights = c("sd", "none", "nseries"), positive = FALSE, parallel = FALSE, num.cores = 2,...) Arguments object Hierarchical time series object of class gts. h Forecast horizon method Method for distributing forecasts within the hierarchy. fmethod Forecasting method to use positive If TRUE, forecasts are forced to be strictly positive weights Weights used for "optimal combination" method. When weights = "sd", it takes account of the standard deviation of forecasts. parallel If TRUE, allow parallel processing num.cores If parallel = TRUE, specify how many cores are going to be used Visualising and forecasting big time series data hts package for R 45

118 Outline 1 Examples of biggish time series 2 Time series visualisation 3 BLUF: Best Linear Unbiased Forecasts 4 Application: Australian tourism 5 Fast computation tricks 6 hts package for R 7 References Visualising and forecasting big time series data References 46

119 References RJ Hyndman, RA Ahmed, G Athanasopoulos, and HL Shang (2011). Optimal combination forecasts for hierarchical time series. Computational statistics & data analysis 55(9), RJ Hyndman, AJ Lee, and E Wang (2014). Fast computation of reconciled forecasts for hierarchical and grouped time series. Working paper 17/14. Department of Econometrics & Business Statistics, Monash University RJ Hyndman, AJ Lee, and E Wang (2014). hts: Hierarchical and grouped time series. cran.r-project.org/package=hts. RJ Hyndman and G Athanasopoulos (2014). Forecasting: principles and practice. OTexts. OTexts.org/fpp/. Visualising and forecasting big time series data References 47

120 References RJ Hyndman, RA Ahmed, G Athanasopoulos, and HL Shang (2011). Optimal combination forecasts for hierarchical time series. Computational statistics & data analysis 55(9), RJ Hyndman, AJ Lee, and E Wang (2014). Fast computation of reconciled forecasts for hierarchical and grouped time series. Working paper 17/14. Department of Econometrics & Business Statistics, Monash University RJ Hyndman, AJ Lee, and E Wang (2014). hts: Hierarchical and grouped time series. cran.r-project.org/package=hts. RJ robjhyndman.com Hyndman and G Athanasopoulos (2014). Forecasting: principles and practice. OTexts. OTexts.org/fpp/. Papers and R code: Rob.Hyndman@monash.edu Visualising and forecasting big time series data References 47