Fast computation of reconciled forecasts in hierarchical and grouped time series

Transcription

1 Rob J Hyndman Fast computation of reconciled forecasts in hierarchical and grouped time series Total A B C AX AY AZ BX BY BZ CX CY CZ 1

2 Outline 1 Optimally reconciled forecasts 2 Fast computation 3 hts package for R 4 References Fast computation of reconciled forecasts Optimally reconciled forecasts 2

3 Hierarchical data Total A B C Y t = AX AY AZ Y t Y A,t Y B,t Y C,t Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t = BX BY BZ } {{ } S Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t }{{} B t CX CY CZ Fast computation of reconciled forecasts Optimally reconciled forecasts 3

4 Hierarchical data Total A B C Y t = AX AY AZ Y t Y A,t Y B,t Y C,t Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t = BX BY BZ } {{ } S Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t }{{} B t CX CY CZ Fast computation of reconciled forecasts Optimally reconciled forecasts 3

5 Hierarchical data Total A B C Y t = AX AY AZ Y t Y A,t Y B,t Y C,t Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t = BX BY BZ } {{ } S Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t }{{} B t CX CY CZ Y t = SB t Fast computation of reconciled forecasts Optimally reconciled forecasts 3

6 Grouped data Y t = AX AY AZ A BX BY BZ B CX CY CZ C X Y Z Total Y t Y A,t Y B,t Y C,t Y AX,t Y X,t Y Y Y,t AY,t Y Y Z,t AZ,t Y AX,t Y Y AY,t = BX,t Y BY,t Y AZ,t Y BZ,t Y BX,t Y BY,t Y CX,t Y Y BZ,t CY,t Y CX,t Y CZ,t }{{} Y CY,t B t Y CZ,t Fast computation of} reconciled forecasts{{ Optimally } reconciled forecasts 4

7 Grouped data Y t = AX AY AZ A BX BY BZ B CX CY CZ C X Y Z Total Y t Y A,t Y B,t Y C,t Y AX,t Y X,t Y Y Y,t AY,t Y Y Z,t AZ,t Y AX,t Y Y AY,t = BX,t Y BY,t Y AZ,t Y BZ,t Y BX,t Y BY,t Y CX,t Y Y BZ,t CY,t Y CX,t Y CZ,t }{{} Y CY,t B t Y CZ,t Fast computation of} reconciled forecasts{{ Optimally } reconciled forecasts 4

8 Grouped data Y t = AX AY AZ A BX BY BZ B CX CY CZ C X Y Z Total Y t Y A,t Y B,t Y C,t Y AX,t Y X,t Y Y Y,t AY,t Y Y Z,t AZ,t Y AX,t Y Y AY,t = BX,t Y BY,t Y AZ,t Y BZ,t Y BX,t Y BY,t Y CX,t Y Y BZ,t CY,t Y CX,t Y CZ,t }{{} Y CY,t B t Y CZ,t Fast computation of} reconciled forecasts{{ Optimally } reconciled forecasts 4 Y t = SB t

9 Forecasts Key idea: forecast reconciliation Ignore structural constraints and forecast every series of interest independently. Adjust forecasts to impose constraints. Let Ŷ n (h) be vector of initial h-step forecasts, made at time n, stacked in same order as Y t. Y t = SB t. So Ŷ n (h) = Sβ n (h) + ε h. β n (h) = E[B n+h Y 1,..., Y n ]. ε h has zero mean. Estimate β n (h) using weighted least squares. Fast computation of reconciled forecasts Optimally reconciled forecasts 5

10 Forecasts Key idea: forecast reconciliation Ignore structural constraints and forecast every series of interest independently. Adjust forecasts to impose constraints. Let Ŷ n (h) be vector of initial h-step forecasts, made at time n, stacked in same order as Y t. Y t = SB t. So Ŷ n (h) = Sβ n (h) + ε h. β n (h) = E[B n+h Y 1,..., Y n ]. ε h has zero mean. Estimate β n (h) using weighted least squares. Fast computation of reconciled forecasts Optimally reconciled forecasts 5

11 Forecasts Key idea: forecast reconciliation Ignore structural constraints and forecast every series of interest independently. Adjust forecasts to impose constraints. Let Ŷ n (h) be vector of initial h-step forecasts, made at time n, stacked in same order as Y t. Y t = SB t. So Ŷ n (h) = Sβ n (h) + ε h. β n (h) = E[B n+h Y 1,..., Y n ]. ε h has zero mean. Estimate β n (h) using weighted least squares. Fast computation of reconciled forecasts Optimally reconciled forecasts 5

12 Forecasts Key idea: forecast reconciliation Ignore structural constraints and forecast every series of interest independently. Adjust forecasts to impose constraints. Let Ŷ n (h) be vector of initial h-step forecasts, made at time n, stacked in same order as Y t. Y t = SB t. So Ŷ n (h) = Sβ n (h) + ε h. β n (h) = E[B n+h Y 1,..., Y n ]. ε h has zero mean. Estimate β n (h) using weighted least squares. Fast computation of reconciled forecasts Optimally reconciled forecasts 5

13 Forecasts Key idea: forecast reconciliation Ignore structural constraints and forecast every series of interest independently. Adjust forecasts to impose constraints. Let Ŷ n (h) be vector of initial h-step forecasts, made at time n, stacked in same order as Y t. Y t = SB t. So Ŷ n (h) = Sβ n (h) + ε h. β n (h) = E[B n+h Y 1,..., Y n ]. ε h has zero mean. Estimate β n (h) using weighted least squares. Fast computation of reconciled forecasts Optimally reconciled forecasts 5

14 Forecasts Key idea: forecast reconciliation Ignore structural constraints and forecast every series of interest independently. Adjust forecasts to impose constraints. Let Ŷ n (h) be vector of initial h-step forecasts, made at time n, stacked in same order as Y t. Y t = SB t. So Ŷ n (h) = Sβ n (h) + ε h. β n (h) = E[B n+h Y 1,..., Y n ]. ε h has zero mean. Estimate β n (h) using weighted least squares. Fast computation of reconciled forecasts Optimally reconciled forecasts 5

15 Forecasts Key idea: forecast reconciliation Ignore structural constraints and forecast every series of interest independently. Adjust forecasts to impose constraints. Let Ŷ n (h) be vector of initial h-step forecasts, made at time n, stacked in same order as Y t. Y t = SB t. So Ŷ n (h) = Sβ n (h) + ε h. β n (h) = E[B n+h Y 1,..., Y n ]. ε h has zero mean. Estimate β n (h) using weighted least squares. Fast computation of reconciled forecasts Optimally reconciled forecasts 5

16 Optimal reconciled forecasts Ỹ n (h) = S ˆβ n (h) = S(S ΛS) 1 S ΛŶn(h) Λ weights (usually inverse of one-step forecast variances, but any choice possible). Easy to estimate, and places weight where we have best forecasts. Ignores covariances. Still difficult to compute for large numbers of time series. Need to do calculation without explicitly forming S or (S ΛS) 1 or S Λ. Fast computation of reconciled forecasts Optimally reconciled forecasts 6

17 Optimal reconciled forecasts Ỹ n (h) = S ˆβ n (h) = S(S ΛS) 1 S ΛŶn(h) Initial forecasts Λ weights (usually inverse of one-step forecast variances, but any choice possible). Easy to estimate, and places weight where we have best forecasts. Ignores covariances. Still difficult to compute for large numbers of time series. Need to do calculation without explicitly forming S or (S ΛS) 1 or S Λ. Fast computation of reconciled forecasts Optimally reconciled forecasts 6

18 Optimal reconciled forecasts Ỹ n (h) = S ˆβ n (h) = S(S ΛS) 1 S ΛŶn(h) Revised forecasts Initial forecasts Λ weights (usually inverse of one-step forecast variances, but any choice possible). Easy to estimate, and places weight where we have best forecasts. Ignores covariances. Still difficult to compute for large numbers of time series. Need to do calculation without explicitly forming S or (S ΛS) 1 or S Λ. Fast computation of reconciled forecasts Optimally reconciled forecasts 6

19 Optimal reconciled forecasts Ỹ n (h) = S ˆβ n (h) = S(S ΛS) 1 S ΛŶn(h) Revised forecasts Initial forecasts Λ weights (usually inverse of one-step forecast variances, but any choice possible). Easy to estimate, and places weight where we have best forecasts. Ignores covariances. Still difficult to compute for large numbers of time series. Need to do calculation without explicitly forming S or (S ΛS) 1 or S Λ. Fast computation of reconciled forecasts Optimally reconciled forecasts 6

20 Optimal reconciled forecasts Ỹ n (h) = S ˆβ n (h) = S(S ΛS) 1 S ΛŶn(h) Revised forecasts Initial forecasts Λ weights (usually inverse of one-step forecast variances, but any choice possible). Easy to estimate, and places weight where we have best forecasts. Ignores covariances. Still difficult to compute for large numbers of time series. Need to do calculation without explicitly forming S or (S ΛS) 1 or S Λ. Fast computation of reconciled forecasts Optimally reconciled forecasts 6

21 Optimal reconciled forecasts Ỹ n (h) = S ˆβ n (h) = S(S ΛS) 1 S ΛŶn(h) Revised forecasts Initial forecasts Λ weights (usually inverse of one-step forecast variances, but any choice possible). Easy to estimate, and places weight where we have best forecasts. Ignores covariances. Still difficult to compute for large numbers of time series. Need to do calculation without explicitly forming S or (S ΛS) 1 or S Λ. Fast computation of reconciled forecasts Optimally reconciled forecasts 6

22 Optimal reconciled forecasts Ỹ n (h) = S ˆβ n (h) = S(S ΛS) 1 S ΛŶn(h) Revised forecasts Initial forecasts Λ weights (usually inverse of one-step forecast variances, but any choice possible). Easy to estimate, and places weight where we have best forecasts. Ignores covariances. Still difficult to compute for large numbers of time series. Need to do calculation without explicitly forming S or (S ΛS) 1 or S Λ. Fast computation of reconciled forecasts Optimally reconciled forecasts 6

23 Outline 1 Optimally reconciled forecasts 2 Fast computation 3 hts package for R 4 References Fast computation of reconciled forecasts Fast computation 7

24 Fast computation: hierarchical data Total A B C Y t = AX AY AZ Y t Y A,t Y B,t Y C,t Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t = BX BY BZ } {{ } S Y AX,t Y AY,t Y AZ,t Y BX,t Y BY,t Y BZ,t Y CX,t Y CY,t Y CZ,t }{{} B t CX CY CZ Y t = SB t Fast computation of reconciled forecasts Fast computation 8

25 Fast computation: hierarchies Think of the hierarchy as a tree of trees: Total T 1 T 2... T K Then the summing matrix contains k smaller summing matrices: S = 1 n 1 1 n 2 1 n K S S S K where 1 n is an n-vector of ones and tree T i has n i terminal nodes. Fast computation of reconciled forecasts Fast computation 9.

26 Fast computation: hierarchies Think of the hierarchy as a tree of trees: Total T 1 T 2... T K Then the summing matrix contains k smaller summing matrices: S = 1 n 1 1 n 2 1 n K S S S K where 1 n is an n-vector of ones and tree T i has n i terminal nodes. Fast computation of reconciled forecasts Fast computation 9.

27 Fast computation: hierarchies S ΛS = S 1Λ 1 S S 2Λ 2 S S KΛ K S K. +λ 0 J n λ 0 is the top left element of Λ; Λ k is a block of Λ, corresponding to tree T k ; J n is a matrix of ones; n = k n k. Now apply the Sherman-Morrison formula... Fast computation of reconciled forecasts Fast computation 10

28 Fast computation: hierarchies S ΛS = S 1Λ 1 S S 2Λ 2 S S KΛ K S K. +λ 0 J n λ 0 is the top left element of Λ; Λ k is a block of Λ, corresponding to tree T k ; J n is a matrix of ones; n = k n k. Now apply the Sherman-Morrison formula... Fast computation of reconciled forecasts Fast computation 10

29 Fast computation: hierarchies (S 1 Λ 1S 1 ) (S ΛS) 1 0 (S 2 = Λ 2S 2 ) cs (S K Λ KS K ) 1 S 0 can be partitioned into K 2 blocks, with the (k, l) block (of dimension n k n l ) being J nk,n l (S k Λ ks k ) 1 J nk,n l (S l Λ ls l ) 1 is a n k n l matrix of ones. c 1 = λ n k (S k Λ ks k ) 1 1 nk. k Each S k Λ ks k can be inverted similarly. S ΛY can also be computed recursively. Fast computation of reconciled forecasts Fast computation 11

30 Fast computation: hierarchies (S 1 Λ 1S 1 ) (S ΛS) 1 0 (S 2 = Λ 2S 2 ) cs (S K Λ KS K ) 1 S 0 can be partitioned into K 2 blocks, with the (k, l) block (of dimension n k n l ) being The recursive (S k Λ calculations ks k ) 1 J nk,n l (S l Λ can ls l ) 1 be done in such a way that we never J nk,n l store is a n k any n l matrix of ones. of the large matrices c 1 = involved. λ n k (S k Λ ks k ) 1 1 nk. k Each S k Λ ks k can be inverted similarly. S ΛY can also be computed recursively. Fast computation of reconciled forecasts Fast computation 11

31 Fast computation: grouped data Y t = AX AY AZ A BX BY BZ B CX CY CZ C X Y Z Total Y t Y A,t Y B,t Y C,t Y AX,t Y X,t Y Y Y,t AY,t Y Y Z,t AZ,t Y AX,t Y Y AY,t = BX,t Y BY,t Y AZ,t Y BZ,t Y BX,t Y BY,t Y CX,t Y Y BZ,t CY,t Y CX,t Y CZ,t Y t = SB t }{{} Y CY,t B t Y CZ,t Fast computation of reconciled forecasts Fast computation 12 }{{}

32 Fast computation: grouped data S = 1 m 1 n 1 m I n I m 1 n I m I n m = number of rows n = number of columns S ΛS = λ 00 J mn + (Λ R J n ) + (J m Λ C ) + Λ U Λ R, Λ C and Λ U are diagonal matrices corresponding to rows, columns and unaggregated series; λ 00 corresponds to aggregate. Fast computation of reconciled forecasts Fast computation 13

33 Fast computation: grouped data S = 1 m 1 n 1 m I n I m 1 n I m I n m = number of rows n = number of columns S ΛS = λ 00 J mn + (Λ R J n ) + (J m Λ C ) + Λ U Λ R, Λ C and Λ U are diagonal matrices corresponding to rows, columns and unaggregated series; λ 00 corresponds to aggregate. Fast computation of reconciled forecasts Fast computation 13

34 Fast computation: grouped data (SΛS) 1 = A A1 mn 1 mn A 1/λ mn A1 mn A = Λ 1 U Λ 1 U (J m D)Λ 1 U EM 1 E. D is diagonal with elements d j = λ 0j /(1 + λ 0j E has m m blocks where e ij has kth element { (e ij ) k = λ 1/2 i0 λ 1 ik λ1/2 i0 λ 2 λ 1/2 j0 λ 1 ik λ 1 ik d k, i = j, jk d k, i j. i λ 1 ij ). M is m m with (i, j) element { 1 + λ i0 k (M) ij = λ 1 λ ik i0 k λ 2 ik d k, i = j, k λ 1 ik λ 1 jk d k, i j. λ 1/2 i0 λ1/2 j0 Fast computation of reconciled forecasts Fast computation 14

35 Fast computation: grouped data (SΛS) 1 = A A1 mn 1 mn A 1/λ mn A1 mn A = Λ 1 U Λ 1 U (J m D)Λ 1 U EM 1 E. D is diagonal with elements d j = λ 0j /(1 + λ 0j E has m m blocks where e ij has kth element { Again, theλ calculations 1/2 i0 (e ij ) k = λ 1 ik λ1/2 can i0 λ 2 ik d be k, done i = j, in such a way λ 1/2 that j0 λ 1 ik λ 1 we jk d never k, store i j. any of the large matrices involved. i λ 1 ij ). M is m m with (i, j) element { 1 + λ i0 k (M) ij = λ 1 λ ik i0 k λ 2 ik d k, i = j, k λ 1 ik λ 1 jk d k, i j. λ 1/2 i0 λ1/2 j0 Fast computation of reconciled forecasts Fast computation 14

36 Fast computation When the time series are not strictly hierarchical and have more than two grouping variables: Use sparse matrix storage and arithmetic. Use iterative approximation for inverting large sparse matrices. Paige & Saunders (1982) ACM Trans. Math. Software Fast computation of reconciled forecasts Fast computation 15

37 Fast computation When the time series are not strictly hierarchical and have more than two grouping variables: Use sparse matrix storage and arithmetic. Use iterative approximation for inverting large sparse matrices. Paige & Saunders (1982) ACM Trans. Math. Software Fast computation of reconciled forecasts Fast computation 15

38 Fast computation When the time series are not strictly hierarchical and have more than two grouping variables: Use sparse matrix storage and arithmetic. Use iterative approximation for inverting large sparse matrices. Paige & Saunders (1982) ACM Trans. Math. Software Fast computation of reconciled forecasts Fast computation 15

39 Outline 1 Optimally reconciled forecasts 2 Fast computation 3 hts package for R 4 References Fast computation of reconciled forecasts hts package for R 16

40 hts package for R hts: Hierarchical and grouped time series Methods for analysing and forecasting hierarchical and grouped time series Version: 4.3 Depends: forecast ( 5.0) Imports: SparseM, parallel, utils Published: Author: Rob J Hyndman, Earo Wang and Alan Lee Maintainer: Rob J Hyndman <Rob.Hyndman at monash.edu> BugReports: License: GPL ( 2) Fast computation of reconciled forecasts hts package for R 17

41 Example using R library(hts) # bts is a matrix containing the bottom level time series # nodes describes the hierarchical structure y <- hts(bts, nodes=list(2, c(3,2))) Fast computation of reconciled forecasts hts package for R 18

42 Example using R library(hts) # bts is a matrix containing the bottom level time series # nodes describes the hierarchical structure y <- hts(bts, nodes=list(2, c(3,2))) Total A B AX AY AZ BX BY Fast computation of reconciled forecasts hts package for R 18

43 Example using R library(hts) # bts is a matrix containing the bottom level time series # nodes describes the hierarchical structure y <- hts(bts, nodes=list(2, c(3,2))) # Forecast 10-step-ahead using WLS combination method # ETS used for each series by default fc <- forecast(y, h=10) Fast computation of reconciled forecasts hts package for R 19

44 forecast.gts function Usage forecast(object, h, method = c("comb", "bu", "mo", "tdgsf", "tdgsa", "tdfp"), fmethod = c("ets", "rw", "arima"), weights = c("sd", "none", "nseries"), positive = FALSE, parallel = FALSE, num.cores = 2,...) Arguments object Hierarchical time series object of class gts. h Forecast horizon method Method for distributing forecasts within the hierarchy. fmethod Forecasting method to use positive If TRUE, forecasts are forced to be strictly positive weights Weights used for "optimal combination" method. When weights = "sd", it takes account of the standard deviation of forecasts. parallel If TRUE, allow parallel processing num.cores If parallel = TRUE, specify how many cores are going to be used Fast computation of reconciled forecasts hts package for R 20

45 Outline 1 Optimally reconciled forecasts 2 Fast computation 3 hts package for R 4 References Fast computation of reconciled forecasts References 21

46 References RJ Hyndman, RA Ahmed, G Athanasopoulos, and HL Shang (Sept. 2011). Optimal combination forecasts for hierarchical time series. Computational statistics & data analysis 55(9), RJ Hyndman, AJ Lee, and E Wang (2014). Fast computation of reconciled forecasts for hierarchical and grouped time series. Working paper 17/14. Department of Econometrics & Business Statistics, Monash University RJ Hyndman, AJ Lee, and E Wang (2014). hts: Hierarchical and grouped time series. cran.r-project.org/package=hts. RJ Hyndman and G Athanasopoulos (2014). Forecasting: principles and practice. OTexts. FastOTexts.org/fpp/. computation of reconciled forecasts References 22

47 References RJ Hyndman, RA Ahmed, G Athanasopoulos, and HL Shang (Sept. 2011). Optimal combination forecasts for hierarchical time series. Computational statistics & data analysis 55(9), RJ Hyndman, AJ Lee, and E Wang (2014). Fast computation of reconciled forecasts for hierarchical and grouped time series. Working paper 17/14. Department of Econometrics & Business Statistics, Monash University RJ Hyndman, AJ Lee, and E Wang (2014). hts: Hierarchical and grouped time series. robjhyndman.com cran.r-project.org/package=hts. RJ Hyndman and G Athanasopoulos (2014). Forecasting: principles and practice. OTexts. FastOTexts.org/fpp/. computation of reconciled forecasts References 22 Papers and R code: Rob.Hyndman@monash.edu