Forecasting: Principles and Practice. Rob J Hyndman. 12. Advanced methods OTexts.com/fpp/9/2/ OTexts.com/fpp/9/3/

Transcription

1 Rob J Hyndman Forecasting: Principles and Practice 12. Advanced methods OTexts.com/fpp/9/2/ OTexts.com/fpp/9/3/ Forecasting: Principles and Practice 1

2 Outline 1 Vector autoregressions 2 Neural network models 3 Time series with complex seasonality Forecasting: Principles and Practice Vector autoregressions 2

3 Vector autoregressions Dynamic regression assumes a unidirectional relationship: forecast variable influenced by predictor variables, but not vice versa. Vector AR allow for feedback relationships. All variables treated symmetrically. i.e., all variables are now treated as endogenous. Personal consumption may be affected by disposable income, and vice-versa. e.g., Govt stimulus package in Dec 2008 increased Christmas spending which increased incomes. Forecasting: Principles and Practice Vector autoregressions 3

4 Vector autoregressions Dynamic regression assumes a unidirectional relationship: forecast variable influenced by predictor variables, but not vice versa. Vector AR allow for feedback relationships. All variables treated symmetrically. i.e., all variables are now treated as endogenous. Personal consumption may be affected by disposable income, and vice-versa. e.g., Govt stimulus package in Dec 2008 increased Christmas spending which increased incomes. Forecasting: Principles and Practice Vector autoregressions 3

5 Vector autoregressions VAR(1) y 1,t = c 1 + φ 11,1 y 1,t 1 + φ 12,1 y 2,t 1 + e 1,t y 2,t = c 2 + φ 21,1 y 1,t 1 + φ 22,1 y 2,t 1 + e 2,t Forecasts: ŷ 1,T+1 T = ĉ 1 + ˆφ 11,1 y 1,T + ˆφ 12,1 y 2,T ŷ 2,T+1 T = ĉ 2 + ˆφ 21,1 y 1,T + ˆφ 22,1 y 2,T. ŷ 1,T+2 T = ĉ 1 + ˆφ 11,1 ŷ 1,T+1 + ˆφ 12,1 ŷ 2,T+1 ŷ 2,T+2 T = ĉ 2 + ˆφ 21,1 ŷ 1,T+1 + ˆφ 22,1 ŷ 2,T+1. Forecasting: Principles and Practice Vector autoregressions 4

6 Vector autoregressions VAR(1) y 1,t = c 1 + φ 11,1 y 1,t 1 + φ 12,1 y 2,t 1 + e 1,t y 2,t = c 2 + φ 21,1 y 1,t 1 + φ 22,1 y 2,t 1 + e 2,t Forecasts: ŷ 1,T+1 T = ĉ 1 + ˆφ 11,1 y 1,T + ˆφ 12,1 y 2,T ŷ 2,T+1 T = ĉ 2 + ˆφ 21,1 y 1,T + ˆφ 22,1 y 2,T. ŷ 1,T+2 T = ĉ 1 + ˆφ 11,1 ŷ 1,T+1 + ˆφ 12,1 ŷ 2,T+1 ŷ 2,T+2 T = ĉ 2 + ˆφ 21,1 ŷ 1,T+1 + ˆφ 22,1 ŷ 2,T+1. Forecasting: Principles and Practice Vector autoregressions 4

7 VARs are useful when forecasting a collection of related variables where no explicit interpretation is required; testing whether one variable is useful in forecasting another (the basis of Granger causality tests); impulse response analysis, where the response of one variable to a sudden but temporary change in another variable is analysed; forecast error variance decomposition, where the proportion of the forecast variance of one variable is attributed to the effect of other variables. Forecasting: Principles and Practice Vector autoregressions 5

8 VARs are useful when forecasting a collection of related variables where no explicit interpretation is required; testing whether one variable is useful in forecasting another (the basis of Granger causality tests); impulse response analysis, where the response of one variable to a sudden but temporary change in another variable is analysed; forecast error variance decomposition, where the proportion of the forecast variance of one variable is attributed to the effect of other variables. Forecasting: Principles and Practice Vector autoregressions 5

9 VARs are useful when forecasting a collection of related variables where no explicit interpretation is required; testing whether one variable is useful in forecasting another (the basis of Granger causality tests); impulse response analysis, where the response of one variable to a sudden but temporary change in another variable is analysed; forecast error variance decomposition, where the proportion of the forecast variance of one variable is attributed to the effect of other variables. Forecasting: Principles and Practice Vector autoregressions 5

10 VARs are useful when forecasting a collection of related variables where no explicit interpretation is required; testing whether one variable is useful in forecasting another (the basis of Granger causality tests); impulse response analysis, where the response of one variable to a sudden but temporary change in another variable is analysed; forecast error variance decomposition, where the proportion of the forecast variance of one variable is attributed to the effect of other variables. Forecasting: Principles and Practice Vector autoregressions 5

11 VAR example > ar(usconsumption,order=3) $ar,, 1 consumption income consumption income ,, 2 consumption income consumption income ,, 3 consumption income consumption income $var.pred consumption income consumption income Forecasting: Principles and Practice Vector autoregressions 6

12 VAR example > library(vars) > VARselect(usconsumption, lag.max=8, type="const")$selection AIC(n) HQ(n) SC(n) FPE(n) > var <- VAR(usconsumption, p=3, type="const") > serial.test(var, lags.pt=10, type="pt.asymptotic") Portmanteau Test (asymptotic) data: Residuals of VAR object var Chi-squared = , df = 28, p-value = Forecasting: Principles and Practice Vector autoregressions 7

13 VAR example > summary(var) VAR Estimation Results: ========================= Endogenous variables: consumption, income Deterministic variables: const Sample size: 161 Estimation results for equation consumption: ============================================ Estimate Std. Error t value Pr(> t ) consumption.l * income.l consumption.l * income.l consumption.l ** income.l const *** Forecasting: Principles and Practice Vector autoregressions 8

14 VAR example Estimation results for equation income: ======================================= Estimate Std. Error t value Pr(> t ) consumption.l e-05 *** income.l ** consumption.l income.l consumption.l *** income.l const ** --- Signif. codes: 0 *** ** 0.01 * Correlation matrix of residuals: consumption income consumption income Forecasting: Principles and Practice Vector autoregressions 9

15 VAR example fcst <- forecast(var) plot(fcst, xlab="year") Forecasting: Principles and Practice Vector autoregressions 10

16 VAR example Forecasts from VAR(3) income consumption Year Forecasting: Principles and Practice Vector autoregressions 11

17 Outline 1 Vector autoregressions 2 Neural network models 3 Time series with complex seasonality Forecasting: Principles and Practice Neural network models 12

18 Neural network models Simplest version: linear regression Input Output layer layer Input #1 Input #2 Input #3 Output Input #4 Coefficients attached to predictors are called weights. Forecasts are obtained by a linear combination of inputs. Weights selected using a learning algorithm that minimises a cost function. Forecasting: Principles and Practice Neural network models 13

19 Neural network models Simplest version: linear regression Input Output layer layer Input #1 Input #2 Input #3 Output Input #4 Coefficients attached to predictors are called weights. Forecasts are obtained by a linear combination of inputs. Weights selected using a learning algorithm that minimises a cost function. Forecasting: Principles and Practice Neural network models 13

20 Neural network models Simplest version: linear regression Input Output layer layer Input #1 Input #2 Input #3 Output Input #4 Coefficients attached to predictors are called weights. Forecasts are obtained by a linear combination of inputs. Weights selected using a learning algorithm that minimises a cost function. Forecasting: Principles and Practice Neural network models 13

21 Neural network models Simplest version: linear regression Input Output layer layer Input #1 Input #2 Input #3 Output Input #4 Coefficients attached to predictors are called weights. Forecasts are obtained by a linear combination of inputs. Weights selected using a learning algorithm that minimises a cost function. Forecasting: Principles and Practice Neural network models 13

22 Neural network models Nonlinear model with one hidden layer Input Hidden Output layer layer layer Input #1 Input #2 Input #3 Output Input #4 A multilayer feed-forward network where each layer of nodes receives inputs from the previous layers. Inputs to each node combined using linear combination. Result modified by nonlinear function before being output. Forecasting: Principles and Practice Neural network models 14

23 Neural network models Nonlinear model with one hidden layer Input Hidden Output layer layer layer Input #1 Input #2 Input #3 Output Input #4 A multilayer feed-forward network where each layer of nodes receives inputs from the previous layers. Inputs to each node combined using linear combination. Result modified by nonlinear function before being output. Forecasting: Principles and Practice Neural network models 14

24 Neural network models Nonlinear model with one hidden layer Input Hidden Output layer layer layer Input #1 Input #2 Input #3 Output Input #4 A multilayer feed-forward network where each layer of nodes receives inputs from the previous layers. Inputs to each node combined using linear combination. Result modified by nonlinear function before being output. Forecasting: Principles and Practice Neural network models 14

25 Neural network models Nonlinear model with one hidden layer Input Hidden Output layer layer layer Input #1 Input #2 Input #3 Output Input #4 A multilayer feed-forward network where each layer of nodes receives inputs from the previous layers. Inputs to each node combined using linear combination. Result modified by nonlinear function before being output. Forecasting: Principles and Practice Neural network models 14

26 Neural network models Inputs to hidden neuron j linearly combined: z j = b j + 4 w i,j x i. i=1 Modified using nonlinear function such as a sigmoid: s(z) = e z, This tends to reduce the effect of extreme input values, thus making the network somewhat robust to outliers. Forecasting: Principles and Practice Neural network models 15

27 Neural network models Weights take random values to begin with, which are then updated using the observed data. There is an element of randomness in the predictions. So the network is usually trained several times using different random starting points, and the results are averaged. Number of hidden layers, and the number of nodes in each hidden layer, must be specified in advance. Forecasting: Principles and Practice Neural network models 16

28 Neural network models Weights take random values to begin with, which are then updated using the observed data. There is an element of randomness in the predictions. So the network is usually trained several times using different random starting points, and the results are averaged. Number of hidden layers, and the number of nodes in each hidden layer, must be specified in advance. Forecasting: Principles and Practice Neural network models 16

29 Neural network models Weights take random values to begin with, which are then updated using the observed data. There is an element of randomness in the predictions. So the network is usually trained several times using different random starting points, and the results are averaged. Number of hidden layers, and the number of nodes in each hidden layer, must be specified in advance. Forecasting: Principles and Practice Neural network models 16

30 NNAR models Lagged values of the time series can be used as inputs to a neural network. NNAR(p, k): p lagged inputs and k nodes in the single hidden layer. NNAR(p, 0) model is equivalent to an ARIMA(p, 0, 0) model but without stationarity restrictions. Seasonal NNAR(p, P, k): inputs (y t 1, y t 2,..., y t p, y t m, y t 2m, y t Pm ) and k neurons in the hidden layer. NNAR(p, P, 0) m model is equivalent to an ARIMA(p, 0, 0)(P,0,0) m model but without stationarity restrictions. Forecasting: Principles and Practice Neural network models 17

31 NNAR models Lagged values of the time series can be used as inputs to a neural network. NNAR(p, k): p lagged inputs and k nodes in the single hidden layer. NNAR(p, 0) model is equivalent to an ARIMA(p, 0, 0) model but without stationarity restrictions. Seasonal NNAR(p, P, k): inputs (y t 1, y t 2,..., y t p, y t m, y t 2m, y t Pm ) and k neurons in the hidden layer. NNAR(p, P, 0) m model is equivalent to an ARIMA(p, 0, 0)(P,0,0) m model but without stationarity restrictions. Forecasting: Principles and Practice Neural network models 17

32 NNAR models Lagged values of the time series can be used as inputs to a neural network. NNAR(p, k): p lagged inputs and k nodes in the single hidden layer. NNAR(p, 0) model is equivalent to an ARIMA(p, 0, 0) model but without stationarity restrictions. Seasonal NNAR(p, P, k): inputs (y t 1, y t 2,..., y t p, y t m, y t 2m, y t Pm ) and k neurons in the hidden layer. NNAR(p, P, 0) m model is equivalent to an ARIMA(p, 0, 0)(P,0,0) m model but without stationarity restrictions. Forecasting: Principles and Practice Neural network models 17

33 NNAR models Lagged values of the time series can be used as inputs to a neural network. NNAR(p, k): p lagged inputs and k nodes in the single hidden layer. NNAR(p, 0) model is equivalent to an ARIMA(p, 0, 0) model but without stationarity restrictions. Seasonal NNAR(p, P, k): inputs (y t 1, y t 2,..., y t p, y t m, y t 2m, y t Pm ) and k neurons in the hidden layer. NNAR(p, P, 0) m model is equivalent to an ARIMA(p, 0, 0)(P,0,0) m model but without stationarity restrictions. Forecasting: Principles and Practice Neural network models 17

34 NNAR models Lagged values of the time series can be used as inputs to a neural network. NNAR(p, k): p lagged inputs and k nodes in the single hidden layer. NNAR(p, 0) model is equivalent to an ARIMA(p, 0, 0) model but without stationarity restrictions. Seasonal NNAR(p, P, k): inputs (y t 1, y t 2,..., y t p, y t m, y t 2m, y t Pm ) and k neurons in the hidden layer. NNAR(p, P, 0) m model is equivalent to an ARIMA(p, 0, 0)(P,0,0) m model but without stationarity restrictions. Forecasting: Principles and Practice Neural network models 17

35 NNAR models in R The nnetar() function fits an NNAR(p, P, k) m model. If p and P are not specified, they are automatically selected. For non-seasonal time series, default p = optimal number of lags (according to the AIC) for a linear AR(p) model. For seasonal time series, defaults are P = 1 and p is chosen from the optimal linear model fitted to the seasonally adjusted data. Default k = (p + P + 1)/2 (rounded to the nearest integer). Forecasting: Principles and Practice Neural network models 18

36 NNAR models in R The nnetar() function fits an NNAR(p, P, k) m model. If p and P are not specified, they are automatically selected. For non-seasonal time series, default p = optimal number of lags (according to the AIC) for a linear AR(p) model. For seasonal time series, defaults are P = 1 and p is chosen from the optimal linear model fitted to the seasonally adjusted data. Default k = (p + P + 1)/2 (rounded to the nearest integer). Forecasting: Principles and Practice Neural network models 18

37 NNAR models in R The nnetar() function fits an NNAR(p, P, k) m model. If p and P are not specified, they are automatically selected. For non-seasonal time series, default p = optimal number of lags (according to the AIC) for a linear AR(p) model. For seasonal time series, defaults are P = 1 and p is chosen from the optimal linear model fitted to the seasonally adjusted data. Default k = (p + P + 1)/2 (rounded to the nearest integer). Forecasting: Principles and Practice Neural network models 18

38 NNAR models in R The nnetar() function fits an NNAR(p, P, k) m model. If p and P are not specified, they are automatically selected. For non-seasonal time series, default p = optimal number of lags (according to the AIC) for a linear AR(p) model. For seasonal time series, defaults are P = 1 and p is chosen from the optimal linear model fitted to the seasonally adjusted data. Default k = (p + P + 1)/2 (rounded to the nearest integer). Forecasting: Principles and Practice Neural network models 18

39 NNAR models in R The nnetar() function fits an NNAR(p, P, k) m model. If p and P are not specified, they are automatically selected. For non-seasonal time series, default p = optimal number of lags (according to the AIC) for a linear AR(p) model. For seasonal time series, defaults are P = 1 and p is chosen from the optimal linear model fitted to the seasonally adjusted data. Default k = (p + P + 1)/2 (rounded to the nearest integer). Forecasting: Principles and Practice Neural network models 18

40 Sunspots Surface of the sun contains magnetic regions that appear as dark spots. These affect the propagation of radio waves and so telecommunication companies like to predict sunspot activity in order to plan for any future difficulties. Sunspots follow a cycle of length between 9 and 14 years. Forecasting: Principles and Practice Neural network models 19

41 Sunspots Surface of the sun contains magnetic regions that appear as dark spots. These affect the propagation of radio waves and so telecommunication companies like to predict sunspot activity in order to plan for any future difficulties. Sunspots follow a cycle of length between 9 and 14 years. Forecasting: Principles and Practice Neural network models 19

42 Sunspots Surface of the sun contains magnetic regions that appear as dark spots. These affect the propagation of radio waves and so telecommunication companies like to predict sunspot activity in order to plan for any future difficulties. Sunspots follow a cycle of length between 9 and 14 years. Forecasting: Principles and Practice Neural network models 19

43 NNAR(9,5) model for sunspots Forecasts from NNAR(9) Forecasting: Principles and Practice Neural network models 20

44 NNAR model for sunspots fit <- nnetar(sunspotarea) plot(forecast(fit,h=20)) To restrict to positive values: fit <- nnetar(sunspotarea,lambda=0) plot(forecast(fit,h=20)) Forecasting: Principles and Practice Neural network models 21

45 NNAR model for sunspots fit <- nnetar(sunspotarea) plot(forecast(fit,h=20)) To restrict to positive values: fit <- nnetar(sunspotarea,lambda=0) plot(forecast(fit,h=20)) Forecasting: Principles and Practice Neural network models 21

46 Outline 1 Vector autoregressions 2 Neural network models 3 Time series with complex seasonality Forecasting: Principles and Practice Time series with complex seasonality 22

47 Examples US finished motor gasoline products Thousands of barrels per day Weeks Forecasting: Principles and Practice Time series with complex seasonality 23

48 Examples Number of calls to large American bank (7am 9pm) Number of call arrivals March 17 March 31 March 14 April 28 April 12 May 5 minute intervals Forecasting: Principles and Practice Time series with complex seasonality 23

49 Examples Turkish electricity demand Electricity demand (GW) Days Forecasting: Principles and Practice Time series with complex seasonality 23

50 TBATS model TBATS Trigonometric terms for seasonality Box-Cox transformations for heterogeneity ARMA errors for short-term dynamics Trend (possibly damped) Seasonal (including multiple and non-integer periods) Forecasting: Principles and Practice Time series with complex seasonality 24

51 TBATS model y t = observation at time t { y (ω) (y ω t 1)/ω if ω 0; t = log y t if ω = 0. y (ω) t = l t 1 + φb t 1 + M i=1 l t = l t 1 + φb t 1 + αd t s (i) t m i + d t b t = (1 φ)b + φb t 1 + βd t p q d t = φ i d t i + θ j ε t j + ε t s (i) t = i=1 k i j=1 s (i) j,t j=1 s (i) j,t = s(i) j,t 1 cos λ(i) j + s (i) j,t 1 sin λ(i) j + γ (i) 1 d t s (i) = j,t s(i) j,t 1 sin λ(i) j + s (i) j,t 1 cos λ(i) j + γ (i) 2 d t Forecasting: Principles and Practice Time series with complex seasonality 25

52 TBATS model y t = observation at time t { y (ω) (y ω t 1)/ω if ω 0; t = log y t if ω = 0. y (ω) t = l t 1 + φb t 1 + M i=1 s (i) t m i + d t Box-Cox transformation l t = l t 1 + φb t 1 + αd t b t = (1 φ)b + φb t 1 + βd t p q d t = φ i d t i + θ j ε t j + ε t s (i) t = i=1 k i j=1 s (i) j,t j=1 s (i) j,t = s(i) j,t 1 cos λ(i) j + s (i) j,t 1 sin λ(i) j + γ (i) 1 d t s (i) = j,t s(i) j,t 1 sin λ(i) j + s (i) j,t 1 cos λ(i) j + γ (i) 2 d t Forecasting: Principles and Practice Time series with complex seasonality 25

53 TBATS model y t = observation at time t { y (ω) (y ω t 1)/ω if ω 0; t = log y t if ω = 0. y (ω) t = l t 1 + φb t 1 + M i=1 s (i) t m i + d t Box-Cox transformation M seasonal periods l t = l t 1 + φb t 1 + αd t b t = (1 φ)b + φb t 1 + βd t p q d t = φ i d t i + θ j ε t j + ε t s (i) t = i=1 k i j=1 s (i) j,t j=1 s (i) j,t = s(i) j,t 1 cos λ(i) j + s (i) j,t 1 sin λ(i) j + γ (i) 1 d t s (i) = j,t s(i) j,t 1 sin λ(i) j + s (i) j,t 1 cos λ(i) j + γ (i) 2 d t Forecasting: Principles and Practice Time series with complex seasonality 25

54 TBATS model y t = observation at time t { y (ω) (y ω t 1)/ω if ω 0; t = log y t if ω = 0. y (ω) t = l t 1 + φb t 1 + M i=1 s (i) t m i + d t Box-Cox transformation M seasonal periods l t = l t 1 + φb t 1 + αd t b t = (1 φ)b + φb t 1 + βd t p q d t = φ i d t i + θ j ε t j + ε t s (i) t = i=1 k i j=1 s (i) j,t j=1 s (i) j,t = global and local trend s(i) j,t 1 cos λ(i) j + s (i) j,t 1 sin λ(i) j + γ (i) 1 d t s (i) = j,t s(i) j,t 1 sin λ(i) j + s (i) j,t 1 cos λ(i) j + γ (i) 2 d t Forecasting: Principles and Practice Time series with complex seasonality 25

55 TBATS model y t = observation at time t { y (ω) (y ω t 1)/ω if ω 0; t = log y t if ω = 0. y (ω) t = l t 1 + φb t 1 + M i=1 s (i) t m i + d t Box-Cox transformation M seasonal periods l t = l t 1 + φb t 1 + αd t b t = (1 φ)b + φb t 1 + βd t p q d t = φ i d t i + θ j ε t j + ε t s (i) t = i=1 k i j=1 s (i) j,t j=1 s (i) j,t = global and local trend ARMA error s(i) j,t 1 cos λ(i) j + s (i) j,t 1 sin λ(i) j + γ (i) 1 d t s (i) = j,t s(i) j,t 1 sin λ(i) j + s (i) j,t 1 cos λ(i) j + γ (i) 2 d t Forecasting: Principles and Practice Time series with complex seasonality 25

56 TBATS model y t = observation at time t { y (ω) (y ω t 1)/ω if ω 0; t = log y t if ω = 0. y (ω) t = l t 1 + φb t 1 + M i=1 s (i) t m i + d t Box-Cox transformation M seasonal periods l t = l t 1 + φb t 1 + αd t b t = (1 φ)b + φb t 1 + βd t p q d t = φ i d t i + θ j ε t j + ε t s (i) t = i=1 k i j=1 s (i) j,t j=1 s (i) j,t = s(i) j,t 1 global and local trend ARMA error Fourier-like seasonal terms+ s (i) j,t 1 sin λ(i) j cos λ(i) j + γ (i) 1 d t s (i) = j,t s(i) j,t 1 sin λ(i) j + s (i) j,t 1 cos λ(i) j + γ (i) 2 d t Forecasting: Principles and Practice Time series with complex seasonality 25

57 TBATS model y t = observation at time t { y (ω) (y ω t 1)/ω if ω 0; t = logtbats y t if ω = 0. Trigonometric M y (ω) t = l t 1 + φb t 1 + s (i) t m i + d t i=1 l t = l t 1 + φb t 1 + αd t b t = (1 φ)b + φb t 1 + βd t p Trendq d t = φ i d t i + θ j ε t j + ε t s (i) t = i=1 k i j=1 Box-Cox ARMA Seasonal s (i) j,t j=1 s (i) j,t = s(i) j,t 1 Box-Cox transformation M seasonal periods global and local trend ARMA error Fourier-like seasonal terms+ s (i) j,t 1 sin λ(i) j cos λ(i) j + γ (i) 1 d t s (i) = j,t s(i) j,t 1 sin λ(i) j + s (i) j,t 1 cos λ(i) j + γ (i) 2 d t Forecasting: Principles and Practice Time series with complex seasonality 25

58 Examples Forecasts from TBATS(0.999, {2,2}, 1, {< ,8>}) Thousands of barrels per day fit <- tbats(gasoline) fcast <- forecast(fit) plot(fcast) Weeks Forecasting: Principles and Practice Time series with complex seasonality 26

59 Number of call arrivals Examples Forecasts from TBATS(1, {3,1}, 0.987, {<169,5>, <845,3>}) fit <- tbats(callcentre) fcast <- forecast(fit) plot(fcast) 3 March 17 March 31 March 14 April 28 April 12 May 26 May 9 June 5 minute intervals Forecasting: Principles and Practice Time series with complex seasonality 27

60 Examples Forecasts from TBATS(0, {5,3}, 0.997, {<7,3>, <354.37,12>, <365.25,4>}) Electricity demand (GW) fit <- tbats(turk) fcast <- forecast(fit) plot(fcast) Days Forecasting: Principles and Practice Time series with complex seasonality 28