Near-equivalence in Forecasting Accuracy of Linear Dimension Reduction Method

Transcription

1 Near-equivalence in Forecasting Accuracy of Linear Dimension Reduction Methods in Large Macro-Panels Efstathia 1 (joint with Alessandro Barbarino 2 ) 1 Applied Statistics, Institute of Statistics and Mathematical Methods in Economics, Vienna University of Technology 2 Federal Reserve Board October 6, 2017

2 Outline 1 Overview of Empirical Results Out-of-Sample Forecasting Exercise Result 1: Empirical Forecast Accuracy Near-Equivalence Result 1: Empirical Forecast Accuracy Near-Equivalence Result 2: Dispersion of Information in the Panel Result 3: Targeting and Parsimony 2 Common Forecasting Framework Forecasting with Many Predictors Common Forecasting Framework Rationalizing Empirical Results 3 Beyond Linear Signals: Sufficient Dimension Reduction SDR Forecasting Framework 4 Conclusions

3 Overview of Empirical Results

4 The Empirical Forecasting Exercise McCracken and Ng (2015) created the FRED-MD macro-panel: 123 monthly macro variables. From January 1960 to 2015, balanced panel of 110 predictors with approx. 660 data. Focus on: CPI inflation, non-farm payrolls, unemployment rate, labor force participation, average hourly earnings in goods-producing industries, industrial production, real personal consumption expenditures and real personal income. Use popular statistical learning methods OLS, PCR, RIDGE, PLS and SIR (Sliced Inverse Regression). Out-of-sample forecasts using recursive window at horizons h = 1, 3, 6, 12. Compare results in classic horse-race against AR(4).

5 DFM Forecasting Dynamic Factor Models (DFMs) are the mainstream methodology in Econometrics for both measuring comovement and forecasting time series. The modeling comprises of two unconnected steps: The factors that replace x t are ordered according to how much variance in X they explain; i.e., in relevance to X and not to Y.

6 Best Estimators Before the Great Recession Table: Recursive Out-of-Sample Window from 1992 to 2007, MSFE Relative to AR4 h=1 h=3 h=6 h=12 Target Estimator MSFE Estimator MSFE Estimator MSFE Estimator MSFE INDPRO RIDGE b RIDGE b R70SIRb c,d R8SIRb c,d PAYEMS R30SIRb c,d R30SIRb c,d R30SIRb c,d R30SIRb c,d UNRATE PC PLS RIDGE b RIDGE b CLF16O RIDGE b PC PC PC CPIAUC PC.BS h SIRb d PC.ON f R8SIRb c,d CES060 PLS PLS-1 1 R8SIRb c,d AR4 1 DPCER AR4 1 AR4 1 AR4 1 AR4 1 RPI R8SIRb c,d R8SIRb c,d R8SIRb c,d RFSIRb c,d

7 Best Estimators In the Recovery Table: Recursive Out-of-Sample Window from 2010 to 2016, MSFE Relative to AR4 h=1 h=3 h=6 h=12 Target Estimator MSFE Estimator MSFE Estimator MSFE Estimator MSFE INDPRO RIDGE b RIDGE b RIDGE b R30SIRb c,d PAYEMS RIDGE b RIDGE b RIDGE b RIDGE b UNRATE PCF.BS i PLS SIRa d PC.BS h CLF16O PC PCF.BS i RIDGE b PLS CPIAUC PC.BS h SIRb d R70SIRb c,d PC CES060 SIRa d SIRb d R8SIRb c,d R8SIRa c,d DPCER PC PC PC PC RPI R8SIRa c,d RFSIRa c,d PLS PLS

8 Best Estimators in Full Sample Table: Recursive Out-of-Sample Window from 1992 to 2016, MSFE Relative to AR4 h=1 h=3 h=6 h=12 Target Estimator MSFE Estimator MSFE Estimator MSFE Estimator MSFE INDPRO PC PC SIRb d R70SIRb c,d PAYEMS R8SIRb c,d R8SIRb c,d R8SIRb c,d R30SIRb c,d UNRATE PC PC PC PC CLF16O PC PC PC PC CPIAUC PLS RFSIRb c,d SIRb d R8SIRb c,d CES060 AR4 1 SIRb d R8SIRb c,d PC DPCER PC.ON f PC RFSIRb c,d R8SIRb c,d RPI PLS PLS PLS PLS-1 0.8

9 Model Deninition b The number after RIDGE Comp. is value of regularization param. c R#SIR is regularized SIR. # refers to number of leading PCs used for regularization. d In type a SIR target is y t+h. In type b SIR target is y t. e RFSIR is regularized SIR on most frequently selected PCs by out-of-sample best subset selection. f PC.ON is PCR in which number of components is chosen using Onatski criterion. g PC.ICP1 is PCR in which number of components is chosen using Bai-Ng criterion. h PC.BS is PCR in which number of components is chosen by best subset selection. i PCF.BS is PCR in which number of components is chosen by best subset selection applied on most frequently chosen PCs in out-of-sample.

10 Near-Equivalence in Forecast Accuracy Great variability of best estimator depending on (target-sample-horizon) triplet. Sometimes beating AR(4) remains a challenging task. SIR type estimators do fairly well, PLS is the estimator represented the least in the tables. Competitive edge of SIR is its parsimony: attains practically the same forecasting accuracy using one or two linear combinations of the predictors No Clear Winner result ubiquitous in macro-forecasting literature. Can we explain the results?

11 Selection of PCs Using Different Criteria Figure: Av. Num. of PC Comp. Selected by BSS versus ICps Over

12 MSFE As A Function of Number of Components

13 Common Forecasting Framework

14 Forecasting with Many Predictors Forecasting target variable y with a large set of predictors x R p. Starting forecasting model includes all x predictors: where y t+h = α 0 + α 1x t + α 2w t + ε t+h w t may contain additional regressors such as lags of y t E(ε t+h ) = 0 α 1 x t and α 2 w t are uncorrelated with ε t+h Naturally conducive to OLS to estimate α 1 and α 2 and the feasible forecast is ŷ t+h = ˆα 0 + α 1x t + α 2w t

15 Dimension Reduction Methods OLS estimation can be problematic when p is large relative to T, or variables in x t are nearly collinear, as in macro forecasting = dimension reduction. Dimension reduction methods in regression fall into two categories: Variable Selection: subset of the original predictors is selected for modeling the response (e.g. Stepwise Regression, All Subset Selection, LASSO) = Some predictors are discarded Feature Extraction: linear combinations of the regressors replace the original regressors, therefore reduction of p happens prior to model fitting (e.g. PCR) = All predictors are retained.

16 Dimension Reduction Methods OLS estimation can be problematic when p is large relative to T, or variables in x t are nearly collinear, as in macro forecasting = dimension reduction. Dimension reduction methods in regression fall into two categories: Variable Selection: subset of the original predictors is selected for modeling the response (e.g. Stepwise Regression, All Subset Selection, LASSO) = Some predictors are discarded Feature Extraction: linear combinations of the regressors replace the original regressors, therefore reduction of p happens prior to model fitting (e.g. PCR) = All predictors are retained.

17 Forecasting with Extracted Linear Features Assuming y t+h = α x t + ε t+h Forecasting with extracted features is implemented in two steps: (i) Extract reduced features f t = β x t (ii) Fit the reduced model y t+h = γ f t + ε t+h = (βγ) x t + ε t+h

18 Justifying Reduced Model Proposition Suppose x is a random p-vector with finite first two moments. Assume y = α x + ε, with E(ε) = 0 and Cov(ε, x) = 0 where α R p is unknown, and f = β x, where the p r matrix β is such that (i) for each α, E(α x β x = f) is linear in f R r (LC); (ii) for each α, Var(α x β x = f) is constant in f R r (CVC). Then, y can be decomposed into the sum of a linear function of f and a remainder or error term, as follows, y = µ y + γ (f E(f)) + ɛ where γ = (β Σ x β) β Σ x α R r, µ y = E(y), E(ɛ f) = 0 and var(ɛ f) is constant.

19 Discussion of Proposition It assumes the DGP for target is: y t+h = α x t + ε t+h It places assumptions on marginal distribution of observable predictors No assumptions of underlying latent factor structure plus assumptions on unobservable quantities. It concludes that the reduced forecasting model y t+h = γ f t + ε t+h is equivalent to the DGP, with f t the linear extracted features. We can remove (CVC) and lose homoskedasticity in regression error.

20 Do (LC) and (CVC) Hold in Practice? Conditions (LC) and (CVC) are difficult to verify in practice since β is unknown. However: Conditions (LC) and (CVC) are satisfied for any β if x is multivariate normal. Joint normality on FRED-MD is rejected. Condition (LC) is satisfied for all β if x has an elliptically contoured distribution (Eaton (1986)) FRED-MD satisfies ellipticity tests. Therefore with (LC) satisfied, we have first piece of the puzzle why different feature extraction methods have similar forecast accuracy.

21 Feature Extraction Feature extraction is carried out by solving: where max Corr 2 (h(y), x β i ) H[Var(x β i )] {β i } {β i β i =1} {β i Σx β j =0}i 1 j=1 OLS: H( ) = 1, h( ) = identity PCR: H( ) = identity, h( ) = 1 PLS: H( ) = identity, h( ) = identity RIDGE: H( ) = Var(x β i ) Var(x β i )+κ, h( ) = identity SIR: H( ) = identity, h( ) = E[β i x y]

22 Feature Extraction Feature extraction is carried out by solving: where max Corr 2 (h(y), x β i ) H[Var(x β i )] {β i } {β i β i =1} {β i Σx β j =0}i 1 j=1 OLS: H( ) = 1, h( ) = identity PCR: H( ) = identity, h( ) = 1 PLS: H( ) = identity, h( ) = identity RIDGE: H( ) = Var(x β i ) Var(x β i )+κ, h( ) = identity SIR: H( ) = identity, h( ) = E[β i x y]

23 Feature Extraction Feature extraction is carried out by solving: where max Corr 2 (h(y), x β i ) H[Var(x β i )] {β i } {β i β i =1} {β i Σx β j =0}i 1 j=1 OLS: H( ) = 1, h( ) = identity PCR: H( ) = identity, h( ) = 1 PLS: H( ) = identity, h( ) = identity RIDGE: H( ) = Var(x β i ) Var(x β i )+κ, h( ) = identity SIR: H( ) = identity, h( ) = E[β i x y]

24 Feature Extraction Feature extraction is carried out by solving: where max Corr 2 (h(y), x β i ) H[Var(x β i )] {β i } {β i β i =1} {β i Σx β j =0}i 1 j=1 OLS: H( ) = 1, h( ) = identity PCR: H( ) = identity, h( ) = 1 PLS: H( ) = identity, h( ) = identity RIDGE: H( ) = Var(x β i ) Var(x β i )+κ, h( ) = identity SIR: H( ) = identity, h( ) = E[β ix y]

25 Feature Extraction Feature extraction is carried out by solving: where max Corr 2 (h(y), x β i ) H[Var(x β i )] {β i } {β i β i =1} {β i Σx β j =0}i 1 j=1 OLS: H( ) = 1, h( ) = identity PCR: H( ) = identity, h( ) = 1 PLS: H( ) = identity, h( ) = identity RIDGE: H( ) = Var(x β i ) Var(x β i )+κ, h( ) = identity SIR: H( ) = identity, h( ) = E[β ix y]

26 Prediction Estimate reduced predictors: f t = β x t Fit the reduced model: y t+h = γ ( β x t ) + ε t+h Prediction coefficient: b = β γ Prediction is: ŷ t+h t = b x 0 = γ β x0 Common pattern for b across feature extraction methods: b = scaling factor signal.

27 Table: Summary of Estimators for y = α x + ɛ Metaparameter Scaling Factor Signal OLS 1 Σ x σ xy RIDGE κ (Σ x + κi) 1 σ xy PCR m Σ x (m) σ xy PLS s W PLS(s) (W PLS (s)σxwpls(s)) 1 W PLS (s) σxy SIR d W SIR(d) (W SIR (d)σxwsir(d)) 1 W SIR (d) σxy RSIR m, d W RSIR(d) (W RSIR (d)σx(m)wrsir(d)) 1 W RSIR (d) σxy

28 These estimators have Same signal (σ x,y ): near-equivalent forecast accuracy Different scaling factor: dimension

29 Focus on Targeted Estimators For targeted estimators PLS and SIR scaling factor uses y. How they differ: PLS: WPLS (s) = ( σ xy, Σ x σ xy,..., Σ x s 1 σ xy ) SIR: W SIR (d) = Therefore the reduction is based: ( ) E(x y), Σ x E(x y),..., Σ d 1 x E(x y) For PLS on a moment of the joint distribution F (y, x). For SIR on a moment of the conditional distribution F (y x).

30 Focus on Targeted Estimators For targeted estimators PLS and SIR scaling factor uses y. How they differ: PLS: WPLS (s) = ( σ xy, Σ x σ xy,..., Σ x s 1 σ xy ) SIR: W SIR (d) = Therefore the reduction is based: ( ) E(x y), Σ x E(x y),..., Σ d 1 x E(x y) For PLS on a moment of the joint distribution F (y, x). For SIR on a moment of the conditional distribution F (y x).

31 Focus on Targeted Estimators For targeted estimators PLS and SIR scaling factor uses y. How they differ: PLS: WPLS (s) = ( σ xy, Σ x σ xy,..., Σ x s 1 σ xy ) SIR: W SIR (d) = Therefore the reduction is based: ( ) E(x y), Σ x E(x y),..., Σ d 1 x E(x y) For PLS on a moment of the joint distribution F (y, x). For SIR on a moment of the conditional distribution F (y x).

32 Focus on Targeted Estimators For targeted estimators PLS and SIR scaling factor uses y. How they differ: PLS: WPLS (s) = ( σ xy, Σ x σ xy,..., Σ x s 1 σ xy ) SIR: W SIR (d) = Therefore the reduction is based: ( ) E(x y), Σ x E(x y),..., Σ d 1 x E(x y) For PLS on a moment of the joint distribution F (y, x). For SIR on a moment of the conditional distribution F (y x).

33 Why is SIR more effective? var(x) = var[e(x Y )] + E[var(X Y )] For simplicity, assume Y is categorical: X Y is the restriction of X in the class defined by Y Signal: var[e(x Y )] is between group variation in X Noise: E[var(X Y )] is within group variation PCR mixes up noise and signal when extracting PCs PLS produces ordering of eigen-components according to their importance to cov(x, Y ), i.e. captures linear dependence of X and Y SIR produces ordering of eigen-components according to their importance to Y, linear and non-linear

34 Why is SIR more effective? var(x) = var[e(x Y )] + E[var(X Y )] For simplicity, assume Y is categorical: X Y is the restriction of X in the class defined by Y Signal: var[e(x Y )] is between group variation in X Noise: E[var(X Y )] is within group variation PCR mixes up noise and signal when extracting PCs PLS produces ordering of eigen-components according to their importance to cov(x, Y ), i.e. captures linear dependence of X and Y SIR produces ordering of eigen-components according to their importance to Y, linear and non-linear

35 Why is SIR more effective? var(x) = var[e(x Y )] + E[var(X Y )] For simplicity, assume Y is categorical: X Y is the restriction of X in the class defined by Y Signal: var[e(x Y )] is between group variation in X Noise: E[var(X Y )] is within group variation PCR mixes up noise and signal when extracting PCs PLS produces ordering of eigen-components according to their importance to cov(x, Y ), i.e. captures linear dependence of X and Y SIR produces ordering of eigen-components according to their importance to Y, linear and non-linear

36 Why is SIR more effective? var(x) = var[e(x Y )] + E[var(X Y )] For simplicity, assume Y is categorical: X Y is the restriction of X in the class defined by Y Signal: var[e(x Y )] is between group variation in X Noise: E[var(X Y )] is within group variation PCR mixes up noise and signal when extracting PCs PLS produces ordering of eigen-components according to their importance to cov(x, Y ), i.e. captures linear dependence of X and Y SIR produces ordering of eigen-components according to their importance to Y, linear and non-linear

37 Why is SIR more effective? var(x) = var[e(x Y )] + E[var(X Y )] For simplicity, assume Y is categorical: X Y is the restriction of X in the class defined by Y Signal: var[e(x Y )] is between group variation in X Noise: E[var(X Y )] is within group variation PCR mixes up noise and signal when extracting PCs PLS produces ordering of eigen-components according to their importance to cov(x, Y ), i.e. captures linear dependence of X and Y SIR produces ordering of eigen-components according to their importance to Y, linear and non-linear

38 Why is SIR more effective? var(x) = var[e(x Y )] + E[var(X Y )] For simplicity, assume Y is categorical: X Y is the restriction of X in the class defined by Y Signal: var[e(x Y )] is between group variation in X Noise: E[var(X Y )] is within group variation PCR mixes up noise and signal when extracting PCs PLS produces ordering of eigen-components according to their importance to cov(x, Y ), i.e. captures linear dependence of X and Y SIR produces ordering of eigen-components according to their importance to Y, linear and non-linear

39 Beyond Linear Signals: Sufficient Dimension Reduction

40 Sufficient Reductions Suppose R(x) : R p R d with d p. R(x) is a Sufficient Reduction when no information about y is lost when x is replaced by R(x), i.e. F (y x) = F (y R(x)) How to use in forecasting: For example, if the reduction is linear, the forecasting model is R(x) = β x y t+h = g(β x, ε t+h )

41 Sufficient Reductions Suppose R(x) : R p R d with d p. R(x) is a Sufficient Reduction when no information about y is lost when x is replaced by R(x), i.e. F (y x) = F (y R(x)) How to use in forecasting: For example, if the reduction is linear, the forecasting model is R(x) = β x y t+h = g(β x, ε t+h )

42 Sufficient Reductions Suppose R(x) : R p R d with d p. R(x) is a Sufficient Reduction when no information about y is lost when x is replaced by R(x), i.e. F (y x) = F (y R(x)) How to use in forecasting: For example, if the reduction is linear, the forecasting model is R(x) = β x y t+h = g(β x, ε t+h )

43 Identifying Sufficient Linear Reductions Linear SDR allows identification and estimation of C(β) with F (Y X) = F (Y β T X) General Idea: find a kernel matrix M so that span(m) span(α) SDR methods: different proposals for M

44 For example: In SIR If E(x β x) is linear in β x for any β (LC) such that F (y x) = F (y β x), then C ( Σ 1 x [E(x y) E(x)] ) = C (Var[E(x y)]) C(β) In SIR, M = Σ 1 x Var(E(X Y ))

45 Forecasting with Sliced Inverse Regression SIR Reduces the predictors: (i) Computes non-parametric estimate of Var(E(x y)) (ii) Sets β d (SIR) to be the first d eigenvectors of 1 Σ x Var(E(x y)) Run non-parametric regression y t+h = g( β d (SIR) x t, ε t+h ) and predict. We did not find any forecasting improvement with non-linear g( ). RSIR formed by substituting x with PCs of Σ x.

46 Conclusions

47 Comparing Forecasting Frameworks Forecasting Framework DGP for y t+h DGP for x t DFM Standard y t+h = γ f t + ε t+h x t = Γf t + u t With Irrelevant Factors y t+h = γ f 1,t + ε t+h x t = Γ 1f 1,t + Γ 2f 2,t + u t No Factors for Target y t+h = α x t + ε t+h x t = Γf t + u t Other No Factors y t+h = α x t + ε t+h Conditions on Σ x and β SDR Non-linear Forecast y t+h = g(α x t, ε t+h ) E [x t α x t] = Aβ x t Linear Forecast y t+h = γ (α x t) + ε t+h E [x t α x t] = Aβ x t Most forecasting literature studies shrinkage under factor structure. We study shrinkage estimators under different DGP for observables.

48 Comparing Forecasting Frameworks Forecasting Framework DGP for y t+h DGP for x t DFM Standard y t+h = γ f t + ε t+h x t = Γf t + u t With Irrelevant Factors y t+h = γ f 1,t + ε t+h x t = Γ 1f 1,t + Γ 2f 2,t + u t No Factors for Target y t+h = α x t + ε t+h x t = Γf t + u t Other No Factors y t+h = α x t + ε t+h Conditions on Σ x and β SDR Non-linear Forecast y t+h = g(α x t, ε t+h ) E [x t α x t] = Aβ x t Linear Forecast y t+h = γ (α x t) + ε t+h E [x t α x t] = Aβ x t Most forecasting literature studies shrinkage under factor structure. We study shrinkage estimators under different DGP for observables.

49 Data Specific Conclusions Since there is no non-linear trend in E(y x) and x is elliptically contoured, Proposition 1 yields that the reduced model y t+h = γ (β x t ) + ɛ t+h is a good approximation model to the true population model. What does this mean for prediction?

50 E(y µ y β x) = α Σ x β(β Σ x β) β (x E(x)) = α P β(σ x )(x E(x)) where P β(σx ) = β(β Σ x β) β Σ x is the projection operator onto span(β) relative to (a, b) = a Σ x b. For a given x value, how close E(y x) will be to the truth is reflected by the norm of I P β(σx ), which is controlled solely by β. In consequence, ordering estimators β x with respect to their forecasting accuracy is tantamount to identifying β s with smaller norm.

51 Can we improve the forecasting ability of the model? Forecasting accuracy results do not support non-linearities in the conditional mean of the target variable If there is non-linear signal, it must be in the variance Dimension two or higher in SIR indicates the presence of nonlinear relationships Plots of the residuals of the fitted forecasting model versus the second SIR component indicate that the variance of the forecasting models varies for most targets

52 Can we improve the forecasting ability of the model? The data passed the multivariate elliptical symmetry test but not the normality test (LC) holds but not (NCV) Model-based SDR: ( and Forzani (2015)) When X Y EC p(µ Y,, g Y ), then R(X) = (α (X E(X)), (X E(X)) Σ 1 x (X E(X)) The minimal sufficient reduction has a non-linear component Need to model the variance using (X E(X)) Σ 1 x (X E(X))

53 References Bai J. and Ng, S. (2008). Forecasting economic time series using targeted predictors. Journal of Econometrics, 146(2), Barbarino, A. and, E. (2015). Forecasting with sufficient dimension reductions. Finance and Economics Discussion Series Washington: Board of Governors of the Federal Reserve System, E. and Cook, R. D. (2001). Extending SIR: The Weighted Chi-Square Test, Journal of the American Statistical Association 96, , E. and Yang, J. (2011). Dimension Estimation in Sufficient Dimension Reduction: A Unifying Approach. Journal of Multivariate Analysis, 102, , E. and Forzani, J. (2014). Sufficient reductions in regressions with elliptically contoured inverse predictors. Journal of the American Statistical Association 110, Forni-Hallin-Lippi-Reichlin2005 Forni, M., Hallin, M., Lippi, M. and Reichlin, L. (2005), The Generalized Dynamic Factor Model: One-Sided Estimation and Forecasting, Journal of the American Statistical Association 100(471),

54 Li, K. C. (1991). Sliced inverse regression for dimension reduction (with discussion). Journal of the American Statistical Association 86, McCracken, M.W. and Ng, S. (2015), FRED-MD: A Monthly Database for Macroeconomic Research, Working Paper A, June Steinberger, L. and Leeb, H. (2015), On conditional moments of high-dimensional random vectors given lower-dimensional projections, Stock, J. H. and Watson, M. 2002a. Macroeconomic forecasting using diffusion indices. Journal of Business and Economics Statistics, 20, Stock, J. H. and Watson, M. 2002b. Forecasting using principal components from a large number of predictors. Journal of the American Statistical Association, 97, Stock, J. H. and Watson, M. (2006), Macroeconomic Forecasting Using Many Predictors, Handbook of Economic Forecasting, Graham Elliott, Clive Granger, Allan Timmerman (eds.), North Holland.