Optimal Density Forecast Combinations

Transcription

1 Optimal Density Forecast Combinations Gergely Gánics Banco de España 3th Annual Conference on Real-Time Data Analysis, Methods and Applications Banco de España October 9, 27 Disclaimer: The views expressed herein are those of the author and should not be attributed to the Banco de España or the Eurosystem.

2 What is a density forecast? Density forecasts provide a probabilistic description of uncertainty surrounding forecasts. Q: Where do we use them? / 3

3 What is a density forecast? Density forecasts provide a probabilistic description of uncertainty surrounding forecasts. Q: Where do we use them? A: Weather forecasts, trac forecasts, economics,... / 3

4 What is a density forecast? Density forecasts provide a probabilistic description of uncertainty surrounding forecasts. Q: Where do we use them? A: Weather forecasts, trac forecasts, economics,... Figure: October 26 Monetary Policy Report, Bank of Canada / 3

5 The problem of density forecast combinations Survey of Professional Forecasters: large-scale, quarterly macroeconomic survey maintained by the Philadelphia Fed. Figure: Forecasters' predictions for US GDP in 25 as of 25:Q 3 Forecaster # Forecaster #2 Forecaster #3 2 Equal weights Realization Note: Normal approximation based on midpoints of bins. Original bins: -3% to 6% with % increments. Equal weights seem to perform well. 2 / 3

6 The problem of density forecast combinations Survey of Professional Forecasters: large-scale, quarterly macroeconomic survey maintained by the Philadelphia Fed. Figure: Forecasters' predictions for US GDP in 24 as of 24:Q 3 Forecaster # Forecaster #2 Forecaster #3 2 Equal weights Realization Note: Normal approximation based on midpoints of bins. Original bins: -3% to 6% with % increments. Equal weights seem to perform well. 2 Forecasts display patterns over time. 2 / 3

7 The problem of density forecast combinations: Summary 2 3 Combining densities: hedging against extremes. Predictive distributions are complex objects, unlike point forecasts. Equal weights often perform well, but......do not depend on data. 3/3

8 The problem of density forecast combinations: Summary 2 3 Combining densities: hedging against extremes. Predictive distributions are complex objects, unlike point forecasts. Equal weights often perform well, but......do not depend on data....excellent forecasts receive just as much weight as implausible ones. 3/3

9 The problem of density forecast combinations: Summary 2 3 Combining densities: hedging against extremes. Predictive distributions are complex objects, unlike point forecasts. Equal weights often perform well, but......do not depend on data....excellent forecasts receive just as much weight as implausible ones....do not help understanding models' performance over time. 4 Why not exploit information on past forecasting performance? 3/3

10 The problem of density forecast combinations: Summary 2 3 Combining densities: hedging against extremes. Predictive distributions are complex objects, unlike point forecasts. Equal weights often perform well, but......do not depend on data....excellent forecasts receive just as much weight as implausible ones....do not help understanding models' performance over time. 4 Why not exploit information on past forecasting performance? Can we do better?

11 The problem of density forecast combinations: Summary 2 3 Combining densities: hedging against extremes. Predictive distributions are complex objects, unlike point forecasts. Equal weights often perform well, but......do not depend on data....excellent forecasts receive just as much weight as implausible ones....do not help understanding models' performance over time. 4 Why not exploit information on past forecasting performance? Can we do better? YES! I propose a formal, data-driven method to combine density forecasts, targeting the true conditional predictive density. 3/3

12 Contribution Consistent estimator of weights that minimize discrepancy between combined forecast density and true predictive density (optimality) using goodness-of-t statistics: KolmogorovSmirnov, Cramervon Mises, AndersonDarling (KullbackLeibler Information Criterion). 4 / 3

13 Contribution Consistent estimator of weights that minimize discrepancy between combined forecast density and true predictive density (optimality) using goodness-of-t statistics: KolmogorovSmirnov, Cramervon Mises, AndersonDarling (KullbackLeibler Information Criterion). 2 Monte Carlo simulations support the proposed methodology. Recommendation for practitioners: use estimators based on AndersonDarling statistic or the KLIC. 4 / 3

14 Contribution Consistent estimator of weights that minimize discrepancy between combined forecast density and true predictive density (optimality) using goodness-of-t statistics: KolmogorovSmirnov, Cramervon Mises, AndersonDarling (KullbackLeibler Information Criterion). 2 Monte Carlo simulations support the proposed methodology. Recommendation for practitioners: use estimators based on AndersonDarling statistic or the KLIC. 3 : predicting US industrial production one month ahead using simple ARDL models. Models' weights display time variation. First study to nd that spread and stock returns were valuable for density forecasts during the Great Recession (Ng and Wright (23): point forecasts). Housing data was of great help before and after the crisis. 4 / 3

15 Roadmap 2 Theory: building blocks Theory: results 3 4 Data and models Results 5 / 3

16 Why and how to combine forecasts? Why combine density forecasts? 2 Misspeci cation. Parameter estimation uncertainty, structural breaks. 6/3

17 Why and how to combine forecasts? Why combine density forecasts? 2 Misspeci cation. Parameter estimation uncertainty, structural breaks. How to combine density forecasts? Large literature on density and point forecast evaluation (Diebold et al., 998; Corradi and Swanson, 26a,b,c; Rossi 2 and Sekhposyan, 23, 24, 26). Numerous theoretical and empirical results on optimal point forecast combinations (Bates and Granger, 969; Stock and 3 Watson, 24; Cheng and Hansen, 25). Few results on density forecast combinations (Hall and Mitchell, 27; Geweke and Amisano, 2; Kapetanios et al., 25). 6/3

18 Why and how to combine forecasts? Why combine density forecasts? 2 Misspeci cation. Parameter estimation uncertainty, structural breaks. How to combine density forecasts? Large literature on density and point forecast evaluation (Diebold et al., 998; Corradi and Swanson, 26a,b,c; Rossi 2 and Sekhposyan, 23, 24, 26). Numerous theoretical and empirical results on optimal point forecast combinations (Bates and Granger, 969; Stock and 3 Watson, 24; Cheng and Hansen, 25). Few results on density forecast combinations (Hall and Mitchell, 27; Geweke and Amisano, 2; Kapetanios et al., 25). Consistent estimator of density combination weights is of theoretical and practical importance. 6/3

19 Theory: building blocks Theory: results 2 Theory: building blocks Theory: results 3 4 Data and models Results 7 / 3

20 Theory: building blocks Theory: results Notation I Forecasting environment y t+h is the variable of interest and X t is a vector of predictors. We want to forecast h periods ahead, h < and xed. At forecast origin f, the researcher estimates models m =,..., M in rolling windows of size R, where each estimation is based on the truncated information set I t t R+, containing information between t R + and t. The total number of estimation windows is G. At each t, each model m implies a predictive density with typical element φ m t+h (y t+h I t t R+ ). φ t+h (y t+h I t t R+ ) is the true conditional density. 8 / 3

21 Theory: building blocks Theory: results Notation I Illustration of estimation scheme φ m f G+2 (y f G+2 ) 2 φ m f (y f ) G f G h R+3 f h f f +h f h R+ h G 9 / 3

22 Theory: building blocks Theory: results Notation I Illustration of estimation scheme φ m f G+ (y f G+ ) φ m f G+2 (y f G+2 ) f G h R+2 f G h+ f G h R+3 f G h+2 2 φ m f (y f ) G f h f f +h f h R+ R R R h G 9 / 3

23 Theory: building blocks Theory: results Notation I Illustration of estimation scheme φ m f G+ (y f G+ ) φ m f G+2 (y f G+2 ) f G h R+2 f G h+ f G h R+3 f G h+2 2 φ m f (y f ) G f h f f +h f h R+ R R R h G 9 / 3

24 Theory: building blocks Theory: results Notation I Illustration of estimation scheme φ m f G+ (y f G+ ) φ m f G+2 (y f G+2 ) f G h R+2 f G h+ f G h R+3 f G h+2 2 φ m f (y f ) G f h f f +h f h R+ R R R h G 9 / 3

25 Theory: building blocks Theory: results Notation II Combining density forecasts Researcher uses the convex combination of a set of M predictive densities: φct+h (yt+h Itt R+ ) M X m= t wm φm t+h (yt+h It R+ ). De nition of probabilistic calibration: for a given M X m= () w t t w m φm t+h (yt+h It R+ ) = φt+h (yt+h It R+ ). (2) Objective of this paper Estimation of weights w M. /3

26 Theory: building blocks Theory: results Notation III Measuring (dis)similarity of distributions Probability integral transform or PIT is the combined CDF evaluated at the realization (Rosenblatt, 952; Diebold et al., 998): PIT t+h yt+h φ C t+h (y It t R+ ) dy = ΦC t+h (y t+h I t t R+ ). KullbackLeibler Information Criterion (KLIC): the expected dierence between true and combined log densities (White, 994; Hall and Mitchell, 27): KLIC(Φ t+h (y t+h I t t R+ ), ΦC t+h (y t+h I t t R+ )) = { } E φ log φ t+h (y t+h I t t R+ )} E φ {log φ C t+h (y t+h I t t R+ ). The rst term does not depend on w. / 3

27 Theory: building blocks Theory: results What makes a density forecast good and how to obtain it? Optimality: given the information (models) available at the forecast origin, we want to ensure that the combined forecast is probabilistically calibrated Probabilistic calibration or as close to it as possible. PITt+h U(, ) (Corradi and Swanson, 26c; Gneiting et al., 27). Does not assume knowledge of true DGP. Measures of closeness: PIT-based: Kolmogorov Smirnov, Cramer von Mises and Anderson Darling statistics. Likelihood-based: KLIC. Idea of this paper Take a set of forecasting models and minimize the statistic of choice over combination weights w M. 2/3

28 Uniformity of the PIT I Optimistic forecaster: φ t+(y t+ I t t R+ ) = N (2.9,.52 ). Uncertain forecaster: φ 2 t+(y t+ I t t R+ ) = N (2.45,.262 ). Oracle: φ t+(y t+ I t t R+ ) =.5N (2.9,.52 ) +.5N (2.45,.26 2 ) Forecasters' predictive densities 3 / 3

29 Uniformity of the PIT II Figure: CDFs of PITs Figure: PDFs of PITs (a) Oracle (b) Optimistic (c) Uncertain / 3

30 Uniformity of the PIT III - dierent weights (a) Predictive densities (b) CDF of PITs (c) PDF of mixture PIT / 3

31 Uniformity of the PIT III - dierent weights (a) Predictive densities (b) CDF of PITs (c) PDF of mixture PIT / 3

32 Uniformity of the PIT III - dierent weights (a) Predictive densities (b) CDF of PITs (c) PDF of mixture PIT / 3

33 Formal statistical problem I Theory: building blocks Theory: results Vertical dierence between CDF of PIT and uniform CDF (45 line) at quantile r [, ]: Sample counterpart: Ψ G (r, w) G Ψ(r, w) P(PIT t+h r) r (3) f h t=f G h+ [PIT t+h r] r (4) 6 / 3

34 Formal statistical problem II Theory: building blocks Theory: results Statistics used as objective functions, T G (w): K G (w) sup C G (w) A G (w) r [,] Ψ 2 G KLIC G (w) G Ψ G (r, w) (KolmogorovSmirnov) (r, w) dr (Cramervon Mises) Ψ 2 G (r, w) dr (AndersonDarling) r( r) f h t=f G h+ log φ C t+h (y t+h I t t R+ ) (KLIC) Estimate w by minimizing the statistic of choice, formally: ŵ argmin T G (w) (5) w 7 / 3

35 Theory: building blocks Theory: results Assumptions and consistency result M-estimators, proof builds on Newey and McFadden (994), leading to strongly consistent estimators. Main assumptions: {(yt, Xt ) } is weakly dependent (φ- or α-mixing). C t Combined CDF Φt+h (yt+h It R+ ) is continuously distributed. C t Combined pdf φt+h (yt+h It R+ ) is dominated (KLIC). Rolling window estimation scheme: h< R< as G, T, and xed. Standard identi cation condition allowing for misspeci cation. Theorem (Consistency) Under the assumptions stated above, strongly consistent as G, the estimators are that is w is the (pseudo) true weight vector. a.s. b w, w where 8/3

36 2 Theory: building blocks Theory: results 3 4 Data and models Results 9 / 3

37 in a nutshell Consistency is clearly demonstrated. 2 Favorable results even for samples of size G = 2. 3 Estimator based on AndersonDarling or KLIC statistic is recommended in practice (lowest MSE). 4 The paper contains a number of Monte Carlo simulations, all supporting the conclusions above: AR processes with high/low persistence, Bi- and trimodal densities, Mixture of ARCH + AR models. 2 / 3

38 Data and models Results 2 Theory: building blocks Theory: results 3 4 Data and models Results 2 / 3

39 Data and models Results Figure: Annualized US IP growth between March 96 and February 26 Note: Shaded areas are NBER recession periods. Predicting US industrial production (IP) growth one month ahead, using the AndersonDarling objective function. 22 / 3

40 Modeling approach Data and models Results Following Stock and Watson (23), Granger and Jeon (24), Gürkaynak et al. (23) and Rossi and Sekhposyan (24), I consider linear Autoregressive Distributed Lag (ARDL) models: y t+ = c + β j y t j + γ j x t j + σ 2 iid ε t+, ε t+ N (, ) j= j= y t is annualized US IP growth, x t is either New Private Housing Permits, ISM : New Orders Index, S&P 5, Moody's Baa Spread or (pure AR(2)). Data from May 26 vintage of FRED-MD (McCracken and Ng, 26). All ve models estimated in rolling windows of R = 2 months, weights calculated over G = 8 months. Forecast target dates March 985 February 26 (P = 372 months), which is the out-of-sample evaluation period. 23 / 3

41 Benchmarks Data and models Results KLIC (Hall and Mitchell, 27; Geweke and Amisano, 2). 2 AR(2)-N (Del Negro and Schorfheide, 23). 3 Equal weights (Kascha and Ravazzolo, 2; Rossi and Sekhposyan, 24). 4 (Bayesian Information Criterion, Bayesian Model Averaging). 24 / 3

42 What do we learn from equal weights? The uninformativeness of equal weights / 3

43 Time variation of estimated weights 985:M3 26:M AndersonDarling weights / 3

44 What did we learn about the models? No single model dominates the model set. Considerable time-variation of well-performing models. AndersonDarling weights have economic meaning. Housing permits contains valuable predictive information before and after the Great Recession. S&P 5 and corporate spread receive large weights during the crisis. This is beyond the results of Ng and Wright (23)! 27 / 3

45 Empirical results - PITs Figure: Normalized histograms of PITs (a) AD weights (b) KLIC weights (c) Equal weights (d) AR(2)-N model Note: Horizontal red dashed line corresponds to uniform density. 28 / 3

46 Data and models Results The AndersonDarling weights pass tests of uniformity! Test statistics and p-values (in parentheses) of Rossi and Sekhposyan's (26) test. H : PIT is uniformly distributed. Models KolmogorovSmirnov Cramervon Mises AD weights.9 (.38).24 (.22) KLIC weights.28 (.8).42 (.6) Equal weights.39 (.5).5 (.4) AR(2)-N.3 (.8).4 (.9) Note: p-values calculated with the HAC estimator by Newey and West (987) using.75p /3 = 5 lags. Number of Monte Carlo simulations was 2,. 29 / 3

47 Conclusions and further research Consistent weight estimator for density forecast combinations. 2 Theoretically appealing and works well in Monte Carlo simulations. 3 the proposed framework delivers high quality density forecasts of US industrial production, outperforming the equal weights benchmark! 4 Weights are economically meaningful (housing bubble and leverage/spread). 5 First paper to nd these patterns for density forecasts (Ng and Wright (23): point forecasts only). 6 Potential extensions: Penalized estimation of w, biasing to zero. Testing for structural breaks. Financial applications, such as Value-at-Risk. 3 / 3

48 The End Thank you for your attention!

49 Additional results Knowledge of true DGP not required Monte Carlo DGP with estimated parameters Additional empirical results In-sample t Robustness check 3 / 3

50 Example inspired by Corradi and Swanson (26b,c) True DGP for yt+ is a stationary normal AR(2) process. True predictive density of yt+ conditional on It = {yt, yt }: φ t+ (yt+ It ) = N (α yt + α2 yt, σ 2 ). The distribution of yt+ conditional on yt (6) alone is also normal, φ t+ (yt+ yt ) = N (e αyt, σ e2 ). (7) If the researcher uses the AR() model instead of the AR(2) model, the forecast is still probabilistically calibrated, as given the information set (yt ), the predictive density is correct. Probabilistic calibration does not require knowledge of the true DGP. Back 32/3

51 Monte Carlo setup Constant and constant plus ARCH() model M : y t+ = c + ν t+ M2 : y t+ = c 2 + σ2,t+ 2 ε t+, σ 2 2,t+ = α + α ε 2 t with ν t+ iid N (, σ 2 ) and ε t+ iid N (, ) DGP is a mixture of M and M2 with (w, w 2 ) = (.4,.6). Irrelevant M3 matches rst 2 moments of true density: M3 : y t+ = c 3 + η t+ η t+ iid N (, σ 2 3 ) c 3 = w ĉ + w 2 ĉ 2 and σ 2 3 = w σ 2 + w 2 σ 2 2,t+ Table: Parameters of M, M2 and M3 Model c σ 2 α α M.3 M / 3

52 Monte Carlo Histograms, w = (.4,.6, ) KS CvM AD KLIC G = G = G = weights of normal density (M) weights of ARCH density (M2) weights of irrelevant density (M3) Note: Histograms and kernel density estimates based on 2 MC replications. Back 34 / 3

53 Further benchmarks Bayesian Information Criterion (Schwarz (978); Kass and Raftery (995); Hoeting et al. (999)): BIC m 2 f t=f R log l m (y t+ z m t ; θ m ) + k m log(r). (8) Bayesian Model Averaging (Kass and Raftery (995); Hoeting et al. (999); Rossi and Sekhposyan (24)): w m = exp(.5bic m) 5 i= exp(.5bic i). (9) 35 / 3

54 Time variation of estimated weights 985:M3 26:M AndersonDarling weights KLIC weights / 3

55 Time variation of estimated weights 985:M3 26:M BIC weights BMA weights / 3

56 Empirical results - PITs Figure: Normalized histograms of PITs (a) AD weights (b) KLIC weights (c) Equal weights (d) AR(2)-N model (e) BIC weights (f) BMA weights Note: Horizontal red dashed line corresponds to uniform density. 38 / 3

57 Tests of uniformity of PITs Rossi and Sekhposyan (26) test on correct specication of conditional predictive densities. H : PIT is uniformly distributed. Models KolmogorovSmirnov Cramervon Mises AD weights.9 (.38).24 (.22) KLIC weights.28 (.8).42 (.6) Equal weights.39 (.5).5 (.4) AR(2)-N.3 (.8).4 (.9) BIC.6 (.8).32 (.7) BMA.28 (.).38 (.2) Note: Test statistic (p-value). p-values calculated with the HAC estimator by Newey and West (987) using.75p /3 = 5 lags. Number of Monte Carlo simulations was 2,. Back 39 / 3

58 Did in-sample t drive weights? Figure: Ratios of inverse in-sample residual variances Note: The sample period (end of the last rolling window of size R = 2) starts in February 985 and ends in January 26, with a total number of P = 372 months. Housing stands for Housing Permits, NOI is ISM: New Orders Index, while Spread is Moody's Baa Corporate Bond Yield minus Fed funds rate. Back 4 / 3

59 Adding US house price index to the set of predictors With house prices Without house prices Back / 3

60 References I Bates, J. M. and Granger, C. W. J. (969). The Combination of Forecasts. OR, 2(4): Cheng, X. and Hansen, B. E. (25). Forecasting with factor-augmented regression: A frequentist model averaging approach. Journal of Econometrics, 86(2): Corradi, V. and Swanson, N. R. (26a). Bootstrap conditional distribution tests in the presence of dynamic misspecication. Journal of Econometrics, 33(2): Corradi, V. and Swanson, N. R. (26b). Chapter 5 Predictive Density Evaluation. In Elliott, G., Granger, C. W. J., and Timmermann, A., editors, Handbook of Economic Forecasting, volume, pages Elsevier. 42 / 3

61 References II Corradi, V. and Swanson, N. R. (26c). Predictive density and conditional condence interval accuracy tests. Journal of Econometrics, 35(-2): Del Negro, M. and Schorfheide, F. (23). DSGE Model-Based Forecasting. In Elliott, G. and Timmermann, A., editors, Handbook of Economic Forecasting, volume 2-A, pages Elsevier, Amsterdam. Diebold, F. X., Gunther, T. A., and Tay, A. S. (998). Evaluating density forecasts. International Economic Review, 39(4): Geweke, J. and Amisano, G. (2). Optimal prediction pools. Journal of Econometrics, 64(): / 3

62 References III Gneiting, T., Balabdaoui, F., and Raftery, A. E. (27). Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2): Granger, C. and Jeon, Y. (24). Forecasting Performance of Information Criteria with Many Macro Series. Journal of Applied Statistics, 3(): Gürkaynak, R. S., Kisacikoglu, B., and Rossi, B. (23). Do DSGE Models Forecast More Accurately Out-of-Sample than VAR Models? volume 32: VAR Models in Macroeconomics - New Developments and Applications: Essays in Honor of Christopher A. Sims of Advances in Econometrics, pages Emerald Group Publishing Limited. 44 / 3

63 References IV Hall, S. G. and Mitchell, J. (27). Combining density forecasts. International Journal of Forecasting, 23():3. Hoeting, J. A., Madigan, D. A., Raftery, A. E., and Volinsky, C. T. (999). Bayesian Model Averaging: A Tutorial. Statistical Science, 4(4): Kapetanios, G., Mitchell, J., Price, S., and Fawcett, N. (25). Generalised density forecast combinations. Journal of Econometrics, 88():565. Kascha, C. and Ravazzolo, F. (2). Combining ination density forecasts. Journal of Forecasting, 29(-2):2325. Kass, R. E. and Raftery, A. E. (995). Bayes Factors. Journal of the American Statistical Association, 9(43): / 3

64 References V McCracken, M. W. and Ng, S. (26). FRED-MD: A Monthly Database for Macroeconomic Research. Journal of Business & Economic Statistics, 34(4): Newey, W. K. and McFadden, D. (994). Large sample estimation and hypothesis testing. In McFadden, D. and Engle, R., editors, Handbook of Econometrics, volume 4, pages Elsevier, Amsterdam. Newey, W. K. and West, K. D. (987). A Simple, Positive Semi-Denite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix. Econometrica, 55(3):73. Ng, S. and Wright, J. H. (23). Facts and Challenges from the Great Recession for Forecasting and Macroeconomic Modeling. Journal of Economic Literature, 5(4): / 3

65 References VI Rosenblatt, M. (952). Remarks on a Multivariate Transformation. Ann. Math. Statist., 23(3): Rossi, B. and Sekhposyan, T. (23). Conditional predictive density evaluation in the presence of instabilities. Journal of Econometrics, 77(2):9922. Rossi, B. and Sekhposyan, T. (24). Evaluating predictive densities of US output growth and ination in a large macroeconomic data set. International Journal of Forecasting, 3(3): Rossi, B. and Sekhposyan, T. (26). Alternative Tests for Correct Specication of Conditional Predictive Densities. Working Paper No. 758, Barcelona GSE. 47 / 3

66 References VII Schwarz, G. (978). Estimating the Dimension of a Model. The Annals of Statistics, 6(2): Stock, J. H. and Watson, M. W. (23). Forecasting output and ination: The role of asset prices. Journal of Economic Literature, 4(3): Stock, J. H. and Watson, M. W. (24). Combination forecasts of output growth in a seven-country data set. Journal of Forecasting, 23(6):4543. White, H. (994). Estimation, Inference and Specication Analysis. Number 22 in Econometric Society Monographs. Cambridge University Press, Cambridge. 48 / 3