Comparing Predictive Accuracy, Twenty Years Later: On The Use and Abuse of Diebold-Mariano Tests

Transcription

1 Comparing Predictive Accuracy, Twenty Years Later: On The Use and Abuse of Diebold-Mariano Tests Francis X. Diebold April 28, / 24

2 Comparing Forecasts 2 / 24

3 Comparing Model-Free Forecasts Models are often non-existent! Forecasts from surveys Forecasts from financial markets Forecasts from prediction markets Forecasts from expert judgment Example: Inflation Forecasts S: Survey (from Survey of Professional Forecasters), {π S t+1,t }T t=1 B: Bond market (from indexed bonds), {π B t+1,t }T t=1 3 / 24

4 S MSE is Lower than B MSE MSE (t = 1,..., T ) S Forecast 1.87 B Forecast 1.92 S wins. But does it really win? Broad issue: How to assess the significance of apparent predictive superiority? 4 / 24

5 The Diebold-Mariano ( DM ) Approach Forecast errors as primitives Assumptions made directly on the forecast errors (actually just on the loss differential...) 5 / 24

6 One Framework, One Assumption, One Test Statistic, One Distribution... Assumption DM : E(d t ) = µ, t cov(d t, d t τ ) = γ(τ), t 0 < var(d t ) = σ 2 <, where d t = L(e t (F 1 )) L(e t (F 2 )) is the loss differential Under assumption DM and H 0 : E(d t ) = 0 we have: d ˆσ d FULL STOP. N(0, 1) END OF STORY. 6 / 24

7 Flexible, Simple, Extensible Flexible: Many loss functions and forecast horizons Simple: Calculation via HAC regression on an intercept (i.e., use ˆσ d = ĝ(0)/t ) Extensible: Easy to allow conditioning via HAC regression on additional covariates 7 / 24

8 A lot hinges on Assumption DM (Covariance Stationarity of the Loss Differential) 8 / 24

9 Thoughts on Assumption DM I: All Assumptions are False; Some are Useful Surely d t is never precisely covariance stationary. But d t may be approximately covariance stationary. 9 / 24

10 Thoughts on Assumption DM II: Special Forecasting Considerations Lend Support Forecasters strive toward forecast optimality, which corresponds to unforecastable covariance-stationary errors (and hence d). White-noise benchmark in 1-step case. 10 / 24

11 Thoughts on Assumption DM III: Non-Stationarities May be Shared Even if forecast error nonstationarities x t do exist, they may be common factors (as information sets overlap across forecasters) and may therefore vanish from the differential. L(e t (F 1 )) = x t + ε 1t, L(e t (F 2 )) = x t + ε 2t, ε 1t cov. stat. ε 2t cov. stat. = d t = L(e t (F 1 )) L(e t (F 2 )) = (ε 1t ε 2t ) cov. stat. 11 / 24

12 Thoughts on Assumption DM IV: Standard Tools Enable Empirical Assessment The approximate validity of assumption DM is ultimately an empirical matter: Plot the loss differential series Examine loss differential correlogram, spectrum Test for unit roots and other nonstationarities Test and monitor for structural evolution/breaks 12 / 24

13 Comparing Models 13 / 24

14 Model Comparison via (Pseudo-) Out-of-Sample Forecasts Base tests on the (pseudo) out-of-sample mean loss differential: d = T t=t +1 (e2 1,t/t 1 e2 2,t/t 1 ) T t, where e t/t 1 is a time-t pseudo-out-of-sample 1-step-ahead forecast error, or recursive residual Unknown models: DM Known models: 1. DM 2. West-Clark-McCracken (WCM) 14 / 24

15 DM vs. WCM for Known Models DM Comparison H 0 : E (L(e(F 1 (M 1 ))) L(e(F 2 (M 2 )))) = 0 Forecast errors as primitives Assumption DM on forecast error loss differential WCM Comparison H 0 : E (L(e(F 1 (M 1 ))) L(e(F 2 (M 2 )))) = 0 Models as primitives Assumption(s) WCM on fully-specified parametric econometric models M 1 and M 2 15 / 24

16 But WCM Model Comparisons Require Traversing a Minefield of Assumptions... Models linear? Non-linear? Models nested? Non-nested? Overlapping? Model disturbances Gaussian? Martingale differences? Model estimation by OLS? GMM? MLE? Model estimation split-sample? Recursive? Rolling? Asymptotics out/in 0? out/in? out/in const? Lengthy surveys of the many flavors of Assumption WCM : W (2006), Forecast Evaluation, Handbook of Economic Forecasting, vol. 1, Elsevier CM (2013), Advances in Forecast Evaluation, Handbook of Economic Forecasting, vol. 2, Elsevier 16 / 24

17 ...and Recent (W)CM Results Support Approximate Normality of the DM Statistic H 0 : E(d t ) 1 T Clark and McCracken (2011, 2013) 17 / 24

18 Summation Model-free environments: DM Model-based environments: Unknown models: DM Known models: DM WCM (closely approximated by DM) So: DM is uniquely relevant in model-free environments, and as good as anything else (and much simpler) in model-based environments 18 / 24

19 But Wait a Minute... Why ever do split-sample model comparisons? 19 / 24

20 (1) Split-Sample Model Comparisons Waste Data (And Hence Sacrifice Power) One should use all of the data for model comparison! True of all optimal model comparison procedures, from both classical and Bayesian perspectives: Classical nested model comparison (LR, LM, Wald) Classical non-nested model comparison (Cox, SIC, AIC, Vuong,...) Bayesian (nested or non-nested) model comparison 20 / 24

21 Classical Model Comparison F = (SSR 1 SSR 2 )/(K 1) SSR 2 /(T K) SSR 1 and SSR 2 are of course full-sample constructs. 21 / 24

22 Bayesian Model Comparison p(m i y) p(m j y) }{{} posterior odds = p(y M i) p(y M j ) }{{} Bayes factor p(m i ) p(m j ) }{{} prior odds The marginal likelihood, p(y M), is a full-sample construct. 22 / 24

23 (2) Split-Sample Model Comparisons Do Not Insure Against Finite-Sample Data Mining All model comparison procedures are subject to strategic data mining in finite samples. Split-sample procedures actually expand the scope of data mining Guarding against strategic data mining requires modeling the mining procedure. That s what White s reality check does. That s not what split-sample does. 23 / 24

24 Whither Pseudo-Out-of-Sample Model Comparison? Interaction with structural break testing Pseudo-out-of-sample model comparison with enforced honesty Non-standard loss functions Comparative historical predictive performance Hansen: Pseudo-out-of-sample procedures might have better, if still imperfect, finite-sample performance Hirano-Wright: Hybrid procedures (e.g., pseudo-out-of-sample with bagging) 24 / 24