Discussion of Comparing Predictive Accuracy, Twenty Years Later: A Personal Perspective on the Use and Abuse of Diebold-Mariano Tests by F. X.

Transcription

1 Dscusson of Comparng Predctve Accuracy, Twenty Years Later: A Personal Perspectve on te Use and Abuse of Debold-Marano Tests by F. X. Debold Dscusson by Andrew J. Patton Duke Unversty 21 Aprl Introducton Ts nterestng artcle provdes an nsder s vew of te early days of te modern forecast comparson lterature, wc mgt be dated as begnnng wt Debold and Marano (1995) and West (1996), papers wc spawned numerous extensons and re nements over te subsequent years. 1 Te artcle also revews ow forecast evaluaton metods ave been employed n te lterature, o erng some crtcsm of te use of splt-sample forecast comparson tecnques (lke te Debold- Marano test) wen appled to model comparson rater tan forecast comparson. My dscusson s a personal perspectve on a personal perspectve on te DM test, and t s peraps a bold move to debate te DM test wt D, but below I wll do so. Te key ngredent n a DM test s te loss d erental: d t = L 1;t L 2;t I tank Ja L for elpful comments. Contact address: Department of Economcs, Duke Unversty, 213 Socal Scences Buldng, Box 90097, Duram NC Emal: andrew.patton@duke.edu. 1 I denote ts te modern forecast comparson lterature, as te lterature on forecast evaluaton and comparson stretces all te way back to te very brt of econometrcs, wt an artcle by Cowles (1933). (1) 1

2 were L : YY f!r + s some loss functon, mappng from te range of possble values for te target varable and te forecast to te non-negatve real lne. 2 Wt ts varable de ned, a test of equal predctve accuracy s easly seen to be equvalent to a smple test tat te mean of ts scalar tme seres s zero: H 0 : E L 1;t = E L 2;t, H 0 : E [d t ] = 0 (2) Debold prases te vrtue of te sngle assumpton tat s requred to mplement te DM test, namely tat of covarance statonarty of d t : Wt suc an assumpton n place, a central lmt teorem for te sample mean, d T ; can be nvoked, and te propertes of te DM test easly establsed. As noted n s footnote 2, te assumpton of covarance statonarty s stronger tan needed; wat s actually desred s smply a set of assumptons tat are su cent for te DM test statstc to be asymptotcally standard Normal under te null ypotess of equal predctve accuracy. Tus, one mgt consder swappng Assumpton DM (equaton 1 n te artcle) wt te followng ger-level assumpton: Assumpton AN: 8 < : p T dt D! N 0; 2 and 9 ^ 2 T s:t: ^ 2 T p! 2 ; as T! 1 (3) In ts dscusson I argue tat neter covarance statonarty nor asymptotc Normalty more generally are needed for a test to be a DM-type test. 2 Asymptotc Normalty s nce but not necessary In s paper, Debold argues tat tere s an mportant dstncton between comparng forecasts over some storcal perod and comparng models at ter pseudo-true parameter values. I completely concur, and I furter suggest tat, fundamentally, te bggest dstncton between te many tests tat ave been proposed n te last twenty years s not te sape of te lmtng dstrbuton of te test statstcs, or te varous terms tat appear n te asymptotc varance, but rater te null ypotess tat te statstc s used to test. Ts ypotess summarzes te economc 2 Te notaton ere s slgtly more general tan tat used n Debold s paper. Ts notaton allows for loss functons tat cannot be expressed n terms of te forecast error, suc as tose based on proportonal errors, y=^y; wc commonly appear n volatlty forecastng applcatons, see Patton (2011) for example, as well as tose based on sgn forecast errors, sgn (y) sgn (^y), see Cumby and Modest (1987) for example. 2

3 queston beng posed, and d erences can matter greatly for te economc conclusons drawn. Ts dstncton s noted n Debold and Marano (1995) and West (1996), but te clearest dscusson of ts dstncton s n Gacomn and Wte (2006). Tese latter autors focus explctly on ypoteses nvolvng estmated parameters, and one of te contrbutons of ter paper s to provde prmtve condtons under wc suc tests, lke te DM test, can be appled. WCM H 0 : E L t ( 1) DM-GW H 0 : E L t (^ 1;t ) = E = E L t ( 2) t (^ 2;t ) L (4) (5) Tests comparng models at ter pseudo-true parameter values correspond to WCM tests, 3 wle tests comparng forecasts (from estmated models, or from surveys, judgement forecasts, etc.) correspond to DM-GW tests. I propose tat te de nng feature of a DM-type test s not te asymptotc Normalty of te test statstc (and te smple use of a HAC estmator of te long-run varance of d t ), but rater ts focus on tests of forecasts, rater tan tests of models evaluated at ter pseudo-true parameter values. If one accepts ts as a key feature, ten any test of a null lke tat n equaton (5) mgt reasonably be called a DM-type test, ncludng tose tat do not ave asymptotcally Normal test statstcs. 3 Non-Normal DM tests In a recent workng paper, L and Patton (2013) extend te applcablty of te tests of Debold and Marano (1995), West (1996), Wte (2000), Gacomn and Wte (2006), McCracken (2007), and oters, to accommodate latent target varables lke tose encountered n nancal applcatons, for example, forecastng volatlty, correlaton, beta, or jump rsk. Te goal n tat paper s to provde condtons under wc tests based on a proxy constructed usng g frequency data (e.g., realzed correlaton ) ave te same propertes as te correspondng test based on te true, latent, target varable (e.g., ntegrated correlaton ). As part of te evaluaton of te proposed teory, te smple case were te target varable s observable s consdered, wc corresponds to a standard 3 I follow Debold n callng tese tests WCM after West (1996) and Clark and McCracken (2001), altoug several oter autors ave focussed on tests of nulls nvolvng populaton parameters, ncludng Wte (2000), Corrad and Swanson (2002), Ellott, Komunjer and Tmmermann (2005) and Ross (2005). 3

4 DM-type test. One smulaton desgn n L and Patton (2013) focuses on correlaton forecastng, usng a bvarate stocastc volatlty process from Barndor -Nelsen and Separd (2004) as te data generatng process, and comparng forecasts from two DCC models (Engle, 2002) for daly correlatons. 4 Te null to be tested s: 2 2 H 0 : E T t rue DCC 1 t ^1;t = E T t rue DCC 2 t ^2;t (6).e., a DM-type null ypotess. Te usual eteroskedastcty and autocorrelaton (HAC) robust t-statstcs for ts test were found to be far from Normal under te null ypotess. After some nvestgaton, t appeared tat te need for many lags n te HAC varance estmaton was dstortng te beavor of te test statstc, promptng te consderaton of te xed b asymptotcs of Kefer and Vogelsang (2005). Te Kefer-Vogelsang approac also uses a t test, wt possbly many autocorrelatons ncluded n te HAC varance estmate, but te lmtng dstrbuton s not Normal; rater t follows a non-standard dstrbuton, wt crtcal values provde by Kefer and Vogelsang. As Table 1 below reveals, te Kefer-Vogelsang approac to obtanng a (non-normal) lmtng dstrbuton for te DM t-statstc provdes better sze control tan te famlar approac based on asymptotc Normalty. [ INSERT TABLE 1 ABOUT HERE ] Oter examples of non-normal DM-type tests exst n te lterature. Gacomn and Ross (2010) propose a forecast comparson test tat allows for te possblty of tme varaton n te relatve performance of te competng forecasts over te sample. Importantly, te null ypotess s a functon of te forecasts evaluated usng estmated parameters, not at te parameters probablty lmts. Te lmtng dstrbuton of te test statstc n ts applcaton s a functonal of a Brownan moton. Te realty ceck of Wte (2000) s presented as a test of a WCM-type null, focusng on forecasts generated usng te pseudo-true values of te parameters, owever a DM-GW verson of te realty ceck s a natural extenson (and ndeed s nted at n Wte s artcle). Te test statstc n tat case as a lmtng dstrbuton tat s te maxmum of a vector of correlated Normal random varables, and s tus also non-normal. 4 Detals on te data generatng process and forecast models are omtted n te nterests of space; nterested readers are referred to L and Patton (2013). 4

5 4 Summary One of te man arguments made n Debold s nterestng artcle s tat f te objectve of an analyss s to compare forecasts over some sample perod, ten te varables of te nterest are te actual forecasts obtaned n tat sample perod, not tose tat would be obtaned usng te pseudo-true parameters of te models (f any) tat generated tem. I agree, and followng troug on ts pont, t suggests tat te key aspect of te DM approac to forecast comparson s te coce to state te null ypotess as a functon of estmated parameters rater tan pseudo-true probablty lmts. Tus, wle covarance statonarty of te loss d erences ( Assumpton DM n te artcle), or, more generally, asymptotc normalty of te sample mean of te loss d erences, s convenent wen applcable, t s not necessary for a DM-type test; n some applcatons a DM-type test wll ave a non-normal lmt dstrbuton. References [1] Barndor -Nelsen, O. E. and N. Separd, 2004, Econometrc Analyss of Realzed Covaraton: Hg Frequency Based Covarance, Regresson, and Correlaton n Fnancal Economcs, Econometrca 72, [2] Clark, T. E. and M. W. McCracken, 2001, Tests of Equal Forecast Accuracy and Encompassng for Nested Models, Journal of Econometrcs 105, [3] Corrad, V. and N. R. Swanson, 2002, A Consstent Test for Out of Sample Nonlnear Predctve Ablty, Journal of Econometrcs 110, [4] Cowles, Alfred 3rd, 1933, Can Stock Market Forecasters Forecast?, Econometrca 1, [5] Cumby, R. E. and D. M. Modest, 1987, Testng for Market Tmng Ablty: A Framework for Forecast Evaluaton, Journal of Fnancal Economcs 19, [6] Debold, F. X. and R. S. Marano, 1995, Comparng Predctve Accuracy, Journal of Busness and Economc Statstcs 13, [7] Ellott, G., I. Komunjer and A. Tmmermann, 2005, Estmaton and Testng of Forecast Ratonalty under Flexble Loss, Revew of Economc Studes 72, [8] Engle, R. F., 2002, Dynamc Condtonal Correlaton: A Smple Class of Multvarate Generalzed Autoregressve Condtonal Heteroskedastcty Models, Journal of Busness and Economc Statstcs 20, [9] Gacomn, R. and B. Ross, 2010, Forecast Comparsons n Unstable Envronments, Journal of Appled Econometrcs 25,

6 [10] Gacomn, R. and H. Wte, 2006, Tests of Condtonal Predctve Ablty, Econometrca 74, [11] Kefer, N. M. and T. J. Vogelsang, 2005, A New Asymptotc Teory for Heteroskedastcty- Autocorrelaton Robust Tests, Econometrc Teory 21, [12] L, J. and A. J. Patton, 2013, Asymptotc Inference about Predctve Accuracy usng Hg Frequency Data, workng paper, Duke Unversty. [13] McCracken, M. W., 2007, Asymptotcs for Out-of-Sample Tests of Granger Causalty, Journal of Econometrcs 140, [14] Newey, W. K. and K. D. West, 1987, A Smple, Postve Semde nte, Heteroskedastcty and Autocorrelaton Consstent Covarance Matrx, Econometrca 55, [15] Patton, A. J., 2011, Volatlty Forecast Comparson usng Imperfect Volatlty Proxes, Journal of Econometrcs 160, [16] Ross, B., 2005, Testng Long-Horzon Predctve Ablty wt Hg Persstence, and te Meese- Rogo Puzzle, Internatonal Economc Revew 46, [17] West, K. D., 1996, Asymptotc Inference about Predctve Ablty, Econometrca 64, [18] Wte, H., 2000, A Realty Ceck for Data Snoopng, Econometrca 68,

7 Table 1: Rejecton probabltes under te null Out-of-sample sze In-sample sze Normal Kefer-Vogelsang Notes: Ts table presents te proporton of smulatons n wc te null ypotess of equal predctve accuracy s rejected at te 0.05 level, n a desgn n wc te null ypotess s true. All tests are based on a standard Debold-Marano test statstc, wt HAC standard errors constructed usng Newey and West (1987). Te n-sample sze refers to te number of observatons used to estmate te forecast models; te out-of-sample sze (P ) refers to te number of observatons used to compare te forecast models. In te left panel te number of lags used n te HAC estmate s set to 3P 1=3 and crtcal values are obtaned from te standard Normal dstrbuton. In te rgt panel te number of lags s set to P=2 and crtcal values are obtaned from Kefer and Vogelsang (2005). Furter detals are avalable n L and Patton (2013). 7