Forecast Evaluation and Comparison Theory

Transcription

1 Forecas Evaluaion and Comparison Theory Andrew Paon Deparmen of Economics Duke Universiy OMI-SoFiE Summer School, 2013 Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

2 Papers o be covered * Wes, K. D., 1996, Asympoic Inference Abou Predicive Abiliy, Economerica, 64(5), * Giacomini, R. and Whie, H. 2006, Tess of Condiional Predicive Abiliy, Economerica, 74(6), * Diebold, F.X., and Mariano,R.S., 1995, Comparing Predicive Accuracy, Journal of Business and Economic Saisics, 13(3), Wes, K.D., 2005, Forecas Evaluaion, in Handbook of Economic Forecasing, G. Ellio, C.W.J. Granger and A. Timmermann ed.s, Norh Holland Press, Amserdam. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

3 De ning an opimal forecas De ne an opimal (poin) forecas for loss funcion L : R Y! R + as: Ŷ+h, arg min E [L (Y +h, ŷ) jf ] ŷ 2Y Tradiional ess of forecas opimaliy/forecaser raionaliy are based on he assumpion ha he forecaser s loss funcion is quadraic (aka MSE) which implies L (y, ŷ) = (y ŷ) 2 Ŷ +h, = E [Y +h jf ] Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

4 Review of basic forecas evaluaion mehods Tradiional mehods for forecas evaluaion include he Mincer-Zarnowiz regression (MZ, 1969, Theil 1958) Y = β 0 + β 1 Ŷ + u H 0 : β 0 = 0 \ β 1 = 1 vs. H a : β 0 6= 0 [ β 1 6= 1 Or ess for predicabiliy in he forecas error, e Y Ŷ e = α 0 + α 1 e 1 + u H 0 : α 0 = α 1 = 0 vs. H a : α 0 6= 0 [ α 1 6= 0 Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

5 Review of basic forecas comparison mehods The simples and mos widely-used forecas comparison es is he Diebold-Mariano (1995) es: L 1 = L Y, Ŷ 1 L 2 = L Y, Ŷ 2 d L 1 L 2 H 0 : E [d ] = 0 vs. H a : E [d ] 6= 0 where L : R Y! R + is he forecas user s loss funcion. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

6 The approach of Wes (1996) Wes s criicism of he above approaches relaes o he fac ha many (in fac, mos) forecass used in economics and nance are based on (parameric) models: Ŷ = Ŷ X, ˆβ where Ŷ is a funcion mapping he predicor variables, X, and he parameers, β, o a forecas. For example: Ŷ X, ˆβ = X 0 ˆβ Which regression o we really wan o run? Y = β 0 + β 1 Ŷ X, ˆβ + u Y = β 0 + β 1 Ŷ (X, β ) + u where ˆβ! p β as T!. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

7 Wes s moivaion Ex-pos forecass are made when one is no ineresed in ex-ane predicion bu evaluaion of predicive abiliy of a model given a pah for some unmodelled (bu endogenous in a larger sysem) se of variables. Tha is, sandard MZ and DM ess, based on Ŷ X, ˆβ, are useful for evaluaing a given forecasing model, Ŷ, when evaluaed a a given sequence of predicor variables and parameer esimaes, raher han across all values of he predicor variables and a he pseudo-rue parameer, β. Sandard ess evaluae he forecas, raher han he forecasing model. Giacomini and Whie (2006) provide a heoreical moivaion for sandard ess, complemening he work of Wes (1996). Wes s conribuion is o provide asympoic heory for ess of he form: H 0 : E f Y, Ŷ (X, β ), Z = 0 Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

8 Diebold s personal perspecive (2012) The Diebold-Mariano es was inended for comparing forecass... The DM es was no inended for comparing models. Unforunaely, however, much of he large subsequen lieraure uses DM-ype ess for comparing models, in (pseudo-) ou-of-sample environmens. In ha case, much simpler ye more compelling full-sample model comparison procedures exis. Tha is, Diebold compleely agrees wih Wes on he inerpreaion of DM (and GW) ess: hey are useful for learning abou comparaive hisorical predicive performance He disagrees on he usefulness of ou-of-sample forecas comparison as a means of learning abou models. We will cover boh of hese mehods in his lecure... Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

9 Wes s framework and noaion E [f ] E f Y, Ŷ (X, β ), Z is he (l 1) vecor of momens ha are of ineres, where Z is a vecor of oher observable daa (may be empy) β is he (k 1) pseudo-rue parameer of he forecasing model. Noe k < (so no non-parameric or semi-parameric models allowed here) ˆβ is an esimaor of β using daa up unil dae. p R consisen τ 1 is he longes forecas horizon (may produce a vecor of forecass a each poin in ime) R and P are he number of obs in he in-sample and ou-of-sample periods respecively. (Boh R, P! ) T = R + P is he full sample, and π = lim P/R as T! 2 [0, ] is lengh of he OOS period relaive o he in-sample period. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

10 Key echnical assumpions f (β) is wice coninuously di ereniable wih respec o β (McCracken 2000 JoE relaxed his assumpion) " h i 0 0 E vec f β, f 0 4d #, h0 < for some d > 1, where f f (β ), f β f (β ) / β, h h (β ) is he momen funcion used o obain ˆβ h vec f β 0, f 0, h0 i 0 is covariance saionary. (Noe ha saionary + srong mixing (nex assumpion) implies ergodiciy) Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

11 Key echnical assumpions, con d h vec f β E i 0 f β, (f 0 E [f 0 ]), h 0 is srong mixing (α-mixing) wih mixing coe ciens of size 3d/ (d 1) (i.e. sronger for d near 1) Le α (G,H) sup fg 2G,H 2Hg jpr [G \ H] Pr [G ] Pr [H]j And α (m) sup n α F n, F n+m, where F b a σ (Z a, Z a+1,..., Z b ) Then if α (m) = O m a δ for some δ > 0, he process is srong mixing of size a The mixing assumpion allows for some ime series dependence in he daa (serial correlaion, ARCH, ec.) bu imposes ha i dies ou evenually. Much beer han imposing uncorrelaedness or iidness Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

12 Key echnical assumpions: he esimaor A key assumpion of he esimaor is ha ˆβ (k 1) β (k 1) = B () H () (k q) (q1) where B ()! B a marix of rank k H () = 1 s.. E [h (β )] = 0 h s (β ) s=1 This may look a bi arbirary or resricive, bu i acually ness all GMM esimaors, which includes OLS, MLE. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

13 Key echnical assumpions: he esimaor, con d Consider a generic GMM opimisaion problem: ˆβ arg min β2b FOC 0 = 1 h s s=1 β h s (β)! W () s=1! 0 ˆβ W () 1 1! h s (β) s=1! h s ˆβ s=1 Take a rs-order mean-value expansion of he second wo erms in he FOC W () 1 h s s=1 ˆβ = W () 1 h s (β ) s=1! 1 h s β +W () s=1 β ˆβ β Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

14 Key echnical assumpions: he esimaor, con d 0 = =! 0 1 h s ˆβ s=1 β 0 W ()! 0 1 h s ˆβ s=1 β 0 W () 1 + 1! 0 h s ˆβ s=1 β 0 W () A () 1 h s (β ) s=1 A () 1! h s ˆβ s=1 h s (β ) s=1 1 h s s=1 β! β 1 h s β s=1 β! ˆβ ˆβ β β Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

15 Key echnical assumpions: he esimaor, con d So ˆβ β = A () 1! 1 h s β 1 A () s=1 β h s (β ) s=1 {z } {z } B () hus H () = 1 h s (β ) s=1 B () = A () 1! 1 h s β A () s=1 β H () Thus he assumpion ha ˆβ common esimaors. β = B () H () allows for a variey of I does rule ou nonparameric and semi-parameric esimaors. (Bayesian esimaors oo?) Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

16 The esimaion and forecasing samples R! and P! as T R + P!. So boh in-sample and ou-of-sample mus be large. (Rules ou rolling-window forecass?) lim P/R = π as T!. π 2 [0, ]. π is aken as given (reasonable?) If π 2 (0, ) hen he wo samples grow a he same rae If π = hen he ou-of-sample grows faser han he in-sample If π = 0 hen he in-sample grows faser han he ou-of-sample (and hen esimaion error in ˆβ can be ignored) Noe ha he max forecas horizon, τ, is xed, so needs i o be τ << R, P. Tha is, we can handle very long-horizon forecass. See Valkanov (JFE 2003) for his ype of problem. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

17 Main resul - par 1 The main resul in Wes (1996) is equaion 4.1 (p1072):! p T 1 P P ˆf +τ E [f+τ] = p T 1 P P (f+τ =R +1 =R +1 +FB p 1 T P H () =R +1 +o p (1) E [f +τ])! where B ()! B F E [ f (β ) / β] And so he asympoic behaviour of he LHS, which is based on esimaed parameers will be a eced by boh he corresponding quaniy wih no esimaion error and a erm which is a eced by he choice of esimaor. If no parameers are esimaed, hen B = 0 and so only he rs erm maers. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

18 Main resul - par 2 To presen he second main resul (Theorem 1) more noaion is needed 0 Γ (j) E (f E [f ]) f j E [f ] h i Γ fh (j) E (f E [f ]) h 0 j h i Γ hh (j) E h h 0 j S ab S Π j= Γ ab (j), for ab 2 f, fh, hhg S S fh B 0 8 < : BS 0 fh BS hh B S S fh B 0 BSfh 0 V β 0 if π = 0 1 π 1 ln (1 + π) if π 2 (0, ) 1 if π = Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

19 Main resul - par 2 Theorem 4.1: p P ( f P E [f ])! N (0, Ω) Ω = S + Π FBS 0 fh + S fhb 0 F 0 + 2ΠFV β F 0 where f P 1 P T ˆf +τ =R +1 If no esimaion is needed, hen B = 0 (and V β = 0) and his resul reduces o he resul of Diebold and Mariano (1995). Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

20 Commens on he main resul If lim P/R π = 0, hen esimaion error in ˆβ can be ignored. I will be of a lower order of magniude han he esimaion error in f P. Migh hink ha his holds in pracice when P/R is small (0.1? 0.25??). Esimaion error can also be ignored if F E [ f (β ) / β] = 0, i.e., if he rs derivaive of he momen of ineres is zero a he pseudo-rue parameer. F Holds when he esimaion objecive funcion maches he OOS loss funcion The exra wo erms ha arise from esimaion error may acually cancel ou if a paricular covariance condiion is sais ed (no likely.) The various componens of he Ω marix can all be consisenly esimaed wih sandard mehods (eg Newey-Wes covariance marix esimaor) evaluaed a he esimaed parameer. (No surprising.) Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

21 Mone Carlo resuls - 1 Wes also provides a Mone Carlo sudy of his disribuion heory, ailored owards macroeconomic applicaions. y 1 = β 11 + β 12 y 2 + β 13 y β 14 x 1 + u 1 y 2 = β 21 + β 22 y 1 + β 23 y β 24 x 2 + u 2 u (u 1, u 2 ) 0 s iid N (0, I ) x 1, x 2 s indep AR (1), indep of u Le v 1 Y 1 Ŷ 1. Three hypoheses were esed: E [v 1 ] = 0 h i h E v1 2 = = E Corr [v, v 1 ] = 0 v 2 1 i Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

22 Mone Carlo resuls - 1, con d Wes (1995), Table 1, p1077 Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

23 Mone Carlo resuls - 2 The second Mone Carlo was designed o compare Wes s es wih one ha ignores he impac of esimain error (eg he Diebold-Mariano es) y 1 = β 11 + β 12 w 1 + u 1 y 2 = β 21 + β 22 w 2 + u 2 y = w 1 + w 2 + v H 0 : E h i h i u1 2 = E u2 2 The parameers are calibraed o monhly DM-USD exchange rae reurns over Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

24 Mone Carlo resuls - 2, con d Wes (1995), Table 2, p1079 Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

25 Commens on he resuls of Wes (1996) Since is publicaion, some imporan feaures of Wes s resuls have been clari ed: Mos imporan is he fac ha Wes s framework canno be applied o ess of equal MSE when he models are nesed. This is a serious limiaion in some macro applicaions. McCracken 1999 JoE relaxed his assumpion) Second, alhough Wes s resuls can in heory be used o compare many models (no jus 2) he asympoic heory only works well when here are a few models. (Whie 2000 exended he heory o deal wih many compeing forecass) Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

26 The approach of Giacomini and Whie (2006) The key idea in GW is o consider a di eren null hypohesis o Wes Wes H 0 : E f Y +1, Ŷ (X +1, β ), Z +1 = 0 GW H 0 : E f Y +1, Ŷ X +1, ˆβ +1, Z+1 jg = 0 There are wo imporan (relaed) di erences in his change of null hypohesis: 1 The forecass are evaluaed using he esimaed parameer, ˆβ, no he pseudo-rue parameer, β 2 The expecaion in he null hypohesis is condiional raher han uncondiional Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

27 Evaluaion of he forecasing mehod no model While Wes is criical of predicive abiliy ess ha ignore esimaion error from he model, GW sugges ha his is an advanage Tesing a null based on ˆβ allows hem o compare forecasing mehods: he model, he esimaion mehod, he window of daa used for esimaion, ec. all a ec forecas performance in pracice. The Wes es only compares he choice of model. No problems comparing nesed or non-nesed models No resricion on choice of esimaion mehod (parameric, nonparameric, Bayesian) Simple o implemen (hard o under-esimae he value of his feaure) Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

28 Tesing *condiional* predicive abiliy By changing he expecaion in he null from uncondiional o condiional DMW H 0 : E f Y +1, Ŷ (X +1, β ), Z +1 = 0 GW H 0 : E f Y +1, Ŷ X +1, ˆβ +1, Z+1 jg = 0 GW are able o es no only for di erences in average forecas performance, bu also di erences in forecas performance ha are predicable given informaion se G. Eg: Two macro models may perform equally well on average, bu one model may do beer during recessions and he oher during booms. Eg: Volailiy forecasing models performing beer/worse during bull/bear markes, or during volaile/ranquil periods, ec. G may be he rivial informaion se, G = f, Ωg, and so GW provide a heoreical jusi caion for he use of DM ess in he presence of esimaed parameers. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

29 Limied memory forecasing mehods The only non-sandard assumpion in GW is ha he forecass mus be based on a nie amoun of daa, even asympoically Tha is, R (in Wes noaion) or m (in GW noaion) mus remain m < as P (Wes) or n (GW)! This allows for rolling window and xed window esimaion, bu no expanding window esimaion. I also allows for random or daa-driven choices of window lenghs, so long as he maximum window lengh is nie Eg: Pesaran and Timmermann (2006, JoE) sugges a procedure for esimaing he opimal window lengh for forecasing in he presence of occasional srucural breaks. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

30 Choice of funcion for he null hypohesis I The null hypohesis considered by GW is H 0 : E [ L +1 jg ] = 0 where L +1 = L Y +1, Ŷ 1 X +1, ˆβ 1,+1 L Y +1, Ŷ 2 X +1, ˆβ 2,+1 Tha is, hey focus exclusively on ess of equal predicive abiliy. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

31 Choice of funcion for he null hypohesis II A more general null GW H 0 : E f Y +1, Ŷ X +1, ˆβ +1, Z+1 jg = 0 could also be esed, for oher choices of he funcion f. Eg: esing for condiional bias, condiional e ciency, condiional forecas encompassing, ec. These are all properies of forecass ha received a lo of aenion in heir uncondiional form, bu which could well be ineresing in a condiional form. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

32 Implemenaion of he es I GW es a necessary condiion for heir general null hypohesis: H 0 : E [ L +1 jg ] = 0 which implies H 0 0 : E [ h (q1) L +1 ] = 0 (11) (q1) for some (q 1) vecor of variables h 2 G. GW call h he es funcion. The resricion in H0 0 is a sandard uncondiional momen resricion, which can be esed wih a χ 2 es if he series fh L +1 g obeys a CLT. (They provide high-level assumpions for a CLT o hold) Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

33 Implemenaion of he es II In ha case, he es sa is T n = n! T n h L +1 ˆΩ n 1 =m! T 1 1 n h L +1 =m! χ 2 q under H 0 where ˆΩ n! Ω " # T 1 1 and Ω = avar p h L +1 n =m Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

34 Esimaing he covariance marix " # T 1 1 Ω = avar p n h L +1 =m GW discuss di eren ways of esimaing Ω for he various ess. For τ = 1 and G = F he null implies ha fh L +1, G g is a MDS, so can jus use he sample covariance marix. For τ > 1 and G = F he null implies ha fh L +1 g has auocorrelaion only up o lag τ 1, so only need o deal wih auocorrelaion up o here For τ 1 and G = f, Ωg, he null does no resric he auocorrelaions of fh L +1 g and so a HAC esimaor (eg Newey-Wes) is needed For τ 1 and G F (no considered in GW) same as for G = f, Ωg Using a HAC esimaor works in all cases, under sandard assumpions, hough here may be a loss of power and/or nie-sample size disorion. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

35 Mone Carlo sudy of size GW use he following se-up for a sudy of nie sample size: Y = c + CPI + ε, ε s iid N 0, σ 2 Ŷ 1 = ˆβ CPI Ŷ 2 = ˆδ + ˆγ CPI Model 1 is mis-speci ed (missing he inercep) while Model 2 is correcly speci ed. However if c is small enough hen he bias in Ŷ 1 may be mached by increased variance in Ŷ 2 from having o esimae wo parameers raher han one. c is obained analyically for MSE loss in Prop 5, and numerically for Linex loss in Prop 6. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

36 Mone Carlo sudy of size, Table 1 GW (2006), Table 1, p1561 Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

37 Mone Carlo sudy of power GW sudy he power of heir es in a very simple fashion: L +1 = µ (1 ρ) + ρ L + ε +1 ε s iid N (0, 1) The es funcion for he GW es is h = [1, L ] 0 Scenario 1: ρ = 0, varying µ. In his case he here is a possible di erence in uncondiional loss, bu i is no predicable. Scenario 2: µ = 0, varying ρ. In his case he here is no di erence in uncondiional loss, bu he condiional loss is non-zero and parially predicable. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

38 Mone Carlo sudy of power, Figure 1 GW (2006), Figure 1, p1562 Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

39 A decision rule for forecas selecion If G 6= f, Ωg and he GW es rejecs he null of equal predicive accuracy, hen i should be possible o predic fuure relaive forecas performance using h in OLS regression: L +1 = δ 0 h + u so ˆL +1 = ˆδ 0 nh If H0 0 : δ = 0 is rejeced, GW sugges he forecas selecion rule: ˆδ 0 nh > c use forecas 2 ˆδ 0 nh < c use forecas 1 This decision rule is prey simplisic - no clear when his would be opimal. Why rule ou he use of non-ineger combinaion weighs, for example. In heir empirical applicaion, GW show ha his simple forecas selecion rule ou-performed forecass from an individual mehod in 26 ou of 30 pair-wise comparisons. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40

40 Summary Wes (1996) provides he asympoic heory for ess of forecass based on esimaed models Accouns for he esimaion error in he es saisics, hus allowing formal saemens o be made abou he model, no jus he forecass from he model When he forecass have no esimaion error (eg: survey forecass or judgemen forecass), or if R is large relaive o P, he Wes es reduces o he Diebold-Mariano (1995) es The DM es is ofen moivaed by a saemen ha he forecas resuls are condiional on he forecass or aking he forecass as primiive Giacomini and Whie (2006) provide a formal jusi caion for aking he forecass as primiive - evaluaing he forecas mehod no jus he model. Their approach also allows for a change in he null hypohesis, o draw conclusions abou relaive condiional predicive accuracy. Paon (Duke) Forecas Evaluaion Theory OMI-SoFiE Summer School, / 40