I) ile: Forecas Evaluaion II) Conens: Evaluaing forecass, properies of opimal forecass, esing properies of opimal forecass, saisical comparison of forecas accuracy III) Documenaion: - Diebold, Francis X. (998). Elemens of Forecasing, Chaper, SW College Publishing. - Hendry, David F., and Clemens Michael P. (00). Economic Forecasing: Some Lessons From Recen Research. European Cenral Bank Working Paper Series, No. 8. - Ruoss, Eveline, and Savioz, Marcel. (00). How accurae are GDP forecass? An empirical sudy for Swizerland. Quarerly Bullein, 3. - Sekler, Herman O. (00). he Raionaliy and Efficiency of Individuals Forecass. In A Companion o economic forecasing. Clemens, M.C., and Hendry, D.F. Oxford: Blackwell. Forecasing opimally he condiional expecaion is he opimal esimaor for any random variable since i minimizes he average loss funcion of he predicion. If we use he squared loss funcion, i.e. L( e) = e where e denoes he forecas error, we speak of he mean squared error (MSE). he condiional expecaion is herefore he minimum mean squared esimaor (MMSE). o see his, consider he example where we wish o forecas he value of he random variable y + based on he vecor of previous observaions X = ( y K y ). We are looking for a funcion such ha y = g ( X + ) (0.) where y + denoes he forecas of y + based on informaion available up o and including ime. Consider he MSE of he suggesed funcion g ( X ) + = + + ( X) E y y E y g = E y E y + E y g( ) = E y E y E E y g + + E ( y+ E y+ )( E y+ g( ) ) X X X = E y E y + E E y g( ) + + X + X X + + X + X ( X) + + X + X X {( )( ( ) ) } + E E y + E y + X E y + X g X X = E y E y E E y g( ) + + X + + X X + E ( E y + X g( X )) E ( y + E y + X ) X = E y+ E y + + E E y+ g( ) X X X { } (0.)
where he las erm drops ou from he second las o he las line since we have {( + + ) } E y E y = 0 X X. Hence, if ( ) g X is o minimize he MSE, we need ha ( ) g X = E y X + (0.3) which means ha he condiional expecaion is he MMSE. Evaluaing forecass he general concep of a measure for he goodness of forecas, i.e. he accuracy of he forecas, is he loss funcion. Le e+ = y+ h E y+ hy = y+ h y+ be he forecas error, hus we can wrie he loss funcion L( y+ E y+ hy ) = L( y+ y+ ) = L( e+ ) in funcion of he forecas error. Rankings of forecass may vary considerably, depending on he explici form chosen for he loss funcion. he mos commonly used accuracy measures for forecass are he following: he mean error measures he bias of he forecas errors: ME = e +, h = + = EV = e ME he error variance measures he dispersion of he forecas errors: ( ) he mean squared error provides an overall accuracy measure for a forecas: MSE = e+ = Noe ha he measuremen unis of he MSE are he square (e.g. dollars ) of he measuremen unis of he forecas error (e.g. dollars). I is herefore ofen convenien o ake he roo mean squared error in order o preserve he original measuremen unis: e+ = RMSE = An alernaive bu less popular overall measure of accuracy is he mean absolue error: MAE = e+ = he ask of comparing he accuracy of differen forecas mehods for varying ime periods is more delicae since he mean and he volailiy of y may be subjec o grea changes over ime. heil s U provides a reasonable soluion o his kind of problem: e, ( Δy ˆ ) Δy RMSE = RMSE = U = = or U RMS Δ = = RMS y Δy = =
heil s U is zero if he forecas is perfec. If U is smaller (bigger) han one, i means ha he forecasing model is performing beer (worse) han he simple forecas no change. Noe ha hose are only saisical measures which may poorly assess he economic loss of a bad forecas. When forecass are used o guide decisions he loss is normally an asymmeric funcion of he error and a simple MSE will no capure his asymmery. Properies of opimal forecass.) Opimal forecass are unbiased, i.e. E e + = 0..) he e +, errors of an opimal forecas are whie noise. 3.) he e + (h>) errors of an opimal forecas are a mos MA( h ). 4.) he forecas error variance of an opimal forecas is non-decreasing in h. esing properies of opimal forecass.) esing e + for bias I is sraighforward o use he sandard -sa in order o es he null ha E e + = 0. his can easily be done by running he OLS regression () e+ = α + ε If he esimaed coefficien ˆα is no significanly differen from zero, we can conclude ha he forecas is unbiased. he esimae no being differen from zero is a sufficien condiion for no bias. Noe however ha for h> we have e+ = y+ h E y+ hy = ε+ h+ θε + h +... + θh ε+ and e + herefore follows (a mos) an MA( h ) process. his implies ha he OLS error erms of () migh show auocorrelaion up o lag h-. his phenomenon is due o he forecas period overlap associaed wih muli-sep-ahead forecass. When regressing () we should herefore allow he error erms o follow an MA( h ) process and use AIC or SIC o find he correc q for he MA( q ) process..) esing e +, for whie noise he sandard ess for whie noise may be used: Durbin-Wason es, Box-Pierce and Ljung- Box saisics or he sample auocorrelaion and parial auocorrelaion funcions ogeher wih he Barle bands. 3
Auocorrelaions of an MA (3) process: he auocorrelaion cus off afer lag 3. 3.) esing e + (h>) for MA( h ) We can es if he auocorrelaions for lags longer han (h-) are significanly differen from zero by using he Barle sandard errors. One could also run a regression similar o (): e+ = α + ε where ε MA( q) q > h, we expec o find for an opimal forecas ha he MA( q ) If we choose ( ) coefficiens for lags longer han ( h ) are no significanly differen from zero. 4.) esing for a non-decreasing forecas error variance h Recall ha σ h = σ + θi and by using his formula i is sraighforward o observe i= wheher σ is increasing in h or no. h Many of hose properies of opimal forecass can be summarized by he unforecasabiliy principle: Opimal forecas errors should be unforecasable given all he informaion ha was available up o he ime when he forecas was made. Conrary o he above menioned ess, which only make use of univariae properies of he forecas errors, he unforecasabiliy principle relaes o heir mulivariae properies as well. In order o fully es for he unforecasabiliy principle one would run he regression () k e = α + α x + u + 0 i i i= where he x i s conain all informaion available a ime. A necessary condiion for opimaliy is ha α0 = α i = 0. One paricular case of () is he Mincer-Zarnowiz regression: y+ h= β0 + β y+ + u which can be obained from () as follows: e = α + α y + u + 0 + y y = α + α y + u + h + 0 + σ h 4
( α ) y = α + + y + u + h 0 + y = β + β y + u + h 0 + he necessary condiion for an opimal forecas is now β 0 = 0 and β =. Noe ha hose condiions are only necessary bu no sufficien for opimaliy! Raionaliy in forecass his discussion will follow he conceps oulined in Sekler (00). he wo main conceps used in forecasing lieraure are weak and srong raionaliy. Weak raionaliy means ha he forecass are condiionally unbiased, i.e. here are no sysemaic errors. Srong raionaliy, which is also called efficiency, implies ha he forecass are neiher biased nor correlaed wih any oher informaion known a he ime he forecas is made, i.e. he forecaser has efficienly used all available daa. Since i s impossible o es wheher a forecaser has used all available informaion, one normally ess for efficiency by using only he forecaser s pas forecass and forecas errors: Efficiency requires ha one-sep-ahead forecas errors are neiher serially correlaed, nor correlaed wih pas forecas values or errors. A forecas is hen said o exhibi weak form informaional efficiency. If in addiion forecass are unbiased, hey may be called weakly raional. One could hink of a Mincer-Zarnoviz regression in order o es for unbiasedness bu since his is only a necessary no a sufficien condiion, one should raher use (). We can also use he regression approach o es for weak and srong efficiency. Consider he regression (3) y+ h= β0 + β y+ + βx+ u where he vecor x conains all informaion available o he forecaser a ime. esing srong efficiency means esing he null ha β = 0. If only pas values of he relevan variable are included in x, esing β = 0 means esing for weak form efficiency. Noe ha in he case of h = here should be no serial correlaion in he error erm of (3). 5
Saisical comparison of forecas accuracy Diebold and Mariano propose a es for comparing he accuracy of forecass. I is designed o deermine wheher wo forecass have he same accuracy. he null hypohesis equal a b accuracy hypohesis is H0 : E L( e ) + = E L( e+ ) for wo differen forecass a and b. I can be shown ha ( d μ )~ N(0, f) a b and f is he variance of he sample mean loss = differenial and μ he populaion mean loss differenial. M Seing μ = 0 and using he esimaor ˆ f = ˆ γ d ( τ ) wih ˆ γ d ( τ) = ( d μ)( d τ μ) τ = M = /3 and M = leads o a saisic similar o a sandard -saisic: d B= ~ N(0,) fˆ / where d = L( e+ ) L( e+ ) he difference consiss only in he use of f, which represens he fac ha he loss differenial series in no necessarily whie noise. A parameric way of esing he equal accuracy hypohesis is o regress d = α + ε wih ε ~ ARMA( p, q). 6