A weighed leas squares procedure o approximae leas absolue deviaion esimaion in ime series wih specific reference o infinie variance uni roo problems J. Marin van Zyl Deparmen of Mahemaical Saisics and Acuarial Science, Universiy of he Free Sae, PO Box 339, Bloemfonein, Souh Africa Absrac: A weighed regression procedure is proposed for regression ype problems where he innovaions are heavy-ailed. his mehod approximaes he leas absolue regression mehod in large samples, and he main advanage will be if he sample is large and for problems wih many independen variables. In such problems boosrap mehods mus ofen be uilized o es hypoheses and especially in such a case his procedure has an advanage over leas absolue regression. he procedure will be illusraed on firs-order auoregressive problems, including he random walk. A boosrap procedure is used o es he uni roo hypohesis and good resuls were found. Keywords: Random walk, Auoregressive model, heavy-ailed, sable, weighed regression. Inroducion. An easy o calculae weighed leas squares procedure is proposed which performs almos similar o leas squares (LS) when he error erm is Gaussian whie noise. his esimaion procedure performs very similar o leas absolue deviaion regression (LAD) in large samples wih respec o bias and efficiency for series where he noise is heavy-ailed and from a sable disribuion. In problems wih large samples and many variables and especially where boosrap procedures is used o es hypoheses he compuaional aspecs of LAD regression can become a problem because of he compuaional complexiy, and his mehod will have an advanage. he second advanage over LAD regression is ha even in problems wih small samples his procedure ouperforms LAD regression in many cases wih respec o bias and mean square error of he esimaed parameers.
he procedure will be illusraed and esed on AR() models, wih random walk models included. he principles involved will be similar in large auoregressive ype regression problems where he errors are heavy ailed disribued and from a sable disribuion. Auoregressive models (AR) and he special case of a random walk play an imporan role in finance, for example in he efficien marke hypohesis. In pracice he disribuion of he errors of he AR model is ofen heavy-ailed disribued wih a finie mean and he variance may possibly no be finie, and no whie noise as in he usual random walk or AR models. he case where he error erm has a symmeric sable disribuion wih locaion parameer zero will be considered and specifically AR() models. Hypoheses can be esed by making use of boosrap mehods based on he resuls of Moreno and Romo (202), Davis and Wu (997). he imporan resul of Davis and Wu (997), made i possible o es hypohesis using boosrap samples, where a random normalizing consan for he es saisic in he case of heavyailed daa was given, which can be used in he original sample and in boosrap samples. I is shown in a simulaion sudy ha he uni roo es can be esed wih power comparable and beer han much more compuaional complicaed mehods. Resuls on idenificaion and oher properies of ime series wih heavy-ailed disribued errors were given in he paper of Adler, Feldman and Callagher (998), Feigin and Resnick (999). Subsampling Boosrap, LAD esimaion and hypohesis esing mehods are considered in he papers of Moreno and Romo (2000), (202), Davis and Wu (997), Jach and Kokoska (2004). here is a huge lieraure abou regular variaion in ime series. A few references of ineres are he papers of Davies and Mikosch (998). Li, Liang and Wu (200), Andrews, Calder and Davis (2009), Chan and Zhang (2009), Dielman (2005), Samarakoon and Knigh (2009). 2
Le { y }, =,..., + denoe a ime series and consider he general auoregressive model of order one (AR()): y = 0 + y + u, () I will be assumed ha he funcion φ ( ) where u ' s has a sable disribuion wih characerisic log ( ) α α φ = σ { isign( ) an( πα / 2)} + iµ, α, and log ( ) α α φ = σ { i sign( )log( )} + iµ, α =. he parameers are he index α (0,2], scale parameer σ > 0, coefficien of skewness [,] and mode µ. I will be assumed ha he errors are symmerically disribued wih µ = 0, = 0. he sable disribuion has power ails of he form: Pr( Y > y) y α, y,0 < α < 2, r wih E( y ) finie for r < α. Special cases of he sable disribuions are for α =, i is a Cauchy disribuion and for α = 2 a normal disribuion. Mos of he heory of uni roo models were derived assuming ha he series of 2 errors is whie noise{ u }, hus i.i.d. wih mean zero and variance σ u. In some financial applicaions, he u ' s are called shocks or innovaions. he parameer 0 is ofen called he drif of he series. I can be shown ha he uncondiional mean and variance are E( y ) = 0 /( ) and var( y ) = σ /( ). For a 2 2 u random walk when = and he mean and variance are boh infinie, and he process is no saionary even if he series{ u } is whie noise. his hypohesis H0 : = is called he uni roo es of saionariy and he Dickey-Fuller es was developed o es he uni roo es. Phillips (987) derived 3
resuls on he momen of such a process and showed for 0 = 0, ˆ is super consisen or lim ( ˆ ) has a well-defined limi. If he u ' s are from a sable disribuion wih index α, < α < 2, he usual esimaors can be unsable. Van Zyl and Schall (2009) proposed an easy o calculae weighed esimaion mehod and showed ha for example his mehod ouperforms he opimal rimmed mean proposed by (Rohenberg, Fisher and ilanus (964), Fama and Roll (968)). A wo sage esimaion mehod is proposed, firs plain leas squares esimaion is performed o esimae he residuals, calculae he weighs using he residuals and hen performing weighed leas squares esimaion. he weighs proposed by Van Zyl and Schall (2009) are proporional o exp( uˆ ˆ j um ), uˆ, =,...,, uˆm is he sample median of he residuals. AR() models including uni roo models will be considered. Wih = or <, he noise, { u } Gaussian whie noise ( α = 2.0 ) or heavy-ailed wih < α 2 a symmerical sable disribuion. A random walk where he incremens are from a sable disribuion wihα >, a finie mean and infinie variance is called a Lévy fligh. In Lévy flighs here are fewer large changes, bu when here are changes, he changes are much larger han hose which would be expeced in he case wih normally disribued shocks. Mandelbro called his he Noah Joseph effec (Mandelbro and Wallace (968)). 2. Proposed esimaion procedure Robus esimaion, making use of weighs as in Van Zyl, Schall (2009) will be invesigaed. he weighs are based on he idea of consrucing a weighed mean which will be asympoically equal o he median. In he papers of Koenker (2000), Pornoy (997) for example, he problem where Laplace found LAD esimaors using weighed medians is discussed. A he momen here is no ye a procedure where LAD problems can be solved in he form of a weighed sum. he 4
proposed argumen used is o approximae an inegral which is equal o he median if he sample is from a symmeric disribuion. Consider he Laplace disribuion wih median u m and he inegral u u m ( m) m, I = c ue du = c + u e d = u c a normalizing consan. Numerically he inegral can be approximaed by he sum n j x j xˆ m, for a sample x,..., n j= I c x e x wih esimaed median x ˆm, c a normalizing consan for he weighs. If he differences x ˆ ˆ x,..., x x have an m n m approximae Laplace disribuion i can be seen ha he sum approximae he median and in he case of an exac Laplace disribuion will be equal o he median asympoically in he limi. In pracice i was found ha his weighed sum ouperforms he median in many problems where he daa is heavy-ailed when esimaing he parameer of locaion. he idea is o apply hese weighs o regression and in his work specifically o ime series daa. In figure he median of Cauchy daa was esimaed for various sample sizes using he median of he sample and also using he weighed mean esimaor. I can be seen ha he esimaors are very close and he weighed esimaor ouperforms he median wih respec o mean absolue deviaion in mos of he cases. A each sample size n, 500 samples were drawn and he mean deviaion a each sample size for boh esimaion mehods is ploed in figure. 5
mean absolue deviaion 0.2 0.8 0.6 0.4 0.2 0. 0.08 0.06 0.04 0.02 0 0 000 2000 3000 4000 5000 6000 7000 8000 9000 0000 sample size Figure. Plo of mean absolue deviaion of Cauchy disribued daa using he median and he weighed mean mehod. he solid line denoes he average deviaion for he weighed mean. For an AR(p) model wih consan of he form: y = + y + + y + u, 0... p p he errors{ u }, =,...,, are symmerically sable disribued wih locaion parameer zero. y y y y p+ p+ 2 =, X =... y y y p y p, he usual esimae of he parameers using leas squares is ˆ ( ' ) = X X X ' y, and a series of esimaed residuals can be calculaed as uˆ = y X ˆ. Calculae a series of weighs w = exp( uˆ uˆ ) / exp( uˆ uˆ ), w, ˆ j u, j =,...,, j j m j m j= 6
wih uˆm he sample median of he esimaed residuals. Le W denoe a marix wih diagonal he weighs w,..., w, wj 0, j =,...,, wj =. he weighed esimae of he parameers is calculaed as j= ˆ = ( ' ) ' w X WX X Wy. For he AR() model he equaions are ˆ 0w wj y j wj y j j= 2 j= 2 =. 2 ˆ w w j y j wj y j wj y j y j j= 2 j= 2 j= 2 Experimenaion showed ha a few ieraions improve he esimaion resuls slighly in especially cases where he error erms are heavy-ailed. 3. Simulaion sudy and applicaion 3. Esimaion resuls In he nex simulaion, he variances of he leas squares and he weighed esimaes were compared. AR() series, wih =250 poins each were simulaed n=000 imes. here is a drif wih 0 = 0.. Le ˆ ˆ 0, denoe he usual leas squares esimaes and denoe he weighed esimaes by ˆ ˆ 0 w, w and LAD esimaed parameers by ˆ ˆ 0L, L. he following able shows he esimaed values, 000 series of 250 each. he drif 0 = 0. was used. he scale parameerσ =.0, symmeric sable errors wih locaion parameer zero was simulaed for he error erm of he series. In he case where α = 2.0 ieraions did 7
no improve resuls, oherwise wo ieraions were used in he weighed regression secion. For α =. here were cases wih singular W marices, and correcions can be made, for example as in ridge regression, keeping he consrain ha he weighs add up o one. α 0 ˆ ˆ ˆ 0 w 0 L ˆ ˆ ˆ w L 2 0.40 (0.074) 0.428 (0.232) 0.449 (0.02) -0.052 (0.0005) -0.062 (0.0005) -0.062 (0.0006).9 0.335 (0.359) 0.06 (0.384) 0.89 (0.043) -0.065 (0.5258) -0.039 (0.4274) -0.04 (0.459).5 0.287 (5.3932) 0.0883 (0.2527) 0.0647 (0.040) -0.092 (0.73) -0.0077 (0.439) -0.0074 (0.459).3 0.360 (02.206) -0.048 (.3982) 0.0400 (0.2306) -0.089 (0.7) -0.006 (0.0246) 0.0038 (0.0505). -.4299 (8003.3) -.374 (5684.9) 0.02 (0.6384) -0.0203 (0.004) -0.008 (0.0002) -0.009 (0.0000) 0.975 2 0.0638 (0.0365) 0.0605 (0.0457) 0.0625 (0.0378) -0.080 (0.7546) -0.080 (0.9750) -0.080 (0.789) 0.975.9 0.075 (0.0458) 0.0600 (0.047) 0.0628 (0.0350) -0.087 (0.7746) -0.059 (0.893) -0.066 (0.6879) 0.975.5-0.0674 (3.6425) -0.0003 (0.0206) 0.0086 (0.0233) -0.074 (0.7262) -0.0040 (0.597) -0.0070 (0.89) 0.975.3 0.0565 (5.0762) 0.0068 (0.083) 0.009 (0.0228) -0.064 (0.709) -0.004 (0.05) -0.0039 (0.0769) 0.975..2677 (448.0537) 0.0099 (0.0465) 0.0095 (0.0209) -0.053 (0.6404) -0.0066 (0.06) -0.007 (0.0235) 0.5 2 0.0062 (0.0087) 0.08 (0.043) 0.0070 (0.0098) -0.029 (3.220) -0.045 (5.864) -0.036 (3.6482) 0.5.9-0.0075 (0.043) -0.0033 (0.029) -0.0049 (0.0094) -0.005 (2.9646) -0.0089 (3.9530) -0.0049 (2.6555) 0.5.5 0.0322 (0.257) 0.003 (0.026) 0.0030 (0.00) -0.004 (2.633) -0.0050 (.704) -0.0053 (.0433) 0.5.3 -.026 (2.2063) -0.0059 (0.008) -0.0033 (0.03) -0.0087 (2.2784) -0.005 (0.4960) -0.0027 (0.4364) 0.5. -2.7439 (7455.4) 0.004 (0.000) 0.0057 (0.023) -0.0083 (.950) -0.0005 (0.53) -0.003 (0.650) able. Bias and MSE in brackes of he esimaed parameers of an AR() and sable disribued innovaions wih differen index values, using leas squares and weighed leas squares. Based on 000 simulaed series. he esimaion of using weighed LS compares favorably wih he LAD esimaion procedure. For α close o.00 and =.00, signs of 8
mulicollineariy was seen when using he weighed leas squares. Ridge regression can help in hose cases. A comparison of he esimaion of in he uni roo model y = 0. + y + u, = for differen sample sizes, is shown in able 2. he errors are from a symmeric sable disribuion wih index α =.5, σ =.0. he resuls are based on 000 simulaed series of lengh each. ˆ w Bias( ˆ w ) MSE( ˆ w ) ˆ L Bias( ˆ L ) MSE( ˆ L ) 50 0.9586-0.044 0.0058 0.948-0.0582 0.0076 75 0.977-0.0229 0.0026 0.965-0.0348 0.0030 00 0.985-0.049 0.0009 0.9758-0.0242 0.005 25 0.9889-0.0 0.0005 0.986-0.084 0.0008 50 0.992-0.0088 0.0003 0.9856-0.044 0.0005 75 0.9930-0.0070 0.0002 0.9888-0.02 0.0003 200 0.9948-0.0052 0.0002 0.9906-0.0094 0.0002 225 0.9946-0.0054 0.000 0.99-0.0089 0.0002 250 0.9960-0.0040 0.000 0.993-0.0069 0.000 300 0.9969-0.003 0.000 0.9947-0.0053 0.000 400 0.998-0.009 0.0000 0.9965-0.0035 0.0000 500 0.9986-0.004 0.0000 0.9973-0.0027 0.0000 600 0.9989-0.00 0.0000 0.9979-0.002 0.0000 700 0.9992-0.0008 0.0000 0.9984-0.006 0.0000 750 0.999-0.0009 0.0000 0.9983-0.007 0.0000 able 2. Esimaion resuls based on 000 simulaed series for various sample sizes of weighed LS mehod and LAD regression. he esimaed ( ) = using he wo esimaion mehods are ploed in figure 2 for he differen sample sizes. I can be seen ha he weighed esimae performs beer wih respec o bias and he wo esimaors converges owards each oher wih an increase in he lengh of he series. 9
0.99 Weighed LS and LAD esimaor 0.98 0.97 0.96 0.95 0.94 00 200 300 400 500 600 700 Figure 2. Plo of average esimaed parameer using wo esimaion mehods. Solid line denoes he mean weighed LS esimaor, based on 000 esimaed parameers. In able 3 resuls are given for esimaing for he model y = y + u, series of lengh =50, and 000 series simulaed each ime. A series of lengh =50 can be considered a small sample size in erms of ime series. he weighed regression mehod ouperforms LAD wih respec o bias and MSE for uni roo series or for close o one, and if he error erm is heavy ailed disribued. α ˆ ˆ w L 2-0.046-0.0322 (0.0043) (0.0047).9 -.0099-0.0284 (0.0034) (0.0035).5-0.0024-0.044 (0.00) (0.004).3-0.0025-0.04 0
. 0.975 2 0.975.9 0.975.5 0.975.3 0.975. 0.5 2 0.5.9 0.5.5 0.5.3 0.5. (0.0006) (0.000) -0.0008-0.0072 (0.0002) (0.0006) -0.008-0.038) (0.0053) (0.0049) -0.006-0.0302 (0.0047) (0.0045) -0.0050-0.094 (0.009) (0.0022) -0.006-0.08 (0.0009) (0.00) 0.0006-0.0066 (0.0004) (0.0006) 0.073-0.096 (0.0249) (0.047) 0.094-0.056 (0.0244) (0.047) 0.008-0.040 (0.0) (0.0079) 0.0088-0.0069 (0.0069) (0.0053) 0.008-0.0058 (0.003) (0.0033) able 3. Bias and MSE of model parameer using weighed regression and LAD regression wih a sable disribued error erm wih differen values of he index. 3.2 Uni roo boosrap es In his secion he esing of he uni roo using boosrap mehods as given by Moreno and Romo (202) will be invesigaed. he model ha will be considered y = y + u, u, sable disribued wih index 0 < α 2. he procedure has is he following hree seps: a. Esimae from he original series { x}, =,...,, as =, ˆ 2 x x / x = 2 using ˆ calculae he residuals { } ε and weighs for he weighed esimae =. he weighs are of he form ˆ 2 w w x x / w x = 2 w = exp( ˆ ε ˆ ε ) / exp( ˆ ε ˆ ε ), w, ˆ j ε j, j =,...,, j j m j m j=
ˆm ε he median of he leas squares residuals. Calculae he residuals ˆ ˆw ε = X X w, =,,. he es saisic o es he uni roo hypohesis is max( X,..., X ) ( ˆ ). making use of he resul of m w Davis and Wu (997), where i was shown ha he random normalizing consan max( X,..., X ) can be used in he original sample and in boosrap samples. m b. Random boosrap samples of size m = / log(log( )) is aken 4000 imes, using he empirical disribuion funcion of he { ˆ ε }. Generae series of lengh m * * * under he null hypohesis, X = X + ε, { ˆ* ε }, =,..., m, he boosrap sample. ˆ Calculae ˆ * * * * from his series and he es saisic max( X,..., X ) m( ˆ ). w w m w c. he es saisic obained from he original series is he compared o he perceniles of he es saisics under he null hypohesis. In he following able he resuls of a simulaion o check he power of using he Moreno and Romo (202) boosrap procedure o es he uni roo hypohesis is checked. he number of series simulaed were 000 each ime, and o es he hypohesis 000 boosraps o esimae he criical value for esing he hypohesis: H : =.0 wih alernaive H : <.0. 0 0 n=50 0.8 0.9 0.95.00 α = 2.0 0.732 0.4.240.00 α =.9 0.784 0.486.256 0.084 α =.5 0.966.749.476 0.084 α =..000.96.800 0.087 n=250 0.8 0.9 0.95.00 α = 2.0.000 0.990 0.826 0.080 α =.9.000 0.997 0.874 0.094 2
α =.5.000.000.994 0.077 α =..000.000.00 0.20 able 4. Power esimaion when esing he uni roo hypohesis using boosrap mehods An indicaion of he performance of his esimaor and he bes performing one in in he resuls of Moreno and Romo (202), for series wih =50 observaions, α =.3 is given in able 5. hey compared hypohesis esing using LAD esimaion, ukey s bisquared funcion and oher robus esimaors. For his specific case ukey s bisquared funcion performed bes using he saisic suggesed by Knigh (989), where a normalizing consan of he form / 2 X, and also wih boosrap samples of size m = / log( ) were = 2 sampled. ukeys s Bisquared Funcion Knigh m = / log( ) Weighed Regression 0.8 97.35 94.50 99.20 0.9 82.35 77.05 90.40 0.95 55.45 50.5 63.00.0 6.65 5.35 8.40.0 27.20 23.65 5.50.05 86.90 87.25 90.00 able 5. Comparison of he resuls of Moreno and Romo (202) and using weighed esimaion, boosrap hypohesis esing. 4. Conclusions he weighed esimaion mehods is very simple from he compuaional viewpoin and performs good when comparing wih he LAD procedure which is more complicaed from he compuaional viewpoin. his proposed procedure also yielded good resuls o es hypohesis using boosrap mehods. 3
For series wih error erms which are no heavy-ailed disribued, very lile is los when using his procedure. hus from a pracical viewpoin, he weighed esimaion migh be very useful as an esimaion procedure wihou having o use compuaional inensive mehods. References Adler, R.J., Feldman, E. and Callagher, C. (998) Analysing Sable ime Series, In A Pracical Guide o Heavy ails, Adler, R., Feldman, R., aqqu, M. (eds.), Birkäuser. Andrews, B., Calder, M. and Davis, R.A. (2009) Maximum Likelihood Esimaion for α -Sable Auoregressive Models, Ann. Saisics, Volume 37: 4, 946 982. Davis, R.A. and Mikosch,. (998) he sample auocorrelaion of heavy-ailed processes wih applicaions o ARCH, Annals of Saisics, 26: 5, 2049 2080. Davis, R.A. and Wu, W. (997) Boosrapping M-Esimaes in Regression and Auoregression wih Infinie Variance. Saisica Sinica 7, 35 54. Dielman,.E. (2005) Leas absolue value regression: recen conribuions, Journal of Saisical Compuaion and Simulaion, 75:4, 263-286. Feigin, P.D. and Resnick, S.I. (999) Pifalls of Fiing Auoregressive Models for Heavy-ailed ime Series, Exremes, 4, 39 422. Jach, A. and Kokoszka, P. (2004) Subsampling Uni Roo ess for Heavy-ailed Observaions, Mehodology and Compuing in Applied Probabiliy, 6, 73 97. Koenker, R. (2000) Galon, Edgeworh, Frisch, and prospecs for quanile regression in economerics. Journal of Economerics, 95, 347 374. Li, J., Liang, W. and Wu, X. (200) Empirical likelihood for he smoohed LAD esimaor in infinie variance auoregressive models, Saisics and Probabiliy Leers, 80, 420 430. Mandelbro, B.B. and Wallis, J. (968) Noah, Joseph and operaional hydrology, Waer Resour. Res., 4(5), 909 98. Moreno, M. and Romo, J. (2000) Boosrap es for uni roos based on LAD esimaion. Journal of Saisical Planning and Inference, 83, 347 367. Moreno, M. and Romo, J. (202) Uni Roo Boosrap ess Under Infinie Variance. Journal of ime Series Analysis, 33:, 32 47. Phillips, P.C.B. (987) ime Series Regression wih a Uni Roo, Economerica, 55 (2), 277 30. 4
Pornoy, S., Koenker, R. (997) he Gaussian Hare and he Laplacian oroise: Compuabiliy of Squared-Error versus Absolue-Error Esimaors. Saisical Science, 2:4, 279 296. Samarakoon, D.M. and Knigh, K. (2009) A Noe on Uni Roo ess wih Infinie Variance Noise. Economeric Reviews, 28:4, 34 334. 5