Adaptive Estimation of Heteroscedastic Linear Regression Models Using Heteroscedasticity Consistent Covariance Matrix

Transcription

1 ISSN Journal of Statstcs Volume 16, 009, pp Adaptve Estmaton of Heteroscedastc Lnear Regresson Models Usng Heteroscedastcty Consstent Covarance Matrx Abstract Muhammad Aslam 1 and Gulam Rasool Pasha For the estmaton of lnear regresson models n the presence of heteroscedastcty of unknown form, method of ordnary least squares does not provde the estmates wth the smallest varances. In ths stuaton, adaptve estmators are used, namely, nonparametrc kernel estmator and nearest neghbour regresson estmator. But these estmators rely on substantally restrctve condtons. In order to have accurate nferences n the presence of heteroscedastcty of unknown form, t s a usual practce to use heteroscedastcty consstent covarance matrx (HCCME). Followng the dea behnd the constructon of HCCME, we formulate a new estmator. The Monte Carlo results show the encouragng performance of the proposed estmator n the sense of effcency whle comparng t wth the avalable adaptve estmators especally n small samples that makes t more attractve n practcal stuatons. Keywords Adaptve estmator, Heteroscedastcty, Heteroscedastcty consstent covarance matrx estmator, Nearest neghbour regresson estmator, Nonparametrc kernel estmator 1. Introducton The usual regresson model assumes that the expected value of all error terms, when squared, s the same at any gven pont. Ths assumpton s called homoscedastcty. If ths assumpton s not met, there s an ndcaton of the exstence of heteroscedastcty. The most common framework n whch 1 Department of Statstcs, Bahauddn Zakarya Unversty, Multan, Pakstan Emal: aslamasad@ bzu.edu.pk Department of Statstcs, Bahauddn Zakarya Unversty, Multan, Pakstan Emal: drpasha@bzu.edu.pk

2 Adaptve Estmaton of Heteroscedastc Lnear Regresson Models usng Heteroscedastcty 9 Consstent Covarance Matrx heteroscedastcty s studed n econometrcs s n the context of the lnear regresson models. Errors of measurements, samplng strateges, model msspecfcatons, and presence of outlers etc. are the man causes to ntroduce heteroscedastcty (see Grffths, 1999; Gujarat, 003 for more detals). In the presence of heteroscedastcty, ordnary least squares (OLS) estmators are although unbased and consstent but no longer effcent. In addton, the standard errors of the estmates become based and nconsstent. Ths, n turn, leads to ncorrect nferences. Dependng on the nature of the heteroscedastcty, sgnfcance tests can be too hgh or too low. These effects are not gnorable as earler noted by Geary (1966), Whte (1980) and Pasha (198), among many others. When the form of heteroscedastcty s known, usng weghts to correct for heteroscedastcty s very smple by usng method of generalzed least squares. If the form of heteroscedastcty s not known, the varance of each resdual can be estmated frst and these estmates can be used as weghts n a second step and the resultant estmates are called the estmated weghted least squares (EWLS) estmates (see for example, Fuller and Rao, 1978; Carroll and Ruppert, 1988; Pasha and Ord, 1994; Greene, 000). But n usual practce, the form of heteroscedastcty s seldom known that makes the weghtng approach mpractcal. In such stuatons, we are requred to formulate such estmators whch gve as adequate results as f we have known heteroscedastc errors and hence adaptve estmaton procedures come n to fulfll such objectves. Specfcally, n the sense of Bckel (198), for a lnear regresson model n the presence of heteroscedastcty, an estmator s sad to be adaptve estmator f t has the same asymptotc dstrbuton as that of an estmator havng nformaton about the form of heteroscedastcty. In the avalable lterature, two adaptve estmators are popular, namely, nonparametrc kernel estmator proposed by Carroll (198) and nearest neghbour regresson (NNR) gven by Robnson (1987). Both of these adaptve estmators requre substantally restrctve condtons for ther applcatons. Whte (1980) ntroduces a heteroscedastcty consstent covarance estmator (HCCME) to draw correct nferences n the presence of heteroscedastcty of unknown form, popularly known as HC0. Then n order to mprove small sample propertes, MacKnnon and Whte (1985) present other versons of HCCME,

3 30 Asad and Pasha known as HC1, HC, and HC3. These estmators are frequently used by the practtoners (e.g., see Long and Ervn, 000; Flachare, 00). Snce all avalable lterature about HCCME lmts up to the consstent estmaton of Var-Cov matrx of the estmated regresson coeffcents and no progress yet has been made to construct weghts based on ths HCCM, ths leads us to formulate weghts based on HCCME and then to conduct EWLS. We present the performance of our formulaton by means of Monte Carlo study. Our study unfolds n such a way that secton presents the heteroscedastc lnear regresson model along wth HCCME. In secton 3, we formulate a new estmator based on HCCME. In secton 4, we present emprcal results based on Monte Carlo smulatons. Secton 5 s reserved for applcaton of our approach and, fnally, secton 6 gves concluson.. Heteroscedastc Lnear Regresson Model and HCCME Consder a heteroscedastc lnear regresson model of the form y = X + u, (.1) where y s an n x 1 vector of observatons, X s an n x p (p < n) nonrandom matrx of covarates, s a p x 1 vector of unknown regresson parameters, and u s an n x 1 vector of random errors wth zero mean and varance. When the errors are homoscedastc,.e.,, under the other standard assumptons, the coeffcents can consstently be estmated by the OLS as follows: ˆ OLS ( X X ) X y wth the covarance matrx, denoted by, ( X X ) X X ( X X ) (.) The estmaton of depends on the assumptons about dag{,,, n}. In the presence of heteroscedastcty of unknown form, the OLS estmate becomes neffcent and ts covarance matrx estmate nconsstent as dscussed earler. In ths stuaton, Whte (1980) derves an asymptotcally justfed form of the

4 Adaptve Estmaton of Heteroscedastc Lnear Regresson Models usng Heteroscedastcty 31 Consstent Covarance Matrx HCCME known as HC0 by lettng ˆ = dag[ u ˆ ], whch results n estmaton of 1 1 (.) as ˆ ( ) ˆ 1 X X X X ( X X ) ( X X ) X dag [ uˆ ] X ( X X ). As shown by Whte (1980) and others, HC0 s a consstent estmator of under both homoscedastcty and heteroscedastcty of an unknown form. However, t can be consderably based n the fnte samples; see, e.g., MacKnnon and Whte (1985); Crbar-Neto and Zarkos (1999, 001). MacKnnon and Whte (1985) rase ther concerns about the performance of HC0 n small samples and present three alternatve estmators to mprove the small sample propertes of HC0. The smplest adjustment, suggested by Hnkley (1977), makes a degree of freedom correcton that nflates each resdual by the factor n /( n k). Wth ths correcton, Macknnon and Whte (1985) obtan the verson of HCCME known as HC1: HC n n 1 ( X X ) X dag[ ˆ ] X ( X X ) HC0 n k u n k Based on work by Horn et al. (1975), MacKnnon and Whte (1985) propose: uˆ HC ( X X ) X dag[ ] X ( X X ) 1 h where h = x ( X X ) x., Followng Efron s (1979) jackknfe estmator, MacKnnon and Whte (1985) gve a thrd varant of HCCME as: uˆ HC3 ( X X ) X dag[ ] X ( X X ). (1 h) Long and Ervn (000) explore the small sample propertes of test usng these four versons of HCCME n lnear regresson model. Ther Monte Carlo smulaton shows that HC0 often results n ncorrect nferences when n 50, whle the other three versons of HCCME, especally HC3, works well even when n s as small as 5.

5 3 Asad and Pasha 3. HCCME-Based Adaptve Estmaton Followng Horn et al. (1975), MacKnnon and Whte (1985) conclude that Var (u ˆ ) underestmates (snce 1/n h 1). By showng, to be a based ˆ estmator of u, they suggest to be less based. Furthermore, Wu (1986) 1 h also estmates by ˆ ˆ / (1 ) u h, where h = ( X X ) x, and ˆ. x uˆ y x OLS ˆ u Followng these deas, we estmate and weghts ˆ ˆ u, and w ˆ. 1 h The estmaton of s the same as used n HC. on the bass of u ˆ as Then we propose the followng HCCME-Based weghted least square (HWLS) adaptve estmator as ˆ HWLS X X w X y w (3.1) Followng the formulaton of HC3, we also present another verson of above adaptve estmator by settng uˆ ˆ (1 h) For convenence, we name both the estmators as HCWLS-estmator (based on HC) and HC3WLS-estmator (based on HC3), respectvely.

6 Adaptve Estmaton of Heteroscedastc Lnear Regresson Models usng Heteroscedastcty 33 Consstent Covarance Matrx 4. Emprcal Results To evaluate the performance of the proposed estmators, we calculate bas, mean square error (MSE) and relatve effcency (R.E). For ths purpose, we conduct a Monte Carlo study and use the same scheme as carred out by Carroll (198) so that our proposed estmator can drectly be compared wth already exstng estmators. The heteroscedastc lnear regresson model, under study, s: y = x +, = 1,,, n, (4.1) where, 0 = 50, 1 = 60, are..d. standard normal varables and x are..d. unform on the nterval (-0.5, 0.5). Furthermore, t s assumed that s are observed ndependent to x. Three dfferent varance models (VMs) are consdered to generate. The frst varance model (VM-I), gven by Jobson and Fuller (1980), s a1 a, x 0 1 We choose a 1 = 100 and a = 0.5. The second varance model (VM-II), a model wth more severe heteroscedastcty, used by Box and Hll (1974), s a 1 exp( a ), where, a 1 = 0.5 and a = The thrd model (VM-III), a model of severe heteroscedastcty, used by Bckel (1978), s a1exp( a ), where, a 1 = 1/4 and a = 1/300. To evaluate the performance of the proposed adaptve estmators wth the change n sample sze, we use dfferent sample szes by takng n = 10, 5, 50, 100, and 50. For each varance model and for each sample sze, we run 1,000 smulatons as done by Carroll (198).

7 34 Asad and Pasha To estmate the model (4.1), we use the followng estmators to hghlght ther comparatve performances under dfferent data generatng schemes: 1. Ordnary least square (OLS) estmator. Adaptve kernel estmators: Followng Carroll (198), we use adaptve kernel estmator. Snce t s well known n the nonparametrc lterature that the choce of the kernel s not crucal as long as t satsfes certan regularty condtons (Roy, 1999), so unlke Carroll (198), we use normal kernel. We refer ths estmator by KWLS estmator for the next dscusson. 3. Nearest neghbor regresson estmator: Followng Robnson (1987), we use nearest neghbor regresson estmator. We choose the number of nearly neghborhood, K = n 4/5 (followng Härdle, 1994). We refer ths estmator by NWLS estmator. 4. HCCME-based adaptve estmator: We use our proposed adaptve estmator (3.1). We use both, above dscussed, versons of ths estmator and refer them as HCWLS and HC3WLS estmators, respectvely. For each VM and for each sample sze, we calculate bases and mean square errors (MSEs) of the estmated coeffcents obtaned by 1,000 smulatons n each case. We compute relatve effcency to compare the results as follows: R.E = MSE (OLS)/MSE (Estmator under consderaton) We carry out all the computatonal work by usng the software, E-Vews 3.0 for the sad smulatons. Table 1 shows the emprcal results of the estmators about bas and MSE and R.E when the sample sze, n = 10. These results show that wth small sample and mld heteroscedastcty (VM-I), OLS performance s not so bad n estmatng both of the coeffcents. When estmatng 0, OLS and H3WLS are almost equally effcent showng relatvely mnmum MSEs whle comparng wth rest of the estmators. Then KWLS and HWLS show equvalent effcent behavour whle NWLS remans the worst so far. On the other hand, when estmatng 1, NWLS agan remans least effcent whle the gan n effcency of our proposed HC3WLS becomes about 11% as compared to that of OLS and 10% as compared to that of the other nonparametrc estmator KWLS.

8 Adaptve Estmaton of Heteroscedastc Lnear Regresson Models usng Heteroscedastcty 35 Consstent Covarance Matrx Although there s no queston of bas n OLS under heteroscedastcty, but n order to compare ths end wth respect to our proposed estmators, we present the comparsons. In case of a small sample of sze 10, HCWLS remans a lttle bt more based whle computng the ntercept and smlar s the case for KWLS whle estmatng 1, as compared to all other estmators. Under VM-II, OLS performs qute badly as expected. Our proposed estmators outperform the two nonparametrc estmators, NWLS and KWLS. The proposed estmators become 3 to 6 tmes more effcent as compared to the OLS. Here, we also see that the performances of NWLS and KWLS are also smlar to our estmators to some extent. We note that HC3WLS performs better than HCWLS whle NWLS performs better than KWLS. As far as basedness s concerned, OLS gves least dfferences. Under VM-III, the bad performance of OLS contnues but the other estmators do not reman as much effcent as they are under VM-II. In ths case (under VM-III), HC3WLS agan holds the leadng role whle all the other estmators perform n qute smlar manner n the terms of relatve effcency. Table (for n = 5) shows almost the same pcture as Table 1 does under VM-I. However, under VM-II, the bad performance of OLS declnes wth ncrease n the sample sze from 10 to 5 relatve to the fgures gven n Table 1. The performance of our proposed estmators s qute encouragng for the varance models, VM-II and VM-III. It s also noted that HC3WLS and HCWLS reman equally effcent under VM-II and VM-III. Through Table 3, we confrm the Carroll s (198) fndngs n terms of relatve effcency of the estmators. Carroll (198) reports the gan n effcency usng the KWLS to be 19%, 188% and 15%, respectvely, for VM-I, VM-II and VM-III whle estmatng 1. We fnd almost the same fgures for gan n effcency usng the KWLS and they are 15%, 177% and 108% for VM-I, VM-II and VM-III, respectvely. We also fnd smlar results whle estmatng 0. We report the gan n effcency to be 30% for HCWLS and 3% for HC3WLS whle comparng wth OLS and ths gan s almost double than the gan for the other two adaptve estmators whle estmatng 1. Our both proposed estmators show excellent performance for severe heteroscedastcty (VM-II) where the gan reaches to about 61% (see HC3WLS for 1 ). The stuaton for VM-III, s also qute encouragng. Moreover, the bas reduces expectedly, wth the ncrease n sample sze for all the cases.

9 36 Asad and Pasha Smlar fndngs are obtaned for n = 100 and 50 (Tables 4 and 5). ). Our proposed estmators perform adequately well n all the cases. For larger samples, the nonparametrc estmators and our proposed estmators become qute dentcal n performance. Table 1: Emprcal Results of the Estmators about MSE and Bas (n = 10) Estmators ˆ (True value = 50) 0 ˆ (True value = 60) 1 Bas MSE* R.E Bas MSE* R.E VM-I OLS KWLS NWLS HCWLS HC3WLS VM-II OLS KWLS NWLS HCWLS HC3WLS VM-III OLS KWLS NWLS HCWLS HC3WLS *The actual MSEs for VM-II are the fgures presented n the Table dvded by 10 whle by 10 3 for VM-III

10 Adaptve Estmaton of Heteroscedastc Lnear Regresson Models usng Heteroscedastcty 37 Consstent Covarance Matrx Table : Emprcal Results of the Estmators about MSE and Bas (n = 5) Estmators ˆ (True value = 50) 0 ˆ (True value = 60) 1 Bas MSE* R.E Bas MSE* R.E VM-I OLS KWLS NWLS HCWLS HC3WLS VM-II OLS KWLS NWLS HCWLS HC3WLS VM-III OLS KWLS NWLS HCWLS HC3WLS * The actual MSEs for VM-II are the fgures presented n the Table dvded by 10 whle by 10 3 for VM-III.

11 38 Asad and Pasha Table 3: Emprcal Results of the Estmators about MSE and Bas (n = 50) Estmators ˆ (True value = 50) 0 ˆ (True value = 60) 1 Bas MSE* R.E Bas MSE* R.E VM-I OLS KWLS NWLS HCWLS HC3WLS VM-II OLS KWLS NWLS HCWLS HC3WLS VM-III OLS KWLS NWLS HCWLS HC3WLS * The actual MSEs for VM-I are the fgures presented n the Table dvded by 10, by 10 3 for VM-II whle by 10 4 for VM-III

12 Adaptve Estmaton of Heteroscedastc Lnear Regresson Models usng Heteroscedastcty 39 Consstent Covarance Matrx Table 4: Emprcal Results of the Estmators about MSE and Bas (n = 100) Estmators ˆ (True value = 50) 0 ˆ (True value = 60) 1 Bas MSE* R.E Bas MSE* R.E VM-I OLS KWLS NWLS HCWLS HC3WLS VM-II OLS KWLS NWLS HCWLS HC3WLS VM-III OLS KWLS NWLS HCWLS HC3WLS * The actual MSEs for VM-I are the fgures presented n the Table dvded by 10, by 10 3 for VM-II whle by 10 4 for VM-III

13 40 Asad and Pasha Table 5: Emprcal Results of the Estmators about MSE and Bas (n = 50) Estmators ˆ (True value = 50) 0 ˆ (True value = 60) 1 Bas MSE* R.E Bas MSE* R.E VM-I OLS KWLS NWLS HCWLS HC3WLS VM-II OLS KWLS NWLS HCWLS HC3WLS VM-III OLS KWLS NWLS HCWLS HC3WLS * The actual MSEs for VM-I are the fgures presented n the Table dvded by 10, by 10 3 for VM-II whle by 10 4 for VM-III. 5. Applcatons To llustrate the performance of our proposed adaptve estmators and to compare them wth the estmators already avalable n the lterature, we take the example of compensaton per employee ($) n Nondurable Manufacturng Industres of US Department of Commerce as quoted by Gujarat (003, pp. 39). We take ths example just to compare our fndngs for practcal data wth the fndngs already avalable n the lterature. In Table 6, we report the performance of all the estmators dscussed above. Frst of all, we apply OLS to the data and fnd t to be heteroscedastc by Whte s Test of Heteroscedastcty (1980) wth p-value Then we report the estmates,

14 Adaptve Estmaton of Heteroscedastc Lnear Regresson Models usng Heteroscedastcty 41 Consstent Covarance Matrx ther standard errors and respectve values of t-statstc. We note that our proposed estmators bear lower standard errors among all the remanng estmators presentng an adequate relablty for ther adaptaton. We also compute HCCME for gvng the correct standard errors for the estmates n the presence of heteroscedastcty. We note that our estmators gve better R and much mproved standard errors of regresson confrmng the adequacy of the ftted model and ths fndng can be vewed n Fg. 1 about ftted values and resduals. Smlarly, the proposed adaptve estmators gve lowest Akake Informaton Crtera (AIC) values that ndcate the rght specfcaton of the weghtng mechansm by usng our proposed estmators. Table 6: Comparatve Statstcs Estmators ˆ 0 Estmaton of 0 Estmaton of 1 SE t-statstc ˆ SE t-statstc R S.E. of Regresson AIC 1 OLS HCCME EWLS KWLS NWLS HCWLS HC3WLS

15 4 Asad and Pasha OLS KWLS NWLS Resdual Actual Ftted Resdual Actual Ftted Resdual Actual Ftted HCWLS 5000 HC3WLS Resdual Actual Ftted Resdual Actual Ftted 6. Concluson Fg. 1: Ftted Values and Resduals In most of the practcal stuatons, we seldom know anythng about the form of heteroscedastcty so t becomes a queston to have correct analyss of regresson problems n the presence of heteroscedastcty. The avalable lterature on ths ssue leads us to use adaptve estmators that gve desrable results that are not possble when just OLS estmaton s taken nto account. Usually, nonparametrc approaches are used for adaptve estmaton. In these avalable approaches, nearest neghbour regresson estmators gve better results as compared to adaptve kernel estmators, n the presence of heteroscedastcty of unknown form. But we propose new estmators that are based on the dea as used n the formulaton of HCCME. We report that our proposed HCCM-based adaptve estmators outperform n all the stuatons, from small to large samples and from mld to severe heteroscedastcty. These estmators become 3 to 6 tmes more effcent as compared to OLS and about one and half tmes more effcent as compared to both already avalable adaptve estmators. Specfcally, our formulaton becomes more attractve n small samples.

16 Adaptve Estmaton of Heteroscedastc Lnear Regresson Models usng Heteroscedastcty 43 Consstent Covarance Matrx References 1. Bckel, P. J. (1978). Usng resduals robustly I: Tests for heteroscedastcty, nonlnearty. Annals of Statstcs, 6, Bckel, P. J. (198). On adaptve estmaton. Annals of Statstcs, 10(3), Box, G. E. P. and Hll, W. J. (1974). Correctng nhomogenety of varance wth power transformaton weghtng. Technometrcs, 16(3), Carroll, R. J. (198). Adaptng for heteroscedastcty n lnear models. Annals of Statstcs, 10(4), Carroll, R. J. and Ruppert, D. (1988). Transformaton and Weghtng n Regresson. Champon and Hall, London. 6. Crbar-Neto, F. and Zarkos, S. G. (1999). Bootstrap methods for heteroskedastac regresson models: Evdence on estmaton and testng. Econometrc Revew, 18(), Crbar Neto, F. and Zarkos, S.G. (001). Heteroskedastcty-consstent covarance matrx estmaton: Whte s Estmator and the Bootstrap. Journal of Statstcal Computaton and Smulaton, 68(4), Efron, B. (1979). Bootstrap methods: Another look at the Jackknfe. Annals of Statstcs, 7(1), Flachare, E. (00). Bootstrappng Heteroskedastcty Consstent Covarance Matrx Estmator. Computatonal Statstcs, 17, Fuller, W. A. and Rao, J. N. K. (1978). Estmaton for a lnear regresson model wth unknown dagonal covarance matrx. Annals of Statstcs, 6(5), Geary, R. C. (1966). A note on resdual heretovarance and estmaton effcency n regresson. Amercan Statstcan, 0, Greene, W. H. (000). Econometrc Analyss. 4 th Ed. Prentce Hall, Upper Saddle Rver, NJ. 13. Grffths, W. E. (1999). Heteroskedastcty. Workng Papers n Econometrcs and Appled Statstcs. ISSN Gujarat, D. N. (003). Basc Econometrcs. 4 th Ed., McGraw-Hll, New York. 15. Härdle, W. (1994). Appled nonparametrc regresson. An unpublshed manuscrpt. Insttut für Statstk und Ökonometre, D-10178, Berln. 16. Hnkley, D. V. (1977). Jackknfng n unbased stuatons. Technometrcs, 19(3), 85-9.

17 44 Asad and Pasha 17. Horn, S. D., Horn, R. A. and Duncan D. B. (1975). Estmatng heteroscedastc varances n lnear model. Journal of the Amercan Statstcal Assocaton, 70, Jobson, J. D. and Fuller, W. A. (1980). Least square estmaton when the covarance matrx and parameter vector are functonally related. Journal of the Amercan Statstcal Assocaton, 75, Long, S. J. and Ervn, L. H. (000). Usng heteroscedastcty consstent standard errors n the lnear regresson model. The Amercan Statstcan, 54, MacKnnon, J. G. and Whte, H. L. (1985). Some heteroskedastcty consstent covarance matrx estmators wth mproved fnte sample propertes. Journal of Econometrcs, 1, Pasha, G.R. (198). Estmaton Methods for Regresson Models wth Unequal Error Varances. Unversty of Warwck, Ph.D. thess.. Pasha, G. R. and Ord, J. K. (1994). Adaptve estmators for heteroscedastc lnear regresson models. Pakstan Journal of Statstcs. 10(1), Robnson, P.M. (1987). Asymptotcally effcent estmaton n the presence of heteroscedastcty of unknown form. Econometrcs, 55, Roy, N. (1999). Is adaptve estmaton useful for panel models wth heteroscedastcty n the unt-specfc error component? Some Monte Carlo Evdence. Econometrc Workng Paper EWP9913, ISSN Whte, H. (1980). A Heteroskedastcty-consstent covarance matrx estmator and a drect test for heteroskedastcty. Econometrca, 48(4), Wu, C. F. J. (1986). Jackknfe bootstrap and other resamplng methods n regresson analyss. Annals of Statstcs, 14,

18 Adaptve Estmaton of Heteroscedastc Lnear Regresson Models usng Heteroscedastcty 45 Consstent Covarance Matrx