Parameter Estimation Method in Ridge Regression

Transcription

1 Paamete Estmaton Method n dge egesson Dougade.V. Det. of tatstcs, hvaj Unvesty Kolhau nda. adougade@edff.com Kashd D.N. Det. of tatstcs, hvaj Unvesty Kolhau nda. dnkashd_n@yahoo.com bstact n ths atcle, we oose a new method fo aamete estmaton n a lnea egesson model when multcollneaty s esent. The oetes of oosed estmato ae dscussed. We comae the efomance of oosed estmato wth genealzed dge egesson, Jackknfed dge egesson and Modfed Jackknfed dge egesson estmato. lso, we demonstate the efomance of the oosed estmato n vaable selecton though numecal examles. Keywods: dge egesson, dge estmato, Multcollneaty, Jackknfed dge estmato.. ntoducton t s well known that efomance of least squaes estmato s unsatsfactoy n the esence of multcollneaty. To ccumvent such oblem, seveal methods ae avalable n the lteatue; one of them s dge egesson and whch s fst ntoduced by Hoel and Kennad 970. Vnod and Ullah 98 showed that a dge estmato havng smalle ME than OL estmato. dge estmato s a based estmato. To educe the bas of the genealzed dge egesson G estmato, sngh et al. 986 oosed Jackknfed dge egesson J estmato usng the Jackknfe ocedue fst ntoduced by Hnkley 977. ecently, Batah et al. 008 suggested a new dge estmato namely Modfed Jackknfed dge egesson MJ estmato by combnng dea of G and J, they establshed the ME sueoty ove both the G and J estmatos. The estmatos OL, G, J and MJ ae dscussed n the next secton. n ths ae, we ntoduce a new based estmato and t s genealzaton of the MJ

2 estmato. We comae the efomance of the oosed estmato wth G, J and MJ estmatos usng ME cteon. lso, we have shown that how the oosed estmato can be used to select subset of vaables n the context of lnea egesson when seveal egessos ae hghly coelated to each othe. The egesson model and vaous tyes of estmatos ae descbed n ecton. new estmato s ntoduced n ecton 3. ome theoetcal esults ae eoted n ecton 4. ecton, evews basng constant detemnaton methods and we demonstate the efomance of the oosed estmato though a smulaton study usng ME cteon. n ecton 6, two examles ae gven to demonstate the efomance of the oosed estmato n vaable selecton. The ae ends wth some conclusons dawn fom the esent eseach.. The Model and Vaous Estmatos Consde a multle lnea egesson model Y = Xβ ε whee Y s a n vecto of obsevatons on a esonse vaable Y, β s a vecto of unknown egesson coeffcents, X s a matx of ode n of obsevatons on edcto egesso vaables X, X X P and ε s a vecto of eos wth E ε = 0 and V ε = σ n. We assume that Two o moe vaables n X ae coelated. Theefoe the model suffes fom the oblem of multcollneaty. B The edcto vaables X, X X P ae standadzed vaables, such that a non-sngula coelaton matx. The most common estmato fo β s the least squaes estmato ˆ β = X X X Y. X X s

3 We ewte the model n canoncal fom Y = Zαε whee = dag,,, beng the th egen value of X X and T s the othogonal matx consstng of egen vectos of X X. Then Z = XT andα = T β. Note that Z Z = T X XT =, see Montgomey et al., 006. The OL estmato of α s gven by αˆ = Theefoe, OL estmato of β s gven by Z Y Fo the model, we get the followng estmatos. βˆ = T αˆ. 3 The G, J and MJ estmatos of α ae gven by esectvely αˆ = [ ] G αˆ = [ ] J MJ αˆ 4 αˆ αˆ = [ ] [ ] αˆ 6 whee =, and = dag,, 0, =,; be the dffeent basng constants. f = =...= = and > 0, then The odnay dge egesson estmato O of α s = Ι, we get the followng odnay estmatos. O ˆα = αˆ 7 mlaly, the odnay Jackknfed dge estmato OJ of α s ˆα = OJ αˆ 8 and Modfed odnay Jackknfe dge estmato MOJ of α s ˆα = MOJ αˆ 9 3

4 whee = t s easly seen fom Eq. 6, that MJ estmato s based estmato ofα. The bas s MJ Bas αˆ = θ α whee = [ ] θ and t s ostve defnte matx Batah et al The vaance of MJ s V αˆ MJ = whee W = [ ] [ ] MJ. σ W W 0 ME αˆ = σ W W θ αα θ 3. The Poosed Estmato n ths secton, we suggest a new estmato called as genealzed Jackknfed dge egesson estmato by combnng dea of G and J estmatos on the smla lne of MJ and t s genealzed veson of MJ estmato. t s denoted by αˆ. The αˆ s defned by αˆ = [ ] [ ] αˆ s 0 ettng = =...= = and 0, then estmato of α s temed as genealzed odnay jackknfed dge egesson estmato GOJ and t can be wtten as αˆ = GOJ αˆ Theefoe, βˆ GOJ = Tαˆ GOJ 3 Thus, the coodnatewse estmatos can be wtten as s ˆ = s α ˆ α L and αˆ GOJ = s s αˆ L s 0, =,,...,. 4 4

5 whee ˆα L ae the ndvdual comonents of αˆ. Obvously, αˆ = αˆ L and αˆ GOJ = αˆ L when = 0 fo all and = 0 esectvely. Bas, Vaance and ME of estmato The bas of the estmato s gven by: Bas αˆ = E ˆ α ] α [ = E [ = ˆ] α α α α = [ ]α Bas αˆ { } α = { [ ] } α = { [ ] } α. = = Φ α whee = [ ] Φ. The Vaance of estmato s ˆ V α = V σ V whee V = The ME of estmato s ME αˆ = V α ˆ [ Basαˆ ][ ˆ ] Basα,. 6 = σ V V Φ αα Φ. 7 emak. educes to J and MJ fo s = 0 and s = esectvely. 4. The Comason between and MJ The estmato s based estmato and bas s obtaned n the evous secton. n ths secton, we dscuss the efomance of usng the ME cteon. Fo the sake of comleteness, let us ove the followng theoems.

6 6 Theoem : Let be a symmetc ostve defnte matx. Then fo any non negatve eal numbe s, [ ] = Φ s ostve defnte matx. Poof: nce, = dag, and = dag, = = dag > 0, =, Usng the nvese and othe oetes of matces, t can be wtten that = dag = dag = dag Fom the above matces, = dag 8 = dag,..., 9 nce 0 > and 0 fo all, each element s ostve. Fom matces gven n 8 and 9, we get Φ = dag,..., 0

7 7 Φ s a dagonal matx wth ostve dagonal elements. By the oety of ostve defnte matx, matx Φ s ostve defnte. Theoem : Let be a symmetc ostve defnte matx. Then the estmato has smalle vaance than the MJ estmato fo s >. Poof: Fom Eq. 0 and 6 MJ ˆ V α - V αˆ = W W σ - V V σ [ ] [ ] = s s σ σ MJ ˆ V α - V αˆ = H σ whee H ={ }{ } s. nce, = dag =dag = dag = dag 6,..., 6 6 Theefoe, H = dag s s s s s s,

8 ..., s s s H s dagonal matx wth ostve dagonal elements, hence H s ostve defnte matx. Theefoe, vaance of estmato s smalle than MJ estmato and ths dffeence s 0 fo s =. Lemma : Faebothe 976 Let be a symmetc ostve defnte matx. α s non zeo vecto. Then - αα s nonnegatve defnte f and only f α - α s satsfed. Fom ths lemma we have the followng theoem. Theoem 3: Let be symmetc ostve defnte matx. Then the dffeence = MEαˆ MJ - MEαˆ s a nonnegatve defnte matx f and only f the nequalty α [ σ H θ αα θ L ] α L s satsfed wth L = Φ. Poof. Fom Eq. and 7 we have = MEαˆ MJ - ME αˆ = V αˆ MJ [ Basαˆ MJ ][ Basα ˆ ] MJ - V α ˆ [ Basαˆ ][ ˆ ] Basα = V αˆ MJ - V α ˆ [ Basαˆ MJ ][ Basα ˆ ] MJ -[ Basαˆ ][ Basα ˆ ] = σ H θ αα θ Φ α α Φ whee H and Φ both ae ostve defnte matx. Let L = Φ L L can be wtten as [ ], the matx L L = L σ H θ αα θ L αα We have s a nonnegatve defnte f and only f L L s nonnegatve defnte, snce σ H θ αα θ s symmetc ostve defnte matx. Hence usng lemma, we may conclude that L L s non-negatve defnte f and only f the 8

9 nequalty α [ σ H θ αα θ L ] α L. Hence s a nonnegatve defnte. Fom ths theoem, we conclude that ME of s smalle than MJ estmato. emak : Let = 0, f and only f L L = 0. We can ewte the Eq. [ L σ H θ αα θ L ] = αα The ank of the left hand matx s, whle the ank of the ght hand matx s ethe 0 o. Theefoe = 0, cannot hold tue wheneve > and s >. t s hold when s = fo any value of. emak 3: The oosed estmato has less ME than MJ. But the MJ estmato has less ME than the G and J estmatos see Batah et al., 008. Theefoe, oosed estmato has smalle ME than the G and J estmatos.. umecal Examles n ths secton, we examned the efomance of the oosed estmato ove the dffeent estmato s G, J and MJ though a smulaton study n the at. n at B we examned the effect of vaous values of s on ME of the oosed estmato. Pat : We consde the tue model as Y = Xβ ε. Hee ε follows a nomal dstbuton 0, σ and the exlanatoy vaables ae geneated see Batah et al., n 008 fom / x j = ρ u jρu, =, n j=,. whee u j ae ndeendent standad nomal andom numbes and ρ s the coelaton between x j and x j fo j, j < and j j. j, j =,,,. When j o j =, the coelaton wll beρ. Hee we consde = 4, ρ = 0.9, 0.99, and

10 These vaables ae standadzed such that X X s n the coelaton fom and t s used fo the geneaton of y wth β =,, 0, 0. We have smulated the data wth samle szes n =, 30, 0 and 00. The vaance of the eo tems ae taken as σ = and. The exement s eeated 00 tmes and obtans the aveage ME as ME αˆ = ˆ α j α = j= whee, αˆ j denote the estmato of the th aamete n the j th elcaton and α, =,,3, 4 ae the tue aamete values. n ths study, estmato s comuted fo s =. n case of odnay dge egesson, basng constants HKB Hoel, Kennad and Baldwn 97, and HMO Masuo Nomua, 988 ae comuted as HKB = ˆ σ ˆ ααˆ / HMO = = ˆ ˆ ˆ / ˆ α α σ =, σ. The ME atos of G, J, MJ ove and MJ, ove OL estmato ae comuted fo vaous values of tlet n,ρ, σ. These ae eoted n Table.. n case of genealzed dge egesson, followng methods ae used fo detemnng the basng constant. ˆ σ HKB = =, ˆ α ˆ σ / HMO = ˆ σ α ˆ α / ˆ =, Hoel and Kennad 970 Masuo Nomua, 988 0

11 whee αˆ s the th element of αˆ, =, and σˆ ˆ Y Yα Z Y = s the OL n estmato of σ. The ME atos of G, J, MJ ove estmato ae comuted fo vaous values of tlet n,ρ, σ. These ae eoted n Table.b. Table. ato of ME of vaous estmatos ove and MJ, ove OL n case of odnay dge egesson. n ρ σ G HKB HMO HKB J MJ G J MJ MJ OL OL

12 Table. ato of ME of vaous estmatos ove n case of genealzed dge egesson. n ρ σ G HKB HMO J MJ G J MJ Fom both Tables.,., t can be clealy seen that the efomance of the oosed estmato s sueo to the G, J, and MJ estmatos. We obseve that efomance of s excellent fo all combnatons of samle sze n, coelaton between edcto vaables ρ and eo vaance σ.

13 n the followng fgue, eesents the ME atos of MJ and ove OL estmato usng basng constant HKB eoted n Table.last two columns. Hee nut values ae n, ρ and σ. These nut values ae odeed accodng to the ncease of values. Fo fxed value of n changes values of ρ and fo fxed n,ρ changes the values ofσ. Thee ae 3 set of n,ρ, σ values. These ae aanged as,0.9,,.0.9,,,00,0.9999, and t s numbeed as,,, 3 esectvely. Fg.. The ME atos of MJ and estmatos ove OL estmato fo dffeent odeed sets of n, ρ, and σ ME ato MJ et Numbe The Fg.. eesents the ME atos of ove OL s less than ME atos of MJ ove OL. The oveall efomance of the s sueo than MJ n the context of ME fo all ossble combnatons of n,ρ, σ. Pat B: The same ocedue gven n at s eeated 00 tmes fo dffeent values of n,ρ, σ and s. ME of the oosed estmato s obtaned usng basng constant σˆ HKB =. The esults ae eoted n Table.3. αˆ αˆ 3

14 Table.3 ME atos of estmato ove OL estmato fo vaous values of s. n ρ σ =

15 Fom the Table.3, we obseve the followng attens.. choce of the s s deends on the samle sze n, coelaton between edctos ρ and eo vaance σ.. Oveall, ME of estmato s deceases when s s nceases. n ae cases, we obseve that ME of estmato s nceases when s s nceases. 3. Not all, but n majoty cases ME atos ae mnmum fo s lcaton of estmato n vaable selecton n ths secton, we demonstate the efomance of the oosed estmatoβˆ vaable selecton n egesson n the esence of multcollneaty. Fo ths study, we have used - statstc oosed by Dougade and Kashd, 00. -statstc s defned as GOJ, n = n = Υˆ Υˆ k σ t H H t HH whee, s the numbe of aametes of the subset model Y β ε. = X ˆ σ ˆ ˆ. σ s elaced by ts sutable estmate = Y Xβ O Y Xβ O n k H = X X X X, H X X X = X. Υˆ k and Υˆ ae the edcted values of Y, based on the full and subset models esectvely. The values of -statstc ae comuted usng the O, OJ, MOJ and GOJ estmates of β fo Hald data and smulated data. Examle 6.: Hald Cement data Montgomey et al., 006: The values of statstc ae comuted fo all ossble subsets and eoted n Table 6..

16 t s clea that, -statstc agee fo the motance of same subset {X, X } fo all estmates of β. The -statstc usng -statstc than othe estmatos. βˆ GOJ towads the exected value of the Table 6. : The values of fo dffeent dge estmatos usng HKB. Model βˆ O βˆ OJ βˆ MOJ βˆ GOJ ,j,k, ndcates the vaable X,X j, X k, n the model Examle 6. We have geneated andom samle fom N 3 0, Σ on X, X and X 3 and andom eo vaable ε s geneated fom nomal wth mean 0 and vaance. whee Σ= esonse vaable Y s geneated usng the followng model.. Y = 3X X ε, ε ~ 0,64 The values of -statstc ae comuted fo all subset models and eoted n Table

17 Fom the Table 6., obtaned usng vaous dge estmatos ck u the same subset {X, X }. -cteon usng βˆ GOJ s close to when subset model s adequate as comaed to the -cteon detemned usng othe dge estmatos. Table 6. : The values of fo dffeent dge estmatos usng HKB. 7. Concluson Model βˆ O βˆ OJ βˆ MOJ βˆ GOJ ,j,k, ndcates the vaable X,X j, X k, n the model n ths ae, we have oosed a new method fo aamete estmaton n lnea egesson when multcollneaty s esent. The efomance of the oosed estmato s evaluated though ME. t ndcates that ME of estmato s smalle than MJ, G, and J fo dffeent values of basng constants. The choce of s s based on the natue of the data set. lso, the efomance of oosed estmato s satsfactoy n vaable selecton method when data exhbts multcollneaty. efeences [] Batah, amnathan and Goe 008, The Effcency of Modfed Jackknfe and dge Tye egesson Estmatos: comason, uveys n Mathematcs and ts lcatons, Vol. 4- No., [] Dougade and Kashd 00, Vaable electon n Lnea egesson Based on dge Estmato, Jounal of tatstcal Comutatons and mulatons To aea. [3] Faebothe 976, Futhe esults on the Mean quae Eo of dge egesson, jounal of the oyal tatstcal ssocaton, se. C, Vol. 38 3, [4] Hoel and Kennad 970, dge egesson: based estmaton fo Nonothogonal oblems, Technometcs, Vol., [] Hoel and Kennad and Baldwn 97, dge egesson: ome mulatons, 7

18 Communcaton tatstcs, Vol. 4, [6] Hnkley 977, Jackknfng n Unbalanced tuatons, Technometcs, Vol. 93, [7] Montgomey Peck and Vnng 006, ntoducton to Lnea egesson nalyss, 3 d ed., John Wley and ons ub. [8] Nomua Masuo 988, On The lmost Unbased dge egesson Estmaton, Communcaton tatstcs. mulaton, Vol. 73, [9] ngh, Chaubey and Dwved 986, n lmost Unbased dge Estmato, ankhya, e.b, Vol. 48, [0] Vnod and Ullah 98, ecent dvances n egesson Methods, Macel Dekke nc. ub. 8