An Application of the Generalized Shrunken Least Squares Estimator on Principal Component Regression

Transcription

1 An Appliction of the Generlized Shrunken Let Squre Etimtor on Principl Component Regreion. Introduction Profeor Jnn-Huei Jinn Deprtment of Sttitic Grnd Vlley Stte Univerity Allendle, MI 0 USA Profeor Chwn-Chin Song nd Mr. W. B. Mou Deprtment of Applied Mthemtic Ntionl Cheng-Chi Univerity Tipei, Tiwn, R.O.C. When colinerity exit in tudy of multiple liner regreion, it will yield lrge etimted vrince for the etimted coefficient in the model nd it i then difficult to detect the ignificnt regreion coefficient. The problem cued by colinerity cn be overcome omewht by ( deleting predictor vrible tht re trongly correlted ( relting the repone vrible to the principl component of the predictor vrible. The repone vrible i then regreed on thee new predictor vrible ( incree mple ize. Thi tudy will propoe modified method to delete predictor vrible tht re trongly correlted. Thi method i bed on Li-Chun Wng (Generlized Shrunken Let Squre Etimtor,0 ide nd the method of deleting predictor vrible by Mnfield, Webter, nd Gunt (. We believe the modified method will reduce more etimted vrince for the etimted coefficient in the model. We pln to ue rel dt et to compre our modified method nd the method propoed by Mnfield, Webter, nd Gunt (.. Multiple Liner Regreion Model nd Multicollinerity. The multiple liner regreion model cn be written Y = β + β +ε (. o ε ~ N n (0, σ I n Where Y i n repone vector β o i n unknown contnt term i n vector with ll element i n k deign mtrix with ll the element h been tndrdized, i.e., = 0 = i nd i. i i --

2 β i k prmeter vector ε i i I n n rndom error vector n n identity mtrix of order n A very deirble condition in et of regreion dt i to hve orthogonlity. The condition of orthogonlity of n experiment deign occur when one truly doe hve the cpbility to control the regreor vrible. Let = [ x, x,..., x k ] where x i n vector of n obervtion for the th predictor. Multicollinerity imply occur when there re ner liner dependence mong the column of. Tht i, there i et of contnt k = (not ll zero for which c c 0 (. x x, the We write ince if the right hnd ide i identiclly zero, the liner dependencie re exct nd thu (/ doe not exit. Ner dependencie, of coure, my exit in rel dt nd produce the effect tht commonly cll multicollinerity. One hould keep in mind tht regreion coefficient i rte of chnge or prtil derivte of the repone with repect to regreor vrible. When the x-dt re conditioned in uch wy tht the regreor re moving with one nother, the let qure procedure never i llowed expoure to the dt tructure tht it truly need to produce cler etimte of thi rte of chnge. Let βˆ = Y/( repreent the vector of let qure etimte of β in the model of (.. The covrince mtrix of βˆ i V( βˆ = σ /(. Suppoe we conider the mtrix (correltion form. We know tht there exit n orthogonl mtrix (ee Grbill ( V= [ v v,..., ] λ, λ,..., λ k. (., v k uch tht V ( V=dig ( The λ i re the eigenvlue of the correltion mtrix. The opertion given by (. i clled the eigenvlue decompoition of. The column of V re normlized eigenvector ocited with the eigenvlue of. For our purpoe here, we need to denote the ith element of the vector v by v. Now if multicollinerity i preent, t let one λi 0. Thu we cn write, for t let one vlue of, k i ' v ( v 0 which implie tht for t let one eigenvector v, v x l l 0. Thu the number of mll eigenvlue of the l= correltion mtrix relte to the number of multicollinerity ccording to the definition in (., nd the weight, the c in (., re the individul element in the ocited eigenvector. --

3 Define D= (/, the digonl element of thi mtrix cn be expreed d = R, =,,...,k where i the coefficient of multiple determintion of the regreion R produced by regreion the vrible x gint the other regreor vrible, the x i ( i. Thu the higher the multiple correltion in the regreion, the lower the preciion in the etimte of the coefficient βˆ. Tht i, when pproche to, the vrince of individul let qure etimtor βˆ, Vr( βˆ = d σ = σ (,will become very lrge. R Define the qured ditnce between the etimted vector βˆ nd the true prmeter vector β L =( βˆ - β ( βˆ - β. The other pproch in illutrting the effect of multicollinerity on the coefficient i the conidertion of the expected qured ditnce between the etimted vector βˆ nd the true prmeter vector β. i.e., R E( L =E[ ( βˆ - β ( βˆ - β ]= σ tr(/ = σ where λ >0 re the eigenvlue of the correltion mtrix, k = λ (. =,,..., k. Thu for n ill-conditioned or ner ingulr, t let one of the eigenvlue will be mll nd thu E[ ( βˆ - β ( βˆ - β ] will be lrge. For the idel ce (orthogonlity, k = =k. It become cler, then, from eqution (., tht ince λ E[ ( βˆ - β ( βˆ - β ]=E( βˆ βˆ -( β β then E( βˆ βˆ =( β β + σ k = λ (. Eqution (. undercore the tendency for multicollinerity to produce vector of regreion coefficient tht i too long, i.e., coefficient tht hve the tendency to be too lrge in mgnitude. If ny of the λ re mll, obviouly ( βˆ βˆ i hevily bied upwrd for ( β β, nd hence one would expect coefficient tht re lrge. Thi i true in pite of the fct tht the βˆ themelve re unbied. For exmple, n eigenvlue of i not t ll uncommon in highly colliner itution. Clerly, from (., for thi itution, k = ˆ β i hevily bied, nd the reult i tendency for ome of the coefficient to be overetimted in mgnitude. --

4 . Multicollinerity Dignotic The following repreent forml multicollinerity dignotic tool. Simple Correltion Among the Regreor Vrible The nlyt normlly h cce to the correltion mtrix of the regreor vrible, i.e.,. Thee number, of coure, indicte pirwie type correltion. However, we hould clrify tht multicollinerity quite often involve ocition mong multiple regreor vrible. A reult, the imple correltion themelve do not lwy undercore the extent of the problem. There re no definite guideline vlue on the imple correltion nd, while they hould be oberved o tht the nlyt cn ee which one-on-one ocition exit, they do not lwy indicte the ctul nture or the extent of the multicollinerity. b Vrince Infltion Fctor Define VIF= the vrince infltion fctor. The VIF repreent the R infltion tht ech regreion coefficient experience bove idel, i.e., bove wht would be experienced if the correltion mtrix were n identity mtrix. It i ey to ee tht it involve the notion of multiple ocition. If R i ner unity, (VIF will be quite lrge. Thi will occur if the ith regreor vrible h trong liner ocition with the remining regreor. The VIF repreent coniderbly more productive pproch for detection thn do the imple correltion vlue. They upply the uer with n indiction of which coefficient re dverely ffected nd to wht extent. It i generlly believed tht if ny VIF greter thn 0, there i reon for t let ome concern; then one hould conider vrible deletion or n lterntive to let qure etimtion to combt the problem. c Condition Number of the Correltion Mtrix We know tht eigenvlue nd eigenvector of the correltion mtrix,, ply n importnt role in the multicollinerity tht exit in et of regreion dt. Indeed, the nerne to zero of the mllet eigenvlue i meure of the trength of liner dependency, while the element of the ocited normlized eigenvector diply the weight on the correponding regreor vrible in the multicollinerity. Of coure, the eigenvlue would ll be if the vrible define n orthogonl ytem o thi provide norm for the nlyt. In ddition, the pectrum of eigenvlue produce nother dignotic. Multicollinerity cn be meured in term of the rtio of the lrget to the mllet eigenvlue. --

5 Define the condition number of the correltion mtrix λmx κ = λmin Lrge vlue ofκ re n indiction of eriou multicollinerity. An exceively lrge κ i evidence tht the regreion coefficient re untble, i.e., ubect to mor chnge with mll perturbtion in the regreion dt. Numericl rule of thumb y κ <00 i indicte wek multicollinerity, 00<κ <,000 indicte moderte multicollinerity, if κ >,000 one hould be concerned bout the effect of multicollinerity. The condition numberκ i more relible for dignoing the impct of dependency thn the eigenvlue λ itelf.. Alterntive to Let Squre When Multicollinerity Exit There re mny etimtion procedure deigned to combt multicollinerity, procedure tht were developed to eliminte model intbility nd to reduce the vrince of the regreion coefficient. At the point in which the nlyt h determined, by the ue of the dignotic, tht Multicollinerity i problem, often ubtntil benefit my be derived from n ttempt to eliminte much of the multicollinerity without reorting to lterntive to let qure. The very preence of multicollinerity in the dignotic ugget tht, in the ce of k regreor vrible, the ctul model-building exercie hould involve fewer thn k vrible. In other word, there i not ufficient informtion in the regreor dt to wrrnt modeling k regreor. A reult, the nlyt often cn eliminte or certinly reduce the effect of multicollinerity by removing one or more regreor.. Ridge Regreion Ridge regreion i one of the mot populr, though controveril, etimtion procedure for combting multicollinerity. The procedure relted to Ridge regreion fll into the ctegory of bied etimtion technique. They re bed on thi notion: though ordinry let qure give unbied etimte nd indeed enoy the minimum vrince of ll liner unbied etimtor, there i no upper bound on the vrince of the etimtor nd the preence of multicollinerity my produce very lrge vrince. A reult, one cn viulize tht, under the condition of multicollinerity, huge price i pid for the unbiedne property tht one chieve by uing ordinry let qure. Bied etimtion i ued to ttin ubtntil reduction in vrince with n ccompnied incree in tbility of the regreion coefficient. The ridge regreion etimtor of the coefficient β i found by olving for βˆ in the ytem of eqution ( +φ I n βˆ R = Y (. -- R

6 whereφ 0 i often referred to hrinkge prmeter. The olution, of coure, i given by βˆ = [/( +φ I ] Y (. R n There re vriou procedure for chooing the hrinkge prmeterφ. A firly imple tudy of the propertie of the ridge etimtor in (. revel the role of φ in moderting the vrince of the etimtor. Perhp the mot drmtic illutrtion of the impct of mll eigenvlue on the vrince of the let qure coefficient i the expreion for ( Vr ˆ β / σ given in eqution (.. In the ce of the ridge regreion etimtor, the equivlent property i given by k Vr ˆ β k λ = σ, R = = λ + ( φ For exmple, in the ce of k= regreor vrible with λ =., λ =0.0, nd λ =0.00, let qure etimtion give = Vr ˆ β σ = = λ = =00.0 (. If ridge regreion with yφ =0.0 i ued, the um of the vrince i given by λ. ( λ + φ = It i cler when multicollinerity i evere, i.e., when there i t let one ner zero eigenvlue, much improvement in vrince, nd thu coefficient tbility, cn be experienced. Eqution (. emphize tht theφ in ridge regreion moderte the dmging impct of the mll eigenvlue tht reult from the collinerity. The bi tht reult for election ofφ >0 i bet quntified by oberving n expreion k k for ( Bi ˆ β = [ E( ˆ β β ], the um of the qured bie of the regreion =, R =, R coefficient. Thi expreion i given by (ee Hoerl nd Kennrd (0( k E ( ˆ β, R β = φ = [ ] β [ +φ I n ] β (. Thu we cn expect tht the procedure of ridge regreion would be ucceful if φ i choen o tht the vrince reduction i greter thn the bi term given in (.. There i no urnce tht thi cn be done becue the nlyt will never know wht the bi i. The choice ofφ belong to the nlyt, of coure, nd prmeter vlue hould be choen where reult how trong evidence tht improvement in the etimte re being experienced. --

7 . Principl Component Regreion Principl component regreion repreent nother bied etimtion technique for combting multicollinerity. We perform let qure etimtion on et of vrible clled the principl component of the correltion mtrix. Bed on the nture of the nlyi, we delete certin number of the principl component to effect ubtntil reduction in vrince. The method vrie omewht in philoophy from ridge regreion but, like ridge regreion, give bied etimte; when ued uccefully, thi method reult in etimtion nd prediction tht i uperior to OLS (ordinry let qure. Principl component re orthogonl to ech other, o tht it become quite ey to ttribute pecific mount of vrince to ech. Conider the mtrix of normlized eigenvector ocited with the eigenvlue ( λ, λ,..., λk of (correltion form. Let V= [ v, v,..., v k ] be k k orthogonl mtrix where the th column vector v of V i normlized eigenvector ocited with the th eigenvlue λ of. We know tht VV =V V=I ince V i n orthogonl mtrix. Hence we cn write the originl regreion model in the form Y = β o +VV β +ε (. Y = β +Wγ +ε (. o where W =V nd γ =V β. W i n n k mtrix nd γ i k vector of new coefficient γ, γ,..., γ k. We cn viulize the column of W (typicl element w i repreenting reding on k new vrible, the principl component. It i ey to ee tht the component re orthogonl to ech other. We hve W W=(V (V=V V= dig ( λ λ,...,, λ k = (. So, if regreion i performed on the w vi the model in (., the vrince of coefficient (the digonl element of (W W prt from σ re the reciprocl of eigenvlue. Tht i, Vr( γˆ = σ λ =,,..., Note tht the γˆ re the let qure etimtor ( ˆ γ = W Y. The let qure prediction eqution then become k Yˆ = Y + ˆ γ (. u = -- k

8 where u = z v i the th principl component vlue for the point t which prediction i deired, z. If there re principl component ocited with the lrget ltent root were retined, then Y * = Y + u = Bed on (. the reidul um of qure for * SSE = n * ( Yi Yi i= ˆ γ (. n * Y i = ( Y i Y λ ˆ γ i=. Generlized Shrunken Let Squre Etimtor Li-Chun Wng (0 propoed Generlized Shrunken Let Squre Etimtor (GSLSE, βˆ GS =VAV βˆ, (.0 to etimte β in the model Y = o 0 =,,...,k, nd V= [ v, v,..., ] i n orthogonl mtrix uch tht V V= V ( V=dig ( λ λ,..., = β + β +ε. Where A=dig (,..., v k, λ k =. For convenience,, k,, λ tify λ λ... λ 0. k > Wrung-Shen Yeh ( derived the leverge formul nd ome propertie for GSLSE. He lo proved tht when there i evere multicollinerity, GSLSE provide better etimte thn let qure etimte, βˆ.. Vrible Selection Method. MWG Vrible Selection Technique Mnfield, E. R., Webter, J. T., nd Gunt, R. F. (MWG, preented n nlytic technique for deleting predictor vrible from liner regreion model when principl component of re removed to dut for multicollineritie in the dt. Conider model (. nd prtition V=[ V : V k ], where V = [ V, V,..., V ] contin the ltent vector correponding to the lrget ltent root. Conequently, W=[ W : Wk ]=[ V : V k ]. Then model (. cn be rewritten γ Y = β o +[ W : Wk ] + ε γ k Let =dig ( λ, λ,..., λ '. Then if γ = ( γ, γ,..., γ, the γˆ ued in (. cn be ' obtined γˆ = ( ˆ γ ˆ, ˆ γ,..., γ = W Y. (. --

9 Conidering deleting r independent vrible from the predictor (.. Since the ordering of the vrible in i rbitrry, uppoe the one to be re the lt r vrible. V Now prtition V = where V i n (k-r mtrix nd V i n r mtrix whoe V row correpond to the r vrible to be from the predictor, i.e., the lt row of. In order to obtin reduced predictor with the r vrible, MWG conidered V n lterntive etimtor of γ : where A =dig,,...,. Note tht if A = I, ( ~ γ = A W Y = A γˆ. (. ~ γ = γˆ. The etimtor of β correponding to (. i ~ β = V ~ ~ V A ˆ γ β γ = = k r V A ˆ γ ~, (. β r where the lt r row in (. correpond to the independent vrible which re to be. MWG howed tht mtrix cn be found which minimize the reidul um of qure ubect to V A A ~ γ =0, i.e., ~ β =0, which remove the lt r vrible from the r reulting predictor. Ue of the principl component etimtor with r vrible reult in prediction eqution where z Y ~ k r ~ =Y + ~γ = Y + β, (. u = z = i vlue of the th independent vrible from the originl model (.. The reidul um of qure for (. cn be written n ~ * * SSE = ( Yi Yi = SSE + ( λ ˆ γ = SSE + u r (. i= where i the incree in the reidul um of qure from (.. u r Since the originl ordering of the vector of the mtrix i rbitrry let the r = to be be the r right-hnd column of. The problem i then to minimize the incree in reidul um of qure, u r = ( λ ˆ γ ubect to the retriction ~ β r = V MWG( proved the element of the digonl mtrix = v ' =-{ [ V V k r ] V γˆ / λ γˆ } A i A ~ γ =0. =,,..., (. βˆ = [- ( V V V ] γˆ (. V V nd u = ( λ ˆ γ = γˆ ( V V r = where V = v, v,..., v. ( V -- V γˆ (.

10 MWG ( propoed the following tep to delete independent vrible: Step: The vlue of u would be computed for ech of the k vrible in (. by uing ˆ ( Yi Yi (.. denote the mllet of thee k u by u i m. Uing MSE= n k (the let qure men qured error from the full model, F= /MSE cn be ued to determine whether thi vrible hould be. If ignificnt delete the correponding u m u m i ufficiently mll (the tet i not from the model nd proceed to tep. Step: The econd tep delete the vrible removed previouly nd ech of the remining (one t time, clculting for ech of the k- pir of vrible. Denote u u the mllet of the u vlue by m. Ue F=( m - m /MSE to determine whether thee vrible hould be. If u m -u m i ufficiently mll, both the vrible involved re ; otherwie, only the vrible in tep i removed nd the proce i terminted. u Step: At thi tep the mllet of k- vlue i determined by clculting (. for the two vrible removed in tep nd ech of the one remining. Denote the mllet of the u vlue by u m. Ue F=( u m - u m /MSE to determine whether thee vrible hould be. If u m - u m i ufficiently mll, ll three vrible involved re nd the procedure i continued; otherwie, the proce i dicontinued with only the two vrible t tep removed. The k-l vrible remining when the elimintion top (i.e., l vrible re re conidered importnt independent vrible yielding informtion on the dependent vrible, Y. The + tep ue F=( u( + m u m /MSE, =0,,,,l the tet ttitic vlue where =0 nd l i the number of vrible. u 0m An lterntive bckwrd elimintion-type technique for deletion of vrible i to proceed in the firt tep bove, but reevlute the model t ech tep. In other word, t tep exmine the ltent root nd vector of the (k- (k- reduced mtrix, remove component correponding to mll ltent root, nd then clculte new u m from the k- u vlue. Uing MSE from the full model, F= u m /MSE cn gin be ued to determine whether thi econd vrible hould be. The uthor (MWG found tht thi ltter bckwrd elimintion h performed more tifctory thn the bove method on everl dt et they exmined. u u -0-

11 . The Reltionhip Between MWG Etimtor nd Generlized Shrunken Let Squre Etimtor The MWG etimtor i defined ~ β = V ~ ~ V A ˆ γ β γ = = k r V A ˆ γ ~ (. β r The generlized hrunken let qure etimtor i defined βˆ =VAV βˆ (.0 ( where A=dig,...,, GS, k 0,,,..., k =, nd V=[ v v,..., ] orthogonl mtrix uch tht V V= V ( V=dig ( λ λ,...,, v k i n, λ k =. The th column vector v of V i normlized eigenvector ocited with the th eigenvlue λ of. For convenience let λ tify λ λ... λ 0. k > Define the mot generlized ridge etimtor of β in the multiple liner regreion model Y = β + β +ε o βˆ MGR = VAV βˆ (. where βˆ i the let qure etimtor of β, A=dig (,..., with < <, =,,..., k. If we chnge <, k i digonl mtrix < in the digonl mtrix A of (. to 0 then βˆ MGR i the βˆ GS of Li-Chun Wng (Generlized Shrunken Let Squre Etimtor,0. In the MWG method they prtitioned V=[ V : ], where V =[ V V,..., V ] V k, contin the ltent vector correponding to the lrget ltent root. If we let Ak =0 in A 0 the digonl mtrix A= then we cn obtin ~ β = V ~ ~ V A ˆ γ β γ = = 0 A k r k V A ˆ γ ~. β r Therefore, ~ β nd βˆ GS re pecil ce of βˆ MGR. If we wnt to improve the preciion of ~ β then we cn try to confine the digonl element of A in [0,] then in thi ce, ~ β become pecil ce of βˆ. GS Thi tudy i to pply the contrint of 0, =,,..., k in the generlized hrunken let qure etimtor to the method of deleting regreor propoed by Mnfield et l. (. We believe tht the ppliction of the contrint to the MWG etimtor will get better reult of vrible election procedure.. Some Propertie of MWG Etimtor Since ~ γ = A W Y = A γˆ = [- V ( V V V ] γˆ (., the expected vlue of ~ γ i E( ~ γ =[- V ( V V V ] γ (. --

12 nd vrince-covrince Vr( ~ γ = σ [ - V ( V V V ]. (. The bi of ~ γ i E( ~ γ -γ = -[ V ( V V V ] γ Since vrince-covrince mtrix of γˆ i Vr( γˆ = σ nd V ( V V V i poitive definite mtrix, it implie V( ~ γ V( γˆ. Tht i, MWG etimtor i better thn the let qure etimtor. The derivtion of (. nd (. re given in Appendix A. In generl, the let qure etimtor h the following two propertie: ( The um of reidul i equl to zero. ( The reidul vector i orthogonl to the prediction vector. We will how tht MWG etimtor, β ~, tifie both propertie. The reidul vector of MWG etimtor i e ~ =Y - Y - ~ β. Since i element h been tndrdized, i.e., = 0 = i nd i. i It i ey to prove tht e ~ = ( Y - Y - ~ β =0. i n k deign mtrix with ll the If we wnt to how tht MWG etimtor h property ( then the β ~ in (. mut tifie ~ β r = V A γˆ =0, nd γˆ '( A - A A γˆ =0 (. (ee Appendix B for the derivtion of (.. The problem i then to minimize uch tht V u r = A = γˆ =0 nd ( λ ˆ γ γˆ '( A - A A γˆ =0. (. The olution of (. i urpriingly the me (.. Thi prove MWG etimtor h property ( (ee Appendix C for the proof. When there i only one independent vrible, i.e. r=, we cn how tht (. tifie (.. Firt, γˆ '( A - A A γˆ cn be rewritten = ˆ ( λ γ nd in (. cn be rewritten =-V ' wherev = V, V,..., V i the lt row of V. Therefore, = λ [-V = k ( k k k ( = V k k / λ ( = V k λ ˆ γ ˆ γ ][ Vk ( = -- V k / k = V ( = V k k = λ ( ] λ ˆ γ / λ ( ˆ ( λ γ ˆ γ γˆ = V k λ ˆ γ in ˆ γ

13 = [ V ˆ kγ ] ˆ γ = = = Vk ( Vk / λ ( [ Vk ( Vk / ( = = = λ V ˆ γ ]/ λ = ( V k / λ ( V k ˆ γ ( V k / λ ( V k ˆ γ ( Vk / λ = = = ( V k / λ ( V k ˆ γ - ( V k / λ ( V k γˆ =0 = = = = = = = k. MWG Vrible Selection Technique With Retriction In ection.., eqution (. ~ γ = A W Y = A γˆ or eqution (. ~ β = V ~ γ = V A γˆ, there were no retriction to the digonl element of mtrix A,therefore ome vrince could become vry lrge. In ection. we found ~ β nd βˆ GS re pecil ce of βˆ MGR. If we wnt to improve the preciion of ~ β then we cn try to confine the digonl element of A in [0,] then in thi ce, ~ β become pecil ce of βˆ. If we pply the contrint of 0, =,,..., k in the GS generlized hrunken let qure etimtor to the method of deleting regreor propoed by Mnfield et l. (, wht will be the effect to the MWG vrible election procedure? We believe tht the ppliction of the contrint to the MWG etimtor will get better reult of thi vrible election technique. In eqution (., βˆ = VAV βˆ, A=dig (,..., MGR, k i digonl mtrix with < <, =,,..., k. We now chnge < < in the digonl mtrix A to 0. The MWG vrible election tep re imilr to thoe tep in ection.. The minimiztion of (. cn be rewritten Min u = ( λ ˆ γ (. = r V tγˆ Vt = =0, t=k-r+, k-r+,,k (. nd 0, =,,..., (. where re element of V = v, v,..., v (n r mtrix whoe row correpond ( to the r vrible to be from the predictor, i.e., the lt row of V. --

14 To find the minimum vlue of (., we propoe the following two method: ( Ignore the contrint of (. nd ue Lgrnge multiplier to find the vlue of (ee eqution (.. Then ue Ro-Ghngurde method to dut o tht (. i tified. If ll the [ r [ re in 0,] then ubtitute them into (. to find u ; if ome re not fll within 0,], the Ro-Ghngurde dutment method i ued follow: If > then ue =; if <0 then ue =0. Subtituting thee duted (long with thoe lredy fll within [0,] into (. nd (. to obtin updted eqution of (. nd (.. Then ue Lgrnge multiplier to find the vlue of. If ll the fll within [0,], then thee re the olution. Otherwie, repet the me procedure until ll the re fll [ within 0,]. ( Ue LINGO oftwre to find vlue of then ue MATLAB to find the increment of the reidul um of qure. We will ue the econd method for the rel dt nlyi in the next ection. If we pply the contrint of 0, =,,..., k it will ffect the increment of the reidul um of qure. Therefore the effect of deleting predictor will be different. In generl, the number of predictor under the condition of 0 will be le thn the number of predictor without thi retriction, i.e., < <. Thi i becue in the firt tep of deleting predictor (ee ection., the increment of the reidul um of qure under the condition of 0 will be much lrger thn the increment of the reidul um of qure under the condition of < <. Therefore the F tet ttitic will be ignificnt under the condition of0 but not ignificnt for the F tet ttitic under < conervtive under the condition of 0.. The Three Mor Fctor of Deleting Predictor <. Hence the deletion of predictor will be more Uing the deletion method of Mnfield et l. (, the order of deleting the predictor nd the number of predictor to be will be ffected by the following three mor fctor: ( determintion of principl component ( fter the deletion of predictor, hould we lo delete the correponding column nd row in the ' nd ' Y mtrice? ( hould we retrict the digonl element (of the digonl mtrix A in βˆ MGR = VAV βˆ under the condition of 0? In the principl component regreion nlyi, we delete thoe principl component correponding to mller eigenvlue, therefore, the number of principl component will be le thn the number of predictor. In the next ection, we will ue rel dt to dicu whether we hould delete thoe principl component correponding to mller eigenvlue becue they will ffect the reult of deleting predictor in the model. --

15 . Dt Anlyi In thi ection, we will ue two exmple to compre the reult of etimting the MWG prmeter nd the deletion of predictor under the two different condition, i.e., 0 nd < <. The correltion coefficient mtrice of thee two dt et re given in Appendix D.. The Pitprop Problem n Exmple..: The Decription of Dt The pitprop dt of Jeffer ( rie from the tudy of phyicl propertie of pitprop, to, nd their reltionhip to the compreive trength of the pitprop (Y. A ummry of the dt of 0 pitprop i given in Jeffer ( in the form of pirwie correltion. The vrible tudied were: (TOPDIAM: the top dimeter of the pitprop in inche; (LENGTH: the length of the pitprop in inche; (MOIST: the moiture content of the pitprop, expreed percentge of the dry weight; (TESTSG: the pecific grvity of the timber t the time of the tet; (OVENSG: the oven-dry pecific grvity of the timber; (RINGTOP: the number of nnul ring t the top of the pitprop; (RINGBUT: the number of nnul ring t the be of the pitprop; (BOWMA: the mximum bow in inche (BOWDIST: the ditnce of the point of mximum bow from the top of the pitprop in inche; (WHORLS: the number of knot whorl; 0 (CLEAR: the length of cler pitprop from the top of the pitprop in inche; (KNOTS: the verge number of knot per whorl; (DIAKNOT: the verge dimeter of the knot in inche; Y=Compreive trength Tble in Appendix D give the coefficient of correltion between ech of the vrible, one terik indicting ignificnce t the 0.0 of probbility nd two terik indicting ignificnce t the 0.0 of probbility. The high degree of intercorreltion between the vrible i evident from thi tble. The two oftwre, MATLAB nd LINGO, re ued to clculte the reult given in the tble of thi ection. --

16 In thi exmple, the ordered eigenvlue of re:.,.,.,.0, 0.0, 0., 0., 0.0, 0., 0., 0.00, 0.0, nd 0.0. The condition number of λmx the correltion mtrix i κ = =./0.0=0.0, it indicte there exit λmin moderte multicollinerity problem. Tble.. give the vlue of etimted regreion coefficient, tndrd devition, t tet ttitic vlue, p-vlue, R vlue for full model (ll the vrible included in the model nd etimted regreion coefficient, tndrd devition for uing bckwrd elimintion method. Tble..: Summry Sttitic for Full Model nd Bckwrd Elimintion Regreion on ll predictor Predictor Regreion Coefficient Stndrd Devition t-vlue p-vlue R Bckwrd Elimintion Regreion Coefficient Stndrd Devition Note: Bckwrd method eliminted, 0, nd. In tble.., there re five predictor with R >0. nd the other four R vlue re le thn 0.. Therefore, ome of the (VIF vlue re quite lrge nd indicte evere multicollinerity problem. Bed on individul tet ttitic vlue nd p-vlue, ll the vrible re not ignificnt. But, the F ttitic vlue for full model how ignificnt reult. Thi indicte tht ome predictor hould be. Becue of the imilrity between MWG vrible deletion method nd bckwrd elimintion method, we lit the etimted regreion coefficient, tndrd devition for uing bckwrd elimintion in tble... --

17 The um of qure of reidul i 0. (for ll the predictor, the men qure error with degree of freedom i When the three predictor (correponding to the three mllet eigenvlue were, the um of qure of reidul incree to 0. nd the men qure error incree to 0.00 The deletion or not of predictor were decided by compring the F-ttitic vlue with F 0. (, =.. Since there re three mor fctor to ffect the deletion of predictor, we hve conidered eight poible itution in the dt nlyi. The eight itution re, MWG method with or without deleting predictor nd whether [0,] or not; MWG modified method (ee ection. with or without deleting predictor nd whether [0,] or not. The etimted regreion coefficient nd tndrd error of MWG method cn be obtined by uing eqution (. nd (.. But the etimted regreion coefficient of MWG method with [0,] cn t be obtined by uing formul. Therefore, the tndrd devition cn t be clculted. Hence, we only dicu the procedure nd reult of deleting predictor. Ce I: Ue MWG Method Without Deleting Principl Component (: Without the condition, [0,], i.e., < < Tble..: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion 0 (DIAKNOT Delete 0 0 (WHORLS 0.0 Delete 0 0 (RINGBUT 0.0 Delete 0 (BOWDIST.00 Stop (keep --

18 Tble..: Etimted regreion coefficient of finl model Predictor Regreion Coefficient Stndrd Devition (TOPDIAM (LENGTH 0.. (MOIST -0.. (TESTSG 0.0. (OVENSG (RINGTOP (RINGBUT (BOWMA (BOWDIST (WHORLS (CLEAR (KNOTS (DIAKNOT Note:,, nd re. Tble..: The vlue of digonl element of digonl mtrix A Ce I: Ue MWG Method Without Deleting Principl Component (b: With the condition, [0,] Tble..: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion 0 (DIAKNOT.000 Keep --

19 Tble..: Etimted regreion coefficient of finl model Predictor Regreion Coefficient (TOPDIAM -0. (LENGTH -0. (MOIST -0. (TESTSG 0. (OVENSG -0.0 (RINGTOP 0. (RINGBUT 0. (BOWMA -0.0 (BOWDIST -0.0 (WHORLS (CLEAR 0.0 (KNOTS -0.0 (DIAKNOT 0.0 Note: The etimted regreion coefficient of MWG method with [0,] cn t be obtined by uing formul. Therefore, the tndrd devition cn t be clculted. Tble..: The vlue of digonl element of digonl mtrix A Ce II: Ue MWG Method nd Deleting Three Principl Component Correponding to the Three Smllet Eigenvlue II(: Without the condition, [0,], i.e., < < Tble..: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion (DIAKNOT 0. Delete (BOWDIST 0. Delete (KNOTS. Stop(Keep -- Tble..: Etimted regreion coefficient of finl model

20 Predictor Regreion Coefficient Stndrd Devition (TOPDIAM (LENGTH (MOIST (TESTSG (OVENSG (RINGTOP (RINGBUT (BOWMA (BOWDIST (WHORLS (CLEAR (KNOTS (DIAKNOT Note: nd re. Tble..0: The vlue of digonl element of digonl mtrix A II(b: With the condition, [0,] Tble..: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion (OVENSG.0 Keep -0- Tble..: Etimted regreion coefficient of finl model

21 Predictor Regreion Coefficient (TOPDIAM -0.0 (LENGTH (MOIST -0.0 (TESTSG -0. (OVENSG 0. (RINGTOP 0. (RINGBUT 0. (BOWMA -0.0 (BOWDIST 0.0 (WHORLS (CLEAR 0.00 (KNOTS 0.0 (DIAKNOT -0.0 Note: The etimted regreion coefficient of MWG method with [0,] cn t be obtined by uing formul. Therefore, the tndrd devition cn t be clculted. Tble..: The vlue of digonl element of digonl mtrix A Ce III: Ue Modified MWG Method Without Deleting Principl Component III(: Without the condition, [0,], i.e., < < Tble..: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion 0 (DIAKNOT Delete 0 0 (WHORLS 0.0 Delete 0 0 (RINGBUT 0. Delete 0 (BOWDIST. Stop (keep -- Tble..: Etimted regreion coefficient of finl model

22 Predictor Regreion Coefficient Stndrd Devition (TOPDIAM -0.. (LENGTH 0..0 (MOIST -0.. (TESTSG 0.. (OVENSG (RINGTOP (BOWMA (BOWDIST (CLEAR (KNOTS Note:,, nd re. Tble..: The vlue of digonl element of digonl mtrix A Note: Uing modified MWG method, there re only 0 predictor left in the tudy, therefore, mtrix A i 0 by 0 mtrix. III(b: With the condition, [0,] Tble..: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion 0 (DIAKNOT.000 Keep -- Tble..: Etimted regreion coefficient of finl model

23 Predictor Regreion Coefficient (TOPDIAM -0. (LENGTH -0. (MOIST -0. (TESTSG 0. (OVENSG -0.0 (RINGTOP 0. (RINGBUT 0. (BOWMA -0.0 (BOWDIST -0.0 (WHORLS (CLEAR 0.0 (KNOTS -0.0 (DIAKNOT 0.0 Note: The etimted regreion coefficient of MWG method with [0,] cn t be obtined by uing formul. Therefore, the tndrd devition cn t be clculted. Tble..: The vlue of digonl element of digonl mtrix A Ce IV: Ue Modified MWG Method nd Deleting Three Principl Component Correponding to the Three Smllet Eigenvlue IV(: Without the condition, [0,], i.e., < < Tble..0: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion (DIAKNOT 0. Delete (BOWDIST 0. Delete (LENGTH. Delete 0 (WHORLS 0. Delete 0 (KNOTS. Delete (CLEAR. Stop(Keep -- Tble..: Etimted regreion coefficient of finl model

24 Predictor Regreion Coefficient Stndrd Devition (TOPDIAM (MOIST (TESTSG (OVENSG (RINGTOP (RINGBUT (BOWMA (CLEAR Note:,,, nd re. 0 Tble..: The vlue of digonl element of digonl mtrix A Note: Uing modified MWG method, there re only predictor left in the tudy, therefore, mtrix A i 0 by 0 mtrix. IV(b: With the condition, [0,] Tble..: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion (OVENSG.0 Keep Tble..: Etimted regreion coefficient of finl model Predictor Regreion Coefficient (TOPDIAM -0.0 (LENGTH (MOIST -0.0 (TESTSG -0. (OVENSG 0. (RINGTOP 0. (RINGBUT 0. (BOWMA -0.0 (BOWDIST (WHORLS (CLEAR 0.00 (KNOTS 0.0 (DIAKNOT -0.0 Note: The etimted regreion coefficient of MWG method with [0,] tndrd devition cn t be clculted. cn t be obtined by uing formul. Therefore, the

25 Tble..: The vlue of digonl element of digonl mtrix A Dicuion of Dt Anlyi Bed on the reult preented in the bove tble, we obtined the following concluion:. The number of predictor With the condition, [0,], i le thn the number of predictor Without the condition, [0,]. b. Without the condition, [0,], nd by compring Tble.. with Tble..; Tble.. with Tble..0, we cn how tht the modified MWG method i more effective thn the originl MWG method, i.e., more predictor will be (Mnfield et l. (. c. Without the condition, [0,], Mnfield et l. ( thought tht deleting principl component correponding to mller Eigenvlue will reult in deleting more predictor. But Tble.. nd Tble.. didn t how the reult. d. The predictor by bckwrd method (, 0, nd re the me Ce III( (ee Tble... The etimted regreion coefficient re lo cloe (ee Tble.. nd Tble.... The Acetylene Dt n Exmple.. The Decription of Dt Thi dt concerning the percentge of converion of n-heptne to cetylene nd three originl explntory vrible (Dougl, C. M., Peck, E. A., nd Vining G.G. (00. The repone vrible i Y=percentge of converion, nd the three originl explntory vrible re =T ((contct time-0.00/0.0, =H (( H (n-heptne-./., =C ((Temperture-.0/0.. The predictor conidered in the principl component regreion model re: =T, =H, =C, =TxH (interction of T nd H, =TxC (interction of T nd C, =CxH (interction of C nd H, = T, = H, = C. The Correltion mtrix nd Y re given in Appendix D. --

26 In thi exmple, the ordered eigenvlue of re:.0,.,.,.0, 0., 0.0, 0.0, 0.00, The condition number of the correltion λmx mtrix i κ = =.0/0.000=0, it indicte there i evere λmin multicollinerity problem. The full qudrtic model for the cetylene dt i Y β o + β + β + β + β + β + β + β + β + β + ε (. = Tble.. give the vlue of etimted regreion coefficient, tndrd devition, t tet ttitic vlue, p-vlue, vlue for model (. nd etimted regreion R coefficient, tndrd devition for uing bckwrd elimintion method. Tble..: Summry Sttitic for model (. nd Bckwrd Elimintion Regreion on ll predictor Predictor Regreion Coefficient Stndrd Devition t-vlue p-vlue R Bckwrd Elimintion Regreion Coefficient Stndrd Devition Note: nd re. In tble.., there re even predictor with R >0. nd the other two R vlue re 0.0 nd 0., repectively. Therefore, mot of the (VIF vlue re quite lrge nd indicte evere multicollinerity problem. Bed on individul tet ttitic vlue nd p- vlue, only,, nd re ignificnt. But, the F ttitic vlue for model (. how ignificnt reult. Thi indicte tht ome predictor hould be. The um of reidul qure i 0. (for ll the principl component, for degree of freedom equl to, the tndrd error i

27 Ce I: Ue MWG Method Without Deleting Principl Component (: Without the condition, [0,], i.e., < < Tble..: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion 0 =C 0.0 Delete 0 =T 0. Delete 0 = H. Stop(Keep Tble..: Etimted regreion coefficient of finl model Predictor Regreion Coefficient Stndrd Devition =T 0 0. =H =C 0 0. =TxH =TxC..0 =CxH = T,. 0. = H = C -0.. Tble..: The vlue of digonl element of digonl mtrix A Ce I: Ue MWG Method Without Deleting Principl Component (b: With the condition, [0,] Tble..: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion 0 =C 0.00 Delete 0 = H. Stop(Keep --

28 Tble..: Etimted regreion coefficient of finl model Predictor Regreion Coefficient =T 0. =H -0. =C 0 =TxH -. =TxC.0 =CxH -.0 = T, 0. = H 0.0 = C -0. Note: The etimted regreion coefficient of MWG method with [0,] cn t be obtined by uing formul. Therefore, the tndrd devition cn t be clculted. Tble..: The vlue of digonl element of digonl mtrix A Ce II: Ue MWG Method nd Deleting Two Principl Component Correponding to the Two Smllet Eigenvlue II(: Without the condition, [0,], i.e., < < Tble..: Etimted regreion coefficient of finl model Step # of principl Predictor to be F-vlue Concluion =TxC 0.0 Delete =T 0.0 Delete = C.0 Delete = H. Stop(Keep --

29 Tble..: Etimted regreion coefficient of finl model Predictor Regreion Coefficient Stndrd Devition =T 0 0. =H =C =TxH =TxC =CxH = T, = H = C 0 0. Tble..0: The vlue of digonl element of digonl mtrix A Ce II: Ue MWG Method nd Deleting Two Principl Component Correponding to the Two Smllet Eigenvlue II(b: With the condition, [0,] Tble..: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion =TxC 0.00 Delete =T 0.00 Delete = H. Stop(Keep Tble..: Etimted regreion coefficient of finl model Predictor Regreion Coefficient =T 0.0 =H -0. =C -0.0 =TxH -. =TxC 0.0 =CxH -. = T, 0.0 = H 0. = C -0. Note: The etimted regreion coefficient of MWG method with [0,] cn t be obtined by uing formul. Therefore, the tndrd devition cn t be clculted.

30 Tble..: The vlue of digonl element of digonl mtrix A Ce III: Ue Modified MWG Method Without Deleting Principl Component III(: Without the condition, [0,], i.e., < < Tble..: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion 0 =C 0.0 Delete 0 = C 0.0 Delete 0 (OVENSG. Stop(Keep Tble..: Etimted regreion coefficient of finl model Predictor Regreion Coefficient Stndrd Devition =T =H =TxH =TxC. 0. =CxH = T = H Note: Uing modified MWG method, there re only predictor left in the tudy, therefore, mtrix A i by mtrix. Tble..: The vlue of digonl element of digonl mtrix A Ce III: Ue Modified MWG Method Without Deleting Principl Component III(b: With the condition, [0,] Tble..: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion 0 =C 0.00 Delete 0 = C 0.0 Delete 0 = T. Stop(Keep -0-

31 Tble..: Etimted regreion coefficient of finl model Predictor Regreion Coefficient =T 0. =H -0. =TxH -.00 =TxC. =CxH -. = T, 0. = H 0. Note: The etimted regreion coefficient of MWG method with [0,] cn t be obtined by uing formul. Therefore, the tndrd devition cn t be clculted. Tble..: The vlue of digonl element of digonl mtrix A Note: Uing modified MWG method, there re only predictor left in the tudy, therefore, mtrix A i by mtrix. Ce IV: Ue Modified MWG Method nd Deleting Two Principl Component Correponding to the Two Smllet Eigenvlue IV(: Without the condition, [0,], i.e., < < Tble..0: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion =TxC 0.0 Delete = T 0.00 Delete =TxH.0 Stop(Keep Tble..: Etimted regreion coefficient of finl model Predictor Regreion Coefficient Stndrd Devition =T =H =C =TxH =CxH = H = C

32 Tble..: The vlue of digonl element of digonl mtrix A Note: Uing modified MWG method, there re only predictor left in the tudy, therefore, mtrix A i by mtrix. Ce IV: Ue Modified MWG Method nd Deleting Two Principl Component Correponding to the Two Smllet Eigenvlue IV(b: With the condition, [0,] Tble..: The tep of deleting predictor Step # of principl Predictor to be F-vlue Concluion =TxC 0.00 Delete = T 0.00 Delete =TxH. Stop(Keep Tble..: Etimted regreion coefficient of finl model Predictor Regreion Coefficient =T 0. =H -0.0 =C -0.0 =TxH -0.0 =CxH -0. = H 0.0 = C 0. Note: The etimted regreion coefficient of MWG method with [0,] tndrd devition cn t be clculted. cn t be obtined by uing formul. Therefore, the Tble..: The vlue of digonl element of digonl mtrix A Note: Uing modified MWG method, there re only predictor left in the tudy, therefore, mtrix A i by mtrix... Dicuion of Dt Anlyi Bed on the reult preented in the bove tble, we obtined the following concluion: --

33 . The number of predictor With the condition, [0,], i le thn the number of predictor Without the condition, [0,]. Thi i the me reult for the dt in the Pitprop dt. b. In the Pitprop exmple, NO predictor re With the condition, [0,]. But, in thi exmple there re predictor being (ee Tble..,..,.., nd... One of the reon i the exiting evere multi-collinerity problem. c. Without the condition, [0,], We found the number of predictor by uing MWG modified method i le thn the ue of originl MWG method. Thi i contrdiction to the concluion in Mnfield et l. (. d. In the pper of Mnfield et l. ( they mentioned tht if we firt delete the principl component correponding to mller Eigenvlue, it will reult in deleting more predictor. We found tht i true no mtter whether we hve the retriction of [0,] or not. (See the comprion of Tble.. with..; Tble.. with..; Tble.. with..0, nd Tble.. with... e. The predictor by bckwrd method ( nd re the me Ce III( (ee Tble.... Concluion nd Suggetion When there exit evere multicollinerity problem, Ridge regreion i one of the mot populr etimtion procedure for combting multicollinerity. Bied etimtion i ued to ttin ubtntil reduction in vrince with n ccompnied incree in tbility of the regreion coefficient. After the dt nlyi of two exmple in ection, we hve the following concluion:. With the condition, [0,], the MWG method of deleting predictor becme more conervtive, i.e., the number of predictor i le thn the number without [0,]. b. Without the condition, [0,], the vlue of digonl element of digonl mtrix A my become very lrge. Lrger vlue of will incree the vrince of MWG etimtion nd ffect the preciion of etimtion reult. For exmple, In Tble.., 0 =0. nd =-.. c. In the firt exmple, there i moderte multicollinerity problem. If we dd the retriction of [0,], the procedure of deleting predictor becme conervtive. Since we hve evere multicollinerity problem in exmple, the effect of deleting predictor lmot the me between the retriction of [0,] nd < <. d. With the condition [0,], the evere multicollinerity problem, nd the mll mple ize (n=, the MSE i lrger nd ll thee mke the F-ttitic not ignificnt. Therefore, more predictor re. --

34 e. There re drwbck in the two exmple we ued to do dt nlye: ( the multicollinerity problem i not evere in exmple, therefore the effect of deleting predictor by dding the retriction of [0,] to MWG method i hrd to be utified. ( the mple ize i too mll in the econd exmple, mll mple ize incree the MSE nd decree the F-ttitic nd will ffect the number of predictor to be. f. It i time conuming to ue the modified MWG method becue we hve to reevlute the model t ech tep, i.e., reclculte the eigenvlue nd eigenvector; redefine the principl component, etc. The following re our uggetion:. In order to ue MWG method or the method propoed in thi rticle to delete predictor, we need lrge dt et with evere multicollinerity problem o tht we one cn build up mot pproprite model for dt nlyi. b. One hould deign rtificil exmple with lrge mple ize nd different degree of multicollinerity o tht we cn tudy MWG method nd our propoed method thoroughly. APPENDI A: The Derivtion of E( ~ γ nd V( ~ γ Since A γˆ - γˆ = A γˆ = ( V V V V γˆ - ( V V ~ γ = A V V γˆ it implie V γˆ =[- ( V V V ] γˆ. γˆ (Mnfield et l. (, we obtined Therefore, E( ~ γ =[- V ( V V V ]E( γˆ = [- V ( V V V ] γ. The expected vlue of ~ β i E( ~ β =E( V ~ γ = V [- V ( V V V ] γ, ince ' γˆ = ˆ γ, ˆ γ,..., ˆ γ = W ( Y, the vrince-covrince mtrix of γˆ i Vr( γˆ =Vr( W Y = W But, W=[ W : ] nd W W=, it implie Wk σ W W W= W ' ' ' W W WWk 0 [ W ] = = ' W k = ' ' W W=. W k W k W Wk Wk 0 k Therefore, Vr( γˆ =Vr( W Y = W W σ = σ. The vrince-covrince of ~ γ i Vr( ~ γ = σ [- V ( V V V ] [- V ( V V V ] = σ [ - ( V V V ][- ( V V V ] V V V V = σ [ - ( V V V - ( V V V + V V V V V V ( V V V ( V V V ] = σ [ - ( ]

35 Therefore, the vrince-covrince of ~ β = V ~ γ i Vr( ~ β = σ V [ - ( V V V ] V V APPENDI B: The Derivtion of (. The reidul vector of MWG etimtor i e ~ = Y - Y - ~ β =Y - Y - V nd the prediction vector i Y + V A A γˆ γˆ. To how the reidul vector i orthogonl to the prediction vector, it implie tht (Y - Y - ~ β (Y + V γˆ ' γˆ ' A A V '' Y - γˆ ' γˆ - γˆ A A V ''V A γˆ '( A - A A γˆ =0 APPENDI C: The proof of (. γˆ =0 A The minimiztion of (. cn be rewritten where = Min u r = = γˆ =0 ( λ ˆ γ uch tht ( λ γ =0 V V [ t ] r = = V tγˆ mtrix. ˆ A =0, t=k-r+, k-r+,,k γˆ =0. Since ( λ ˆ γ =0, ( λ ˆ γ cn be trnformed ( λ ˆ γ, the = = problem i then Mx u = λ γˆ = γˆ ' Let c= / r = / A / uch tht ( λ γ = γˆ ' / A (I- A = V ˆ t = = V γ A W A ˆ γˆ = V / A / γˆ, x= c, = V /, the problem cn be rewritten γˆ = / γˆ =0 γˆ =0, t=k-r+, k-r+,,k Mx u r =c x uch tht c x-x x=0 nd W x=0 Define L=c x+ ρ (c x-x x+ μ' W x, where ρ nd μ' re Lgrnge multiplier. Setting the derivtive with repect to x equl to zero give x=(c+ ρ c+ W μ /( ρ ( Setting the derivtive with repect to ρ equl to zero give c x-x x=0 ( --

36 Subtituting x into ( we obtined c [(c+ ρ c+ W μ /( ρ ]- {[(c+ ρ c+ W μ (c+ ρ c+ W μ ]/( ρ }=0 μ W ( ρ c c-c c- μ' W c-c W μ - ' W μ /( ρ =0 ( c c-c c-c W μ - ' W μ =0 ( ρ μ W Setting the derivtive with repect to μ equl to zero give obtined W [(c+ ρ c+ W μ /( ρ ]=0. Solve for μ, W x=0. Subtituting x we μ = ( W W ' ( Wc + ρwc ( Subtituting ( into (, we obtined ρ c c-c c-c [ ( W W ' W c + ρw ] [ ( W W ' W c + ρw ] W W ( c [ ( W W ' ( Wc + ρwc ]=0 W ( W W ' W ρ W ( c ρ c c-c c+c c - c ( W ' W c=0. Subtituting the W olution of ρ = into (, we obtined μ = - ( W W ' W c. Subtituting the olution of ρ nd μ into (, we obtined A c=[i - ( ]c W W Since = V /, c= A A / γˆ = / / γˆ = W W W W W ' W =c - ( W ' c=[i - W ( W ' W ]c / γˆ, it implie V γˆ - / ( V V / γˆ - / ( V V V / V / V γˆ W γˆ W Oberve the th element on both ide of the bove eqution one cn obtin the following eqution / / / ' λ ˆ γ = λ ˆ γ - λ v ( V V V γˆ ' =-{ [ V V v ] V γˆ / λ γˆ } =,,..., (. --

37 APPENDI D: Two Exmple Tble : The Correltion Mtrix of Pitprop Problem ** ** 0.** ** 0.** 0.** * 0.0** ** 0.** 0.* 0.** 0.** ** 0.0** * 0.** 0.** ** 0.** ** ** 0.** ** 0.** 0.** ** 0.** ** 0.** 0.** 0.** * ** ** 0.* -0.* ** - 0.** 0.** - 0.** * indicte ignificnce t the 0.0 of probbility. ** indicte ignificnce t the 0.0 of probbility. 0.** - 0.0** 0.** ** * The bove tble give the mtrix of of the independent vrible (predictor. Y=[ ] The predictor re: =TOPDIAM, =LENGTH, =MOIST, =TESTSG, =OVENSG, =RINGTOP, =RINGBUT, =BOWMA, =BOWDIST, 0 =WHORLS, =CLEAR, =KNOTS, =DIAKNOT Y=Compreive trength Tble : The Correltion Mtrix of Acetylene Problem The bove tble give the mtrix of of the independent vrible (predictor. Y=[ ] --

38 The predictor re: =T, =H, =C, =TxH (interction of T nd H, =TxC (interction of T nd C, =CxH (interction of C nd H, = T, = H, = C. Y= percentge of converion REFERENCES. Mon, Gunt nd Webter [], Regreion Anlyi nd Problem of Multicollinerity, Communiction Sttitic., (, p. -.. Mnfield, Webter nd Gunt [], An Anlytic Vrible Selection Technique for Principl Component Regreion Appl Stt., (, p.-0.. Grybill, F. A. (. Theory nd Appliction of the Liner Model. Boton, Mchuett:Duxbury Pre.. Jeffer [], Two Ce Studie in the Appliction of Principl Component Anlyi Appl Stt.,, p. -.. Hoerl nd Kennrd [0(], Ridge Regreion: Bied Etimtion for Nonorthgonl Problem Technometric.,, p. -.. Hoerl nd Kennrd [0(b], Ridge Regreion Appliction to Nonorthgonl Problem Technometric.,, p. -.. G.A.F. Seber [], Liner Regreion Anlyi, Wiley, New York.. Dougl C, Montgomery Elizbeth A, Peck C Ceoffery Vining [00], Introduction to Liner Regreion Anlyi, rd ed., Wiley, New York.. John O. Rwling, Stry G. Pntul, Dvid A. Dickey [], Applied Regreion Anlyi A Reerch Tool, nd ed., Springer, New York. 0. Richrd A. Johnon, Den W. Wichern [], Applied Multivrite Sttiticl Anlyi, Prentice Hll.. Li-Chun Wng (0, Generlized Shrunken Let Squre Etimtor, Applied Probbility nd Sttitic (in Chinee, Vol. (, p -. --