4.1 Other Interpretations of Ridge Regression

Size: px

Start display at page:

Download "4.1 Other Interpretations of Ridge Regression"

Mark Cox
5 years ago
Views:

1 CHAPTER 4 FURTHER RIDGE THEORY 4. Oher Inerpreaions of Ridge Regression In his secion we will presen hree inerpreaions for he use of ridge regression. The firs one is analogous o Hoerl and Kennard reasoning while he second one is based on a Bayesian approach. In addiion, in recen lieraure one new characerizaion for ridge regression is presened based on an opimizaion problem. 4.. Resriced Leas Squares Inerpreaion Ridge regression may be viewed as leas squares subjec o a spherical resricion on he parameers. Suppose ha he regression problem under sudy is in correlaion form and ha we perform leas squares subjec o he spherical resricion β β c, (4.. c is a specified value. A resriced leas squares esimaor can be esimaed by subjec o he consrain (4... Using he mehod of minimizing ( Y β ( Y β Lagrange mulipliers, we can form Seing F β 0 gives he equaions ( β ( Y β + k( β c F Y β, (4.. ( + I β Y which is he ridge soluion. (Vinod and Ullah, 98. k, (

2 4.. Bayesian Inerpreaion The Bayesian approach o ridge regression is based on he assumpion ha we have a regression siuaion ~ ( β, Iσ Consider he case he individual regression coefficiens in ( β,..., Y N. (4..4 β are exchangeable ( an assumpion ha may no be appropriae as emphasized by Lindley and Smih, 97 i.e. hey are unalered by a permuaion of he suffixes (i,,, p. Suppose furher ha (, σ β j ~ N ξ β. (4..5 If we suppose vague prior knowledge for ξ, hen he Bayes esimae is β { I + ( ( I p J } βˆ p p p k, (4..6 β p k σ σ β and J is a marix of ones. If we assume ξ 0, and hus imply ha β s are small hen he Bayes esimae is given by i β { I + ( } βˆ p k. (4..7 When σ, he residual regression variance, and σ β, he variance of he regression coefficiens are boh unknown we can esimae hem and calculae k as follows: k s s β. In he esimaes above k is a variance raio and is esimaed from he daa while in Hoerl and Kennard s argumen k is he consan he regression esimaes sabilize. Like ridge mehod he Bayesian mehod aemps o avoid some of he problems caused by non-orhogonaliy in he daa bu in addiion i has he advanage of dispensing wih he raher arbirary choice of k and allows daa o esimae i (Lindley and Smih,

3 4..3 An Opimizaion Problem J is a Consider linear esimaors ha can be wrien as B JR, B 0 p p marix, B 0 is he ordinary LS esimaor and R (he correlaion marix. Since B is a linear ransform of B 0, i is a biased esimaor unless J R. We have ( B JR β E. From (3.5. i can be shown ha MSE( B D( B + σ r( JR J, D ( B is he squared bias erm of B and is equal o ( JR I β. Ridge regression is a biased esimaion mehod based on linear esimaors. Qannari e al. (997 presen an opimizaion problem, which leads o he ridge esimaor bu from anoher viewpoin. They sugges keeping he oal variance of he parameer esimaes a an accepable level, whiling allowing he smalles possible bias. Consider he inequaliy ha holds for he Euclidean norm of a marix ( B 0 D JR I β (i D ( B is zero when, i seems ha J R, (ii or approaches zero when JR I approaches zero. Therefore he auhors, as explained earlier, sugges minimizing he bias, i.e. min JR J I, under he consrain ha he oal variance is fixed, i.e. r( c c is a fixed posiive scalar. Solving he Lagrangian problem we obain ( R + k J I, which is he ridge esimaor (Qannari e al., 997. JR J, 4. Applicaion of Ridge Regression in Special Cases In chaper 3 we only consider he use of ridge regression in he mulivariae linear regression model. However, many auhors have used ridge regression in differen cases, 67

4 for example, in logisic regression. We will discuss some cases which we consider raher useful. 4.. Rank deficien model Le us consider he case our model is rank deficien. Brown (978 examines he ridge esimaor in he conex of a linear model, which may be rank deficien ( is an ( p esimaor, ( + I Y T given marix of rank m ( p. In such a case he ridge k is no defined a k 0, so Brown (978 suggess he following definiion. Le β ˆ ( k ( + ki + Y Y for for k > 0 k 0 + denoes he Moore-Penrose pseudoinverse (Appendix A. 4.. Sraigh line regression wih a small number of οbservaions Carmer and Hsieh (978 ry o apply biased echniques o sraigh line regression wih a small number of observaions. Having Y and in sandardized form leads o a LS esimae equal o he simple correlaion rˆ, beween and Y; he regression sum of squares is equal o ˆr and he residual mean square is ˆ σ ( rˆ ( T, T is he number of observaions. ~ β. The biased esimae of he sandardized regression coefficien is ~ r rˆ ( + k Farebroher (in Carmer and Hsieh, 978 proposed for an esimae of k he following: k ˆ σ rˆ ( rˆ ( T rˆ The resuls of he simulaion sudy of he auhors showed ha none of he biased procedures are recommended for use in sraigh line regression problems wih a small number of observaions. According o he auhors all he procedures raher severely 68

5 reduced he esimae of he slope, relaive o leas squares, and none of he procedures produced dramaic improvemens in he mean square error Models wih lagged effecs In models wih lagged effecs we have α + β0 + β βll U ;,,..., n Y + Y is a dependen variable, (4.. represens he marix of regressors and U he random error. As we can noice from (4.. he regressors involve ime series which are ofen auocorrelaed. So using ridge regression paricularly for large values of l is a way o ackle his problem. However, a problem of lagged effecs model is o selec an appropriae number of lagged erms, i.e. he righ l. Erickson (98 deals wih he opic of variable selecion uilizing ridge regression. In order o selec variables he minimizes a predicion error, or a leas an esimae of he predicion error based on ridge regression- Ridge Regression Predicion crierion (RP. RP depends on which observaions and regressors are used and on he value of k- he ridge consan. Using ridge regression on some daa he auhor shows ha in order o find he righ esimaes for he number of lagged erms one should firs calculae a k ha minimizes he RP crierion for each value of l and hen find he overall minimum of l s Subse selecion The ridge regression has also been used as a subse selecion echnique by Hoerl e al. (986. They propose a ridge selecion mehod ha examines a full ridge soluion and hen delees erms ha are no significan. The deleion of he erms is based on a βˆ k S, modified -es, ( Ri σ ( + ki ( + ki : E( ˆ( k 0 H β. 0 S Ri is he i h diagonal elemen of. This means ha we are acually esing he hypohesis 69

6 4..5 Logisic regression Consider he logisic regression model: (, 0 p π (4.. β ( + e β β β,..., β and π is he probabiliy ha he even Y occurs, π P( Y. The unknown parameer vecor β can be esimaed by βˆ, he maximum likelihood esimaor (MLE of β. Schaefer e al (in Lee and Silvapulle, 988 have derived he ridge esimaor for he logisic regression model as V ˆ V ( βˆ { βˆ ( k } MSE{ βˆ } ( ( V ˆ + ki ( V ˆ β ˆ βˆ k,. They have also shown ha if he degree of mulicollineariy is high hen MSE < for many observaions and small value of k. Lee and Silvapulle (988 propose a mehod for he deerminaion of k using Bayesian mehods. They obained he following wo choices of k: ˆ ( π + ( βˆ βˆ a ( cov( βˆ βˆ ( cov( βˆ k, (4..3 k ˆ b r β. (4..4 Afer a Mone Carlo sudy for he examinaion of he performance of he above esimaors he auhors concluded ha kˆ a is considered he bes choice for k Auocorrelaed disurbances Firinguei (989 sudies he effec of collineariy and auocorrelaed disurbances in he performance of several ridge regression esimaors. The use of ridge regression in generalized linear models has been considered by oher auhors oo. Ye i had only been discussed in cases he error variance-covariance marix ( Ω σ was known. Firinguei suggess ha even when one has o esimae k and Ω, condiions can be found he ordinary ridge regression esimaor dominaes he generalized leas squares (GLS esimaor. 70

7 Consider he model Y β + U as described in (.., U is a vecor of T disurbances such ha u ρ + ε, ρ <,,,..., T (4..5 u and ( 0, ε ~ N σ, ( 0 E ε ε for each,. (4..6 The GLS esimaor b Ω Ω, (4..7 ( Y T ρ... ρ T ρ... ρ Ω (4..8 ρ.... T T ρ ρ... is he minimum variance unbiased esimaor. Since in pracice ρ is usually unknown i is esimaed by ρˆ T e e T esimaor of β becomes e e Y βˆ, he OLS residuals. Then he GLS ( Ωˆ Ωˆ Y bˆ. (4..9 In case when collineariy is presen in a GLRM, he auhor suggesed considering a generalized version of some well-known ridge esimaors. For example, he generalized Hoerl, Kennard and Baldwin RR (GHKB esimaor is: wih bˆ ( ( Ωˆ I Ωˆ k + k Y ( Ωˆ I Ωˆ + k bˆ ( Y bˆ Ωˆ ( Y bˆ k and s ps b ˆ b ˆ ( n p One can also define he generalized Lawless and Wang RR (GLWR esimaors as bˆ ( ( Ωˆ I Ωˆ k + k Y. 7

8 ( Ωˆ I Ωˆ + bˆ k ps wih k. bˆ Ωˆ bˆ Comparing he differen esimaors using MSE and absolue bias he auhor suggess ha in he presence of mulicollineariy and auocorrelaion he generalized ridge regression esimaors can perform beer han he oher mehods. 4.3 A Recen Advance in Ridge Regression I is no unusual o have collineariy and influenial cases simulaneously in a daa se. Walker and Birch (988 discuss abou he effec ha collineariy can have on he influence of any given case and propose some influence measures in case we use ridge regression. Par C in he Appendix provides a brief overview of influence analysis Influence in Ridge Regression denoed as Using a differen noaion for convenience he ridge esimaor of β is now The ridge residuals are defined as ( + I Y b k, (4.3. e Y b. In order o measure he influence of a single case a version of DFFITS (difference in fi sandardized for RR can be used, namely ( DFFITS ( i ( b b ( i x, i SE( xib b i is he ridge esimaor of β wihou he i h case, ( x i b SE is an esimaor of he sandard error (SE of he fied value wihou he i h case and x i is he i h row of marix. The auhors also define wo versions of Cook s disance D i 7

9 D i ( b b ( i b b ( i ( or ps D i ( ˆ ˆ Y Y ( i Yˆ Yˆ ( i ps ( and D i ( b b ( i ( + ki ( ( + ki b b ( i ps (. For choosing he value of k he auhors sugges he value of k ha minimizes he following quaniy C ( SSR s T r( H, (4.3. k k + SSR is he sum of squares of residuals from RR and ( + I k H k. As one can conclude from he definiions of DFFITS and Cook s disance, he influence of each case is a funcion of he ridge parameer k. I is ineresing o noe ha while he influence of some cases decreases he influence of some ohers increases. Thus, he auhors advise o deermine he value of k and hen compue he influence measures for ha k. If i is necessary o delee cerain cases, he process described should be repeaed Local change of small perurbaions Shi and Wang (999 presened anoher approach in order o measure he influence of observaions on he ridge esimaor. Insead of examining he influence of case deleion hey perform local influence analysis. In local influence analysis we ry o esimae he local change of small perurbaions on he variance or on he explanaory variables. The funcions used o esimae hese changes are he generalized influence funcion (GIF and he generalized Cook saisic (GC Perurbing he variance The variance of he errors becomes of ( ω ω,...,ω n σ W W diag( ω. The perurbed version of he ridge esimaor is The generalized influence funcion of ( ( W + ki WY wih diagonal elemens b ω. (4.3.3 b under he perurbaion is given by 73

10 ( b, l ( + ki D( e l GIF, D( diag( e Again wo versions of he generalized Cook saisic of and ( b l l D( e HD( e l e and l is a uni-lengh vecor. i b can be defined GC, ps, (4.3.4 ( b l l D( e H D( e l GC, ps, (4.3.5 H + k and H is he ha marix of LS regression. ( I Perurbing he explanaory variables Similar influence measures can be defined when we have perurbaion of he explanaory variables. Finally, recall he quaniy (4.3. and consider he perurbaion of he variance. Le C ( ω, SSR ( ω and ( ω k respecively. Then k C H denoe he perurbed versions of C k, ( ( ω SSR ( ω s T r H ( ω k k + SSR k and H,. (

R t. C t P t. + u t. C t = αp t + βr t + v t. + β + w t

R t. C t P t. + u t. C t = αp t + βr t + v t. + β + w t Exercise 7 C P = α + β R P + u C = αp + βr + v (a) (b) C R = α P R + β + w (c) Assumpions abou he disurbances u, v, w : Classical assumions on he disurbance of one of he equaions, eg. on (b): E(v v s P,