PENALIZED LEAST SQUARES AND PENALIZED LIKELIHOOD

Transcription

1 PENALIZED LEAST SQUARES AND PENALIZED LIKELIHOOD HAN XIAO 1. Penalized Leas Squares Lasso solves he following opimizaion problem, ˆβ lasso = arg max β R p+1 1 N y i β 0 N x ij β j β j (1.1) for some 0. We can use some oher penaly on he parameers β j as well, and consider he following general penalized leas squares problem ˆβ = arg max β R p+1 1 N y i β 0 x ij β j p ( β j ) N, (1.) where p ( ) is he penaly funcion. Bes subse selecion corresponds o p () = ( /)I( 0). If we ake p () =, hen (1.) becomes he Lasso problem (1.1). Seing p () = a + (1 a) wih 0 a 1 resuls in he mehod of elasic ne. Wih p () = q for some 0 < q, i is called bridge regression, which includes he ridge regression as a special case when q =. Some penaly funcions and heir derivaives are shown in Figure 1. Usually he inercep is no penalized, so in he res of his secion, we cener y and x j firs and consider he following opimizaion problem from now on ˆβ = arg max β R p 1 N N y i x ij β j p ( β j ), (1.3) wih he implici assumpion ha 1 x j = 0, 1 j p and 1 y = 0. To have more insighs on he effecs of differen penaly funcions, we consider he canonical regression model which assumes he columns of he inpu marix N 1/ X are orhonormal, i.e. X ls X/N = I. Under his model, denoing ˆβ j = N 1 x j y and ŷ ls = XX y, he penalized leas squares (1.3) can be rewrien as ˆβ = arg max β R p 1 [ y ŷ ls + 1 ( ) ] ˆβls N j β j p ( β j ), (1.4) so ha he opimizaion problem becomes maximizing for each β j, { 1 ( ) } ˆβ j = arg max ˆβls β j R j β j + p ( β j ). (1.5) In paricular, Lasso esimaes are ˆβ lasso j The soluion of (1.5) is he hard-hresholding esimaor ˆβ j = ls ls = sign( ˆβ j )( ˆβ j ) +. (1.6) ls ls ˆβ j I( ˆβ j > ), if p () = 1 [ ( ) +]. This is a supplemenary reading for FSRM588. Las updaed on Sepember 30, 01. If you are ineresed in hese opics, you may wan o read he review paper Fan and Lv (010). Do no reproduce or disribue he lecure noes. Unauhorized reproducion or disribuion of he conens of his noes is a copyrigh violaion. 1

2 Penaly Funcions Derivaives penaly L1 Hard derivaive L1 Hard Figure 1. L 1, and hard hresholding penaly funcions (lef panel) and heir derivaives (righ panel). The parameers are chosen as = 1.04 for L 1 penaly, = 1.0 and a = 3.7 for ; and = for hard hresholding. Le us consider a simplified case of (1.4). Assume we have N raining poins (x 1, y 1 ),..., (x N, y N ) from he following model y i = β 1 x i1 + β x i + ɛ i, wih β 1 > 0 and β = 0, and ɛ i are i.i.d. wih mean zero and variance σ. Here we have wo predicors, and one of hem is no useful for predicion. Recall ha we have assumed X X/N = I. If we were o do variable selecion and esimaion simulaneously, he following wo properies are desired. (1) As N goes o infiniy, β is esimaed as zero wih probabiliy approaching one. () β 1 can be esimaed wih small bias. More specifically, we would like he esimae ˆβ 1 o have he propery N( ˆβ 1 β 1 ) N (0, σ ). A firs glance, i seems from (1.6) ha Lasso esimae is biased even if i esimaes β 1 as nonzero. Bu you may wan o argue ha Lasso can give a unbiased esimae for β 1 if we le = N o depend on N, and le i decreases o zero as N approaches infiniy. Le us ake a close look o see wheher i is possible for Lasso o achieve boh (1) and (). Rewrie N = N 1/ η N. From he ideniy Lasso P ( ˆβ = 0) = P ( ) ls ˆβ N 1/ η N = P ( ) ls N ˆβ η N, we see ha in order for his probabiliy o approaching one, we mus le he sequence η N diverge as N grows. On he oher hand, ( ) N ˆβlasso 1 β 1 ( ) N ˆβls 1 N β 1 = ( ) N ˆβls 1 β 1 η N, from which we find ha in order for () o hold, he sequence η N mus converge o zero. This simple example ells us ha Lasso canno achieve boh (1) and () a he same ime Fan and Li (001) inroduced he smoohly clipped absolue deviaion (), whose derivaive is given by { p () = I( ) + (a ) } + I( > ), (1.7) (a 1)

3 where p = 0 and a >. Ofen a = 3.7 is used. Graphs of and is derivaives are shown in Figure 1. The soluion of (1.5) is given by sign(βj ls)( βls j ) +, when βj ls ; ˆβ j scad = {(a 1)βj ls sign(βls j )a}/(a ), when < βls j a; (1.8) βj ls, when βls j > a. Exercise 1.1. (a) Show ha he soluion of (1.5) is given by (1.8). (b) For he example given in Secion 1, show ha i is possible o choose N o achieve boh (1) and () when N goes o infiniy. 1.. Oracle propery. We have seen ha is able o realize boh (1) and () for a very special case (1.4). This is acually also rue generally for (1.3). Theorem 1.1 (Fan and Li (001)). Assume we have N raining poins (x 1, y 1 ),..., (x N, y N ) generaed from y i = x iβ + ɛ i ; (1.9) where (x i, ɛ i ), 1 i N are i.i.d.; x i have mean zero and covariance marix Σ; ɛ i have mean zero and variance σ ; x i and ɛ i are independen; and he join disribuion of (x i, ɛ i ) saisfies some regulariy condiions. Suppose he rue parameer vecor β = ( β (1) β () ) consis of a sub-vecor β(1) whose componens are all nonzero, and anoher sub-vecor β () = 0. If N 0 and N N as N, hen (i) wih probabiliy ending o one, ˆβ scad (ii) he asympoic normaliy holds for () = 0; ˆβ scad (1) as N ( ˆβscad (1) β (1) ) N (0, σ Σ 1 (1) ). Even if we know ha β () = 0 ahead of ime and perform he leas squares only for β (1), he asympoic scad disribuion of he esimae will be he same wih ha for ˆβ (1) given in (ii) of Theorem 1.1. So he message from Theorem 1.1 is ha alhough we do no know he ruh, bu he performance is as well as if we knew he ruh. This is referred o as he oracle propery in he lieraure Local linear approximaion. is nice in he sense ha i has he oracle propery, bu i also has drawback in compuaion. The cos funcion (1.3) wih penaly is non-concave so he opimizaion is in general challenging. Zou and Li (008) proposed a unified algorihm based on he local linear approximaion (LLA) for maximizing he penalized likelihood for a broad class of concave penaly funcions. We firs inroduce heir algorihm for he problem (1.3). Suppose we have some iniial esimae ˆβ ; for example, we may ake ˆβ ls = ˆβ be he leas squares esimae. The penaly funcion p ( β j ) can be locally approximaed around ˆβ by a linear funcion p ( β j ) = p ( ˆβ j ) + p ( ˆβ j )( β j ˆβ j ), for β j ˆβ j. (1.10) Figure illusrae he LLA for. Wih his approximaion, he problem (1.3) becomes ˆβ (1) = arg max β R p 1 N y i x ij β j p ( ˆβ j ) β j N, (1.11) which can be solved using Lasso. More specifically, le us illusrae using he LARS algorihm. Le I 1 = {j : p ( ˆβ j ) = 0} and I = {j : p ( ˆβ j ) 0}. Le X 1 be he sub-marix of X consising of columns indexed by I 1, and define X similarly. Le P 1 be he projecion marix o he column space of X 1. Sep 1. For j I, define x j = p ( ˆβ j ) x j, y = (I P 1 )y, and le X be he marix wih columns {x j : j I } and X = (I P 1 )X. 3

4 penaly penaly Figure. Local linear approximaion for wih = and a = 3.7. The lef panel is an approximaion a = 4, and he righ panel is a = 1. Sep. Apply he LARS algorihm o solve ˆβ = arg max β { 1 N y X β β 1 }. Sep 3. Compue ˆβ = (X 1X 1 ) 1 X 1(y X ˆβ ). Sep 4. We use I 1 o index he componens of β, and I o index he componens of β. The final esimae of (1.11) is given by ˆβ (1) j = { ˆβ j when j I 1 ; ˆβ j when j I. p ( ˆβ j ) The preceding algorihm can be ieraed unil convergence. Bu Zou and Li (008) poined ou ha esimaes produced by one sep LLA already have he oracle propery. Figure 3 illusraes soluion pahs of problem (1.5) using differen algorihms. Example 1.1. For he Boson housing daa. The soluion profiles given by LLA are depiced in Figure 4. The R code is given in a separae file.. Penalized Likelihood If we assume ɛ i in (1.9) are i.i.d. N(0, σ ), hen (σ ) 1 N (y i x i β) is he logarihm of he condiional likelihood of y give X, and hence (1.) can also be viewed as penalized likelihood. In general, he penalized likelihood funcion akes he form Q(β) = 1 N l i (β) p ( β j ), (.1) N In he lars package, is used as 1 RSS+ β 1 1. The plus package implemens Lasso wih being used as N RSS+ β 1. If he normalize argumen is se as TRUE, in lars each column is sandardized o have a uni lengh; while in plus, each column is normalized o have a lengh N. Using and η o denoe he uning parameers in he original and normalized models respecively, we have he following correspondence: plus = lars /N and η plus = η lars / N. If inercep and normalize are boh se as TRUE, boh packages perform cenering before normalizing. 4

5 Esimaes LLA Lasso Esimaes LLA Lasso z Figure 3. Soluion pahs of (1.5). soluions are given boh exacly and approximaely using LLA. In he lef panel, he LS coefficien is fixed a 1, and varies. In he ls righ panel, is fixed a 1, and he LS coefficien (denoed here by z insead of ˆβ j ) varies. where l i (β) := l i (x i β, y i, φ) is he log likelihood of he i-h raining poin (x i, y i ), wih φ being some dispersion parameer. Le l(β) = N l i(β). For a given iniial value ˆβ (e.g. MLE), he log likelihood funcion can be locally approximaed by a quadraic funcion l(β) l( ˆβ ) + l( ˆβ )(β ˆβ ) + 1 (β ˆβ ) l( ˆβ )(β ˆβ ). (.) A he MLE ˆβ, he gradien l( ˆβ ) = 0, and hence he LLA esimae is given by ˆβ (1) 1 = arg max β R p (β ˆβ ) l( ˆβ )(β ˆβ ) p ( ˆβ j ) β j. (.3) Wrie µ i = x i β and l i = l i (µ i, y i ), hen he Hessian marix can be wrien as where D is a N N diagonal marix wih D ii = l i (µ i ) µ i l( ˆβ ) = X DX, (.4) ˆµ i, ˆµ i = x i ˆβ. We see ha (.3) can also be solved using LARS algorihm. Deails are omied. References Jianqing Fan and Runze Li. Variable selecion via nonconcave penalized likelihood and is oracle properies. J. Amer. Sais. Assoc., 96(456): , 001. Jianqing Fan and Jinchi Lv. A selecive overview of variable selecion in high dimensional feaure space. Sais. Sinica, 0(1): , 010. Hui Zou and Runze Li. One-sep sparse esimaes in nonconcave penalized likelihood models. Ann. Sais., 36(4): ,

6 Coefficiens Coefficiens Figure 4. Profiles of coefficiens given by one sep LLA (op panel) and ieraive LLA (boom panel) for he Boson housing daa, as he uning parameer is varied. The original variable NOX is muliplied by 5 o resul in a beer plo. 6