Technical Proofs for Homogeneity Pursuit

Transcription

1 Techical Proofs for Homogeeity Pursuit bstract This is the supplemetal material for the article Homogeeity Pursuit, submitted for publicatio i Joural of the merica Statistical ssociatio. B Proofs B. Proof of Theorem 3.5 Sice τ is cosistet with groups i β 0, there exists = j < j < < j K+ = p+ such that k = {τ(j k ), τ(j k + ),, τ(j k+ )} for all k. We shall write τ(j) = j without loss of geerality. I the first part of the proof, we show that β M, ad it satisfies the sig restrictios sg( β,k+ β,k ) = sg(β,k+ 0 β0,k ), k =,, K. Whe ρ(t) = t, Q (β) is strictly covex. So β is the uique global miimizer if ad oly if it satisfies the first-order coditios: xt ε + xt X( β β 0 ) λ sg( β β ), 0 = xt j ε + xt j X( β β 0 ) + λ sg( β j β j ) λ sg( β j+ β j ), xt p ε + xt p X( β β 0 ) + λ sg( β p β p ), j p where sg(t) = whe t > 0, whe t < 0, ad ay value i [, ] whe t = 0. Therefore, it suffices to show that there exists β M that satisfies the sig restrictios ad the first-order coditios simultaeously. For β M, we write µ = T ( β) ad µ 0 = T (β 0 ), where the mappig T is the same as that i the proof of Theorem 3.. The sig restrictios ow become sg( µ k+ µ k ) = sg(µ 0 k+ µ0 k ) for all k =,, K. Note that β j = β j+ whe predictors j ad

2 (j + ) belog to the same group i. The first-order coditios ca be re-expressed as xt j ε + xt j X ( µ µ 0 ) + λ sg( µ k µ k ) λ r j, j = j k 0 = xt j ε + xt j X ( µ µ 0 ) + λ r j λ sg( µ k+ µ k ), j = j k+ () xt j ε + xt j X ( µ µ 0 ) + λ r j λ r j, elsewhere, where r j s take ay values o [, ] ad we set sg( µ µ 0 ) = sg( µ K+ µ K ) = 0 by default. Deote by δk 0 = sg(µ0 k+ µ0 k ) whe k K ad δ0 k = 0 whe k = 0, K; similarly, δ k for k K. I (), we first remove r j s by summig up the equatios correspodig to idices i each k. Usig the fact that x,k = j k x j, we obtai xt,k ε + xt,k X ( µ µ 0 ) + λ δk λ δk = 0, k =,, K. Uder the sig restrictios δ k = δk 0, k =,, K, it becomes a pure liear equatio of ( µ µ 0 ): XT ε + XT X ( µ µ 0 ) λ d 0 = 0, where d 0 is the K-dimesioal vector with d 0 k = δ0 k δ0 k, as defied i Sectio 3.4. It follows immediately that µ µ 0 = λ (X T X ) d 0 + (X T X ) X T ε. () Secod, give ( µ µ 0 ), () ca be viewed as equatios of r j s ad we ca solve them directly. Deote ξ = XT X ( µ µ 0 ) XT ε. For each j k, defie kj = {j k,, j} ad kj = {j +,, j k+ }. The solutios of () are r j = δ k + λ ξ i = δ k λ ξ i, j k, j j k+. i kj i kj Here the two expressios of r j are equivalet because λ i k ξ i = δ k δ k from (). It follows that ay covex combiatio of the two expressios is also a equivalet expressio of r j. Takig the combiatio coefficiets as kj / k ad kj / k, ad pluggig i the sig restrictios δ k = δk 0, k =,, K, we obtai ( r j = λ kj k i kj ξ i kj k i kj ( = λ kj w j (ξ) + k δ0 k + kj ) k δ0 k, ) ( kj ξ i + k δ0 k + kj ) k δ0 k

3 where the fuctio w j ( ) is defied as i (36). Here r j s still deped o ( µ µ 0 ) through ξ. Combiig () to the defiitio of ξ gives ξ = [ XT I X (X T X ) X T ] ε + λ X T X (X T X ) d 0 XT P ε + λ b 0, where P = I X (X T X ) X T ad b0 is defied as i Sectio 3.4. By pluggig i the expressio of ξ, we ca remove the depedece o ( µ µ 0 ) of the solutios r j s: ( r j = λ w j (X T P ε) + w j (b 0 kj ) + k δ0 k + kj ) k δ0 k. (3) Now, to show the existece of β M that satisfies both the sig restrictios ad first-order coditios, it suffices to show with probability at least ɛ 0 K ( p), (a) the r j s i (3) take values o [, ]; (b) the µ i () satisfy the sig restrictios, i.e., sg( µ k+ µ k ) = sg(µ 0 k+ µ0 k ) for all k =,, K. Cosider (a) first. I (3), by Coditio 3.4, the sum of the last two terms is bouded by ( ω ) i magitude. To deal with the first term, recall that i derivig (38), we write w j (X T ε) = a T j ε. It follows immediately that w j(x T P ε) = a T j P ε = ( P a j ) T ε. Sice P a j a j, similarly to (38), we obtai max j k w j (X T P ε) C σ k k log( p)/, k K, except for a probability at most ( p). Therefore, by the choice of λ i (mai-8), the absolute value of the first term is much smaller tha ω. So max j r j except for a probability at most ( p), i.e., (a) holds. Next, cosider (b). Sice µ 0 k+ µ0 k b, it suffices to show that µ µ 0 < b. Note that () ca be rewritte as µ µ 0 = D ( D X T X D ) (λ D d 0 + D X T ε). It follows from Coditio 3. that µ µ 0 c (λ D d 0 + D D X T ε ). First, D d 0 4 K k= k. Secod, from (6), D X T ε C K log(), except a probability of at most K. Moreover, D = (mi k k ) /. These together imply µ µ 0 Cλ ( K k= ) / K log() + C k. 3

4 From (mai-8), the right had side is much smaller tha b. It follows that µ µ 0 b. This proves (b). I the secod part of the proof, we derive the covergece rate of β β 0. Note that β β 0 = D( µ µ 0 ), ad from (), D( µ µ 0 ) = ( D X T X D ) ( λ D d 0 + D X T ε ). Therefore, β β 0 c (λ D d 0 + D X T ε ), where D d 0 4 K k= ad D X T ε = O p( K) by (4). Combiig these gives β β 0 = O p ( K/ + λ ( k ) ) / k. k B. useful propositio ad its proof The requiremet that Υ preserves the order of β 0 implies restrictios o how much the orderig (i terms of icreasig values) of coordiates i β deviates from that of β 0. This is reflected o how the segmets {B,, B L } itersect with the true groups {,, K }. Recall that V kl = k B l. We have the followig propositio: Propositio B.. Whe Υ preserves the order of β 0, for each k, there exist d k ad u k such that k = dk l u k V kl, ad V kl = B l for d k < l < u k. For each l, there exist a l ad b l such that B l = al k b l V kl, ad V kl = k for a l < k < b l. Propositio B. idicates that there are two cases for each k : either k is cotaied i a sigle B l or it is cotaied i some cosecutive B l s where except the first ad last oes, all the other B l s are fully occupied by k. Similarly, there are two cases for each B l : either it is cotaied i a sigle k or it is cotaied i some cosecutive k s where except the first ad last oes, all the other k s are fully occupied by B l. Proof. Cosider the first claim. Give k, let d k = mi{l : V kl } ad u k = max{l : V kl }. The k = u k l=d k V kl. Moreover, for ay d k < l < u k, β 0,k max i B dk β 0 i mi j B l β 0 j max j B l β 0 j mi i B uk β 0 i β 0,k, where the first ad last iequalities are because k B dk ad k B uk, ad the iequalities i betwee come from Defiitio.3. It follows that β 0 j = β0,k for all j B l. This meas B l k, ad hece V kl = B l. 4

5 Cosider the secod claim. Give l, let a l = mi{k : V kl } ad b l = max{k : V kl }, ad hece, B l = b l k=a l V kl. For ay a l < k < b l ad l < l, max βi 0 mi βi 0 β,a 0 i B l i B l < β,k 0, l where the first iequality comes from Defiitio.3, the secod iequality is because al B l ad the last iequality is from β 0, < β0, < < β0,k ad a l < k. It follows that B l k =. Similarly, for ay l > l, B l k =. s a result, k B l ad V kl = k. B.3 Proof of Theorem 4. Recall the mappigs T, T ad T defied i the proof of Theorem 3.. Write Q (β) = L (β) + P (β), where L (β) = y Xβ ad P (β) = P Υ,λ,λ (β). For ay µ R K, let ad defie Q (µ) = L (µ) + P (µ). L (µ) = L (T (µ)), P (µ) = P (T (µ)), We oly eed to show that β oracle is a strictly local miimizer of Q with probability at least ɛ 0 K ( p). Let E be the evet that the segmetatio Υ preserves the order of β 0, ad defie the evet E ad B, a eighborhood of β 0, the same as i the proof of Theorem 3.. Recall the statemets (a) ad (b) i the proof of Theorem 3.. For a evet E 3 to be defied such that P ((E 3 )c ) ( p), we shall show that (a) ad (b) hold o the evet E E E 3. The coclusio the follows immediately. Cosider (a) first. Same as before, it suffices to show (9). Recall that V kl = k B l. Defie m,kk = L l= (V klv k (l+) + V k lv k(l+) ) ad m,kk = L l= V klv k l, for k < k K. Write for short ρ ( ) = ρ λ ( ) ad ρ ( ) = ρ λ ( ). It follows that P (µ) = λ k<k K Therefore, it suffices to check m,kk ρ (µ k µ k ) + λ k<k K m,kk ρ (µ k µ k ). mi µ k k k µ k > a max{λ, λ }, for ay β B, µ = T (β). The left had side is lower bouded by b β β 0 b C K log()/ b > a max{λ, λ }, which proves (9). 5

6 Next, we cosider (b). Same as before, it suffices to show (33). For β B, deote by β = T T (µ) its orthogoal projectio oto M. By Taylor expasio, Q (β) Q (β ) = (y Xβm ) T X(β β ) + K + K, p j= P (β m ) β j (β j β j ) where β m is i the lie betwee β ad β. Let ρ i (t) = ρ i (t)sg(t), i =,. Rearragig the sums i K, we ca write L K = λ l= l= ρ (βi m β m [ j ) (βi β j ) (βi βj ) ] i B l,j B l+ L +λ ρ (βi m β m [ j ) (βi β j ) (βi βj ) ]. i,j B l For those (i, j) ot belogig to the same true group, β m i β m j b β m β 0 b β β 0 b β β 0 b β β 0 > b C K log()/. From the coditios o (b, λ, λ ), it is easy to see that ρ l (β m i β m j O the other had, for those (i, j) belogig to the same true group, β i = β j sg(β m i L K = λ β m j ) = sg(β i β j ). Together, we fid that l= i B l,j B l+,i j L λ l= i B l,j B l+,i j ρ (β m i β m j )β i β j + λ L ρ (t )β i β j + λ L l= i,j B l,i j l= i,j B l,i j ) = 0, l =,. ad hece ρ (β m i β m j )β i β j ρ (t )β i β j, (4) where i j meas i ad j are i the same true group, ad the last iequality comes from the cocavity of ρ ad the fact that β m i β m j βm β β β t. Now, we simplify K. Let z = z(β m ) = X T (y Xβ m ) ad write K = zt (β β ). Note that for each j k, βj = k i k β i = uk k l=d k i V kl β i, where V kl, d k ad u k are as i Propositio B.. K = = = K u k k= l=d k K u k k= l=d k K k= k j V kl z j (β j β j ) z j k j V kl u k u k u k l =d k j V kl l=d k l =d k j V kl j V kl 6 (β j β j ) (z j z j )(β j β j )

7 = K k= K k= k k K + K. u k l=d k j,j V kl (z j z j )(β j β j ) d k l<l u k j V kl,j V kl (z j z j )(β j β j ) Usig otatios i Propositio B., K uk k= l=d k = L bl l= k=a l. Therefore, K = = L b l l= k=a l L l= j,j k (z j z j )(µ j µ j ) V kl j,j B l,j j θ jj (z)(µ j µ j ), (5) where θ jj (z) k (z j z j ) for j, j k. To simplify K, ote that give ay (j, j ) such that j V kl ad j V kl, for some k ad l < l, we have β j β j = l h=l+ V kh { (il },i l+,,i l ): i l =j, i l =j ; i h V kh,h=l+,,l Pluggig this ito the expressio K, we obtai K = = K k= K k= k k d k l<l u k {(i l,i l+,,i l ): i h V kh } l d k l<l u k h=l l (β ih β ih+ ). h=l l (z il z il ) l h=l+ V kh h=l j V kh,j V k(h+) ω jj,ll h(z)(β j β j ), where for (j, j, l, l, h) such that j V kh, j V k(h+) ad l h l, ω jj,ll h(z) = z j z j, l = h = l V kl V k(l+) (z j z kl ), l = h < l V kl V kl V kh V k(h+) ( z kl z kl ), l < h < l V kl V k(l ) ( z kl z j ), l < h = l ad z kl is the average of {z j : j V kl }. By rearragig terms, K L bh h= k=ah (l,l ):d k l h<l u k. Therefore, K = L h= b h k=a h k j V kh,j V k(h+) [ h u k l=d k l =h+ k=, (β ih β ih+ ) d k l<l u k l h=l = ] ω jj,ll h(z) (β j β j ) 7

8 = L h= j B h,j B h+,j j τ jj (z)(β j β j ), (6) where τ jj (z) = = = k h h k + k u k l=d k l =h+ u k l=d k l =h+ u k l =h+ h V kl ( u k k l=d k u k ( h k l =h+ ω jj,ll h(z) V kl V kl V kh V k(h+) ( z kl z kl ) + h k l=d k V kl V k(h+) (z j z kl ) + k (z j z j ) l =h+ V kl ) V kh V k(h+) l=d k V kl )V kl V kh V k(h+) z kl + uk k l =h+ V kl V k(h+) V kl V kh ( z kl z j ) z j z kl h l=d k V kl z j. k V kh Let kh = l hv kl ad kh = l>hv kl. The, for ay (j, j ) such that j B h, j B h+ ad j, j k, we have the followig expressio τ jj (z) = Combiig (5) ad (6) gives ( kh V kh V k(h+) k i k(h ) z i kh k i k(h+) ( kh + k V k(h+) z j kh ) k V kh z j. (7) z i ) K L l= i B l,j B l+, i j τ ij (z)β i β j + L θ ij (z)β i β j. (8) l= i,j B l,i j Usig the iequalities o K ad K, i.e., (4) ad (8), we have Q (β) Q (β ) L l= i B l,j B l+,i j + L l= i,j B l,i j [ λ ρ (t ) τ ij (z) ] β i β j [ λ ρ (t ) θ ij (z) ] β i β j. Therefore, showig (33) reduces to showig that, over the evet E E, for sufficietly small t, max ij τ ij (z) < λ ρ (t ) ad max θ ij (z) < λ ρ (t ), (9) ij 8

9 except for a probability of at most ( p). Note that z = X T ε η η m, where η = X T X(β β 0 ) ad η m = X T X(β m β ). It is see that η m λ max (X T X) β β λ max (X T X)t. So τ ij (z) = τ ij (X T ε+η)+rem, where the remaider term is uiformly bouded by g (t ), for some fuctio g ( ) such that g (0+) = 0. Similar situatios are observed for θ ij (z). s a result, to show (9), it suffices to show that over the evet E E, ad except for a probability of at most ( p). First, cosider (0). Let E 3 be the evet max θ ij (X T ε + η) < λ ρ (0+), (0) ij max τ ij (X T ε + η) < λ ρ (0+), () ij max θ ij (X T ε) k 6c 3 log(( p))/, for all k. i,j k Note that θ ij (X T ε) = k (x i x j ) T ε, where x i x j. pplyig Coditio 3.3 ad the uio boud, P ((E 3) c ) K P k= i,j k ( ) (x i x j ) T ε > x i x j 3c 3 log(( p)) < ( p). Moreover, θ ij (η) k max i η i η k, where η k is the average of {η i : i k }. Note that max i k η i η k ν k β β 0 ad β β 0 β β 0 because β is the orthogoal projectio of β oto M. Noticig that β B, we obtai max θ ij (η) Cν k k K log()/. i,j Combig the above results to the choice of λ gives max i,j θ ij (z) λ, ad (0) follows. Next, cosider (). First, we boud τ jj (X T ε). From (7), τ jj (X T ε) = ã T jj ε, where ã jj = [ kh V kh V k(h+) k X k(h ) kh ] k(h ) k X k(h+) k(h+) kh + k V k(h+) x j kh k V kh x j. Recall that σ k is the maximum eigevalue of X T k X k. It follows that ( ã jj kh k(h ) 4σ + kh k(h+) ) kh k V kh V k(h+) k + k V k(h+) + kh k V kh 9

10 4σ k k B h B h+ + B h+ + B h, h > d k, h + < u k k + +, B h V k(h+) k B h h > d k, h + = u k k V kh B h+ + B h+ + k, h = d k, h + < u k, h = d k k, h + = u k k σ k mi{ k 3, mi dk h u k {B h }} = σ kφ k. () Here i the secod iequality, we have used the followig facts: ()From Propositio B., for d k < h < u k, V kh = B h ad V k(h+) = B h+. () Whe h = d k, kh = V kh; whe h+ = u k, kh = V k(h+). (3) k(h ) < kh k, k(h+) < kh k, ad kh + kh = k. I the third iequality, we have used the fact that V kh whe V kh. Let E 3 be the evet that max τ jj (X T ε) C σ k φ k log( p)/, for all k. (3) j,j pplyig Coditio 3.3, () ad the uio boud, it is easy to see that P ((E 3 )c ) < ( p) for some large eough costat C > 0. Secod, we boud τ jj (η). We observe from (7) that τ jj (v) = 0, for ay v with equal elemets i k. Thus, τ jj (η) = τ jj (η η k ), where η k is the average over the elemets of η i k. By similarly aalysis to that i (), we fid that τ jj (η) = τ jj (η η k ) φ k ( max i k {η i η k } ). By defiitio, max i k {η i η k } ν k β β 0 Cν k K log(). It follows that max τ jj (η) Cν k uk K log()/. (4) j,j Combiig (3) ad(4), we the obtai () from the coditio o λ. B.4 Proof of Theorem 4. Sice β oracle β 0 = (X T X ) (X T ε), to show the claim, it suffices to show B (X T X ) / X T ε d N(0, H). Equivaletly, for ay a R q, a T B (X T X ) / X T ε d N(0, a T Ha). (5) 0

11 Let v = X (X T X ) / B T a, ad write the left had side of (5) as v T ε = i= v iε i. The v i ε i s are idepedetly distributed with E[v i ε i ] = 0 ad E[v i ε i ] = v i. Let s = i= E[v iε i ]. By Lideberg s cetral limit theorem, if for ay ɛ > 0, the s i= v iε i from the Slutsky s lemma. lim s E [ v i ε i {v i ε i > ɛs } ] = 0, (6) d N(0, ). Sice s = a T B B T a a T Ha, (5) follows immediately It remais to show (6). Usig the formula E[X{X > ɛ}] = ɛp (X > ɛ) + ɛ P (X > u)du for X = v i ε i, we have E [ v i ε i {v i ε i > ɛs } ] = ɛ s P (v i ε i > ɛs ) + From Coditio 3.3, P (v i ε i > ɛs ) e c 3ɛ s /v i v i 4 c 3 ɛ4 s 4, ɛs P (v i ε i > u)du. where the last iequality is due to that exp( x) x k for ay x > 0 ad positive iteger k. Similarly, Note that s ɛs P (v i ε i > u)du ɛs e c 3u/v i = O() sice s a T Ha. We have E [ v i ε i {v i ε i > ɛs } ] s C i= du = v i c 3 e c 3ɛs /v i v 4 c 3 ɛs. v i 4 = C X (X T X ) / B T 4 4 i= C ( X (X T X ) / B T,4 a ) 4. The right had side is o() by assumptio. This proves (6). B.5 Proof of Corollary 4. It is easy to see that the asymptotic variace of a T ( β ols β 0 ) is a T (X T X) a = v. Cosider a T ( β β 0 ). Notig that β β 0 = M D( β β 0 ), we ca write a T ( β β 0 ) = a T M D(X T X ) / (X T X ) / ( β β 0 ), where D = diag( /,, K / ). Take B = a T M D(X T X ) / ad apply Theorem 4.. It implies that the asymptotic variace of a T ( β β 0 ) is B B T = a T M D(X T X ) DM T a.

12 Observig that X = XM D, the above quatity is equal to a T M (M T X T XM ) M T a = v. Next, we show v > v. Sice M T M = I K, there exists a orthogoal matrix Q such that M is equal to the first K colums of Q. Write b = Q T a ad G = Q T X T XQ. Direct calculatios yield v = b T G b ad v = b T G b, where b is the subvector of v formed by its first K elemets ad G is the upper left K K block of G. From basic algebra, v v. B.6 Proof of Theorem 4.3 The proof of β oracle β 0 = O p ( K/) is the same as that i Theorem 3.. We oly eed to show that β oracle is a strictly local miimizer of Q sparse, with probability at least ɛ 0 K ( s) ( s). Without loss of geerality, we assume S = {,, p} ad s = p. Let B = {β : β β 0 C K log()/}, for a sufficietly large costat C > 0. By assumptio ad (5), β oracle B except for a probability of at most (ɛ 0 + K). For ay β B, let β S be the vector such that β S,j = β j {j S}, where S is the support of β 0 ; ad let β S be the orthogoal projectio of β S oto M, amely, β S,j = k i k β j for ay j k, ad βs,j = 0 for ay j / S. We aim to show that except for a probability of at most ( s) + ( p), (a) For ay β B, Q sparse (β S) Q sparse ( β oracle ), (7) ad the iequality is strict wheever β S β oracle. (b) There exists a positive sequece {t } such that, for ay β B, β S β oracle t, ad the iequality is strict wheever β S β S. Q sparse (β S ) Q sparse (β S), (8) (c) There exists a positive sequece {t } such that, for ay β B, β β oracle t, ad the iequality is strict wheever β β S. Q sparse (β) Q sparse (β S ), (9)

13 Suppose (a)-(c) hold. Cosider the eighborhood of β oracle defied as B = {β B : β β oracle mi{t, t }}. It is easy to see that β β oracle t ad β S β oracle β β oracle t for ay β B. s a result, Q sparse (β) Q sparse ( β oracle ) for β B, ad the iequality is strict except that β = β S = β S = β oracle. It follows that β oracle is a strictly local miimizer of Q sparse. Now, we show (a)-(c). We claim that (a) ad (b) hold except for a probability of at most ( s). The proofs are exactly the same as those for (7) ad (8), by otig that Q sparse (β) = Q (β) for ay β B whose support is cotaied i S. To show (c), ote that β β S β S β oracle, sice β S is the projectio of β oto the coordiate space of S ad β oracle belogs to this space. So it suffices to show that (9) holds for all β B such that β β S t. By Taylor expasio, Q sparse (β) Q sparse (β S ) = (y Xβm ) T X(β β S ) + λ ρ(βj m )β j, where β m lies i the lie betwee β ad β S. Let z = z( β m ) = X T (y Xβ m ). First, ote that sg(β m j ) = sg(β j) for j / S. Secod, β m β S β β S t. Hece, for j / S, β m j t. By the cocavity of ρ, ρ (β m j ) ρ (t ). Combiig the above, we get j / S Q sparse (β) Q sparse (β S ) ρ j / S[λ (t ) z j ]β j. Write z = X T ε + η + η m, where η = X T X(β 0 β S ) ad η m = X T X(β S β m ). Sice β S β m β S β t, η m λ max (X T X)t. Cosequetly, Q sparse (β) Q sparse (β S ) [ λ ρ (0+) X T ε + η g (t ) ] β j, j / S where g (t ) = λ [ρ (0+) ρ (t )] + λ max (X T X)t satisfyig g (0) = 0. Therefore, if X T ε + η < λ ρ (0+), (0) the there always exits sufficietly small t such that (9) holds. It remais to show (0). First, by Coditio 3.3 ad applyig the probability uio boud, X T ε (/c 3 ) log(( p)), except for a probability of at most ( p). Secod, η X T X S, β S β 0 X T X S, C K log()/. where we have used the fact that β 0 β S β β 0 C K log()/. Combiig the two parts, X T ε + η C ( log( p)/ + X T ) X S, K log()/ λ, which proves (0). 3