SUPPLEMENT TO ROP: MATRIX RECOVERY VIA RANK-ONE PROJECTIONS 1

Transcription

1 SUPPLEMENT TO ROP: MATRIX RECOVERY VIA RANK-ONE PROJECTIONS BY T. TONY CAI and ANRU ZHANG Deartment of Statstcs The Wharton School Unversty of Pennsylvana We rove n ths sulement the techncal lemmas used n the roofs of the man results. Proof of Lemma.. Suose X : R! R n s gven by (.6), we consder rank- matrces: A = e ( ) e ( )T, A = () () k () k k ()T k Here e ( ),e ( ) are the - and - dmensonal vectors wth frst entry and others, resectvely. Then we have ka k F = ka k F =. EkX (A )k = E ( () ) ( () ) = n VarkX (A )k = Var ( () ) ( () ) = nvar By Chebyshev s nequalty, we have for all t>, =8n. (.3) P kx (A )k tn ale On the other hand, kx (A )k =k () k k () k + ( = k () k k () k ( ) 8 n(t ) ()T () k () k ) ( ( ) ()T () k () k ) The research was suorted n art by NSF FRG Grant DMS , NSF Grant DMS-898 and NIH Grant R CA

2 By Lemma n [8], we know P Hence, ( ) / ale ex( /4), P ( ) / ale ex( /4) (.4) P kx (A )k /4 ale ex( /4) + ex( /4). Combnng (.3) and (.4), we can see (.5) C /C kx (A )k /( nka k F ) kx (A )k /( nka k F ) = s kx (A )k kx (A )k r /4 tn holds wth robablty at least e /4 e /4 8 n(t ). Proof of Lemma 3.. Frst, the common used defnton for sub-gaussan dstrbuton of random varable X nclude the followng two. (.6) 9c, C >, such that P ( X t) ale C ex( ct ) (.7) 9c >, such that Ee tx ale ex(c t /) Suose P s fnte, then we have Ee tx = X k= t k (k)! EXk ale X k= t k (k)! k (k )!! = X k= (t ) k k k! namely X s sub-gaussan. Now suose X s sub-gaussan, then EX k =(k) = kc c k Z Z P ( X >t)t k (ct ) k dt ale kc e ct d(ct )= k!c c k whch mles that P ale max(c, )/c s fnte. Z t k e ct dt =ex(( t) /) max(c, ) k ale (k )!! c Proof of Lemma 7.. Wthout confuson, we smly use to reresent P n the roof. Note that we can multly A by a scale wthout loss of generalty. So we assume throughout the roof that kak F =. We ll rove ths lemma by stes. Frst, we show an nequalty on the even moments of T A ; next, we gve a bound on the moment generaton functon of T A. Fnally, we gve the desred tal bound.

3 . Ste : Even moments of T A. Assume that x =(x,,x ),y =(y,,y ) are two random..d. standard normal dstrbuted vectors. Based on the defnton of P n (3.), we know 3 (.8) E k = E j k ale k Ex k = k Eyj k ; E k = E j k = Ex k = Eyj k = Consder the exanson of E( T A ) k, where the non-zero terms can be wrtten as ky Y Y A l,j l E s E t j j. l= Here s + + s = t + t = k. By(.8), ths term can be bounded as j= ky l= Y A l,j l E s Y j= E t j j ale ky l= Y A l,j l 4k Ex s Y j= Ey t j j The rght hand sde s exact the term n the exanson of 4k E(x T A abs y) k, where A abs s the the element-wse absolute value of A. Therefore, we have (.9) E[ T A ] k ale 4k E[x T A abs y] k. Now we suose A abs has sngular value decomoston A abs = X a u v T = Udag(a)V T where U, V are orthogonal and a =(a,,a ) s the sngular value vector of A abs. A well-known fact s that P a = ka abs k F = kak F. Snce x, y are standard normal dstrbuted, we can see that x T A abs y and x T dag(a)y has the same dstrbuton. So Next, we note X E[x T A abs y] k = E[ a x y ] k z = X a y x q P j= a j y j,

4 4 q P then z s standard normal dstrbuted and ndeendent of j= a j y j snce a y A =. q P j= a j y j and z gven y,,y s always standard normal dstrbuted. For nteger k, v k E[x T A abs y] k u X X =E z t = E z k E( a jyj ) k j= =(k )!! = ale X k,k,,k, k + +k=k X k,k,,k, k + +k=k =((k )!!) =((k )!!) ( a j y j Together wth (.9), we have (.) E[ X k,k,,k, k + +k=k k! Q j= (k j)! k! Q j= k j! X k! Q j= k j! Y Y k,k,,k, k + +k=k X a ) k =((k T A ] k ale 4k ((k a k j= Y E(a y ) k Y (k )!! a k ((k + k + k ) )!! k! Q j= k j! Y (a ) k )!!) kak k F )!!) kak k F. Log-moment generaton functon of T A. By the bound of the even moments of T A, we can also gve the estmate of odd moments, for nteger k, ale E T A k+ ale q E[ T A ] k E[ T A ] k+ ale 4k+ (k )!! (k + )!! ale 4k+ (k + )! Also, E T A k ale 4k ((k )!!) ale 4k (k)!

5 So for all k ale t<,, E T A k ale k k!. Denote µ = E T A, then for X Ee t T A t k =+ k! E T A k ale +tµ+ For k= <t<, we have Ee t T A = + tµ + X t k 4k k= X k= 5 X t k k =+tµ+ t 4 t k= t k+ (k + )! E T A k+ ale+tµ + t 4 t 4 ale +tµ + t 4 Hence, we have for all / <t</, Ee t T A ale +tµ + t 4 t. Note that log( + x) ale +x for all (.) log E ex(t( T A µ)) = log E ex(t t <x<, wehave T A ) tµ ale t 4 t. for all / <t</. 3. The tal bound of kx (A)k /n. Fnally, we estmate the tal bound of kx (A)k/n. Note that kx (A)k /n (j)t A (j) A /n, j= based on (.), the logarthm of moment generatng functon of kx (A)k /n satsfes t log E ex (t(kx (A)k /n µ)) = n log E ex n ( T A µ) ale t 4 /n t /n By the roof of Lemma n [8], we know kx (A)k /n bound, P (kx (A)k /n µ (x/n + x/n)) ale ex( x) P (kx (A)k /n µ ale (x/n + x/n)) ale ex( x) µ has the tal Fnally we set = x/n, by Lemma 7., /(3 4 ) ale µ ale, we fnsh the roof of Lemma.

6 6 Proof of Lemma 7.. Snce P s symmetrc and of varance, we have E = E j =, E = E j =, E 4 = E j 4 ale 3 4 P for all, j. Thenby some exansons and calculatons, E T A = X,j A j j A = X,j E A j By the frst art n the roof of Lemma 7., wehave By Hölder s nequalty, E E T A ale T A 4 ale 9 8 PkAk 4 F q E T A = kak F j = X,j A j = kak F whch gves the rght of the orgnal nequalty. For the left, note that So E E Proof of Lemma 7.3. T A ale E T A T A /3 E T A 4 /3 s (E T A ) 3 kak F E T A 4 3 P 4. We frst rove the sub-gaussan art of the lemma. The moment generatng functon of z (t ) and z ( ale t</( )) satsfy Ee t z = Ee tz = ale + Z Z ex(t )dp ( X )=+ t ex(t ale +t ex(t /) Z ale +tex t / ale ex(t /) +t ale ex t /+ t ale + =+ Z Z /( ))d ex( ( ex(t )dp ( X )=+ 4t ex t /( ) t Z Z (/( ) t) d P ( X )d ex(t ) ) /( ))d P ( X )d ex(t )

7 Then the moment generatng functon of kzk /n and kzk /n satsfes Ee tkzk/n = Ee tkz n k/n ale ex t /(n)+ t Ee tkzk /n t/n n t = + /( ale ex ) t/n /( ) t/n Hence for C, P (kzk /n C) ale E ex(tkzk /n) ale ex t ex (tc) =ex@ n t + n( C) 7 /(n)+t C! n( C) A For C>, we can set t = n(c P (kzk /n C) ale ex,then ) n(c! ) Now we consder the tal bound of kzk /n. For C>4, P (kzk /n C) = E ex(tkzk /n) t C/n t(c 4 ) =ex ex(tc). t/n We set t = C 4 4C /n, P kzk /n C ale ex Fnally we consder kzk, n(c 4 ) n(c 4 ) 8 C( ale ex t/n) 8 C P (kzk ale C log n ) ale n ex( C log n /( )) = n (C / ) Next, we consder the Gaussan art of the lemma. The bound of kzk s already gven by Lemma 5. n [7]. For kzk, we can see E z =, Z E z = xe x /() dx = / Hence, E(kzk /n) = /, Var(kzk /n) = Var( z )/n = ( / ) /n. By Chebyshev nequalty, P (kzk /n ) ale P kzk /n / ( / ) ale Var(kzk /n) ( = + / / ) ( / )n

8 8 For the bound of kzk,wehave P (kzk log n ) ale ale n P ( z ale log n ) log n ex( 4 log n) = n log n Proof of Lemma 7.4. Agan wthout confuson, we smly use to reresent P n the roof. The roof also requres some knowledge of moment generaton functon and "-net method. We ll rove by stes. Moment Generaton Functon of a T X (z)b. Suose a R, b R are fxed unt vectors. In order to handle the oerator norm of X (z), we frst consder a T X (z)b. Note that (.) a T X (z)b = z a T () ()T b Denote X = a T (), Y = b T (),then{x }, {Y } are two ndeendent sets of..d. sub-gaussan samles. Moreover by (), () are..d. from symmetrc dstrbuton P, one can show E(X ) k = E( X j a j j ) k =, X E(X ) k = E( a j j ) k = ale j= X k + k =k k! k! k! X k + k =k Y k! k! k! a k X = k E( a x ) k ale k (k )!! Y E(x k ) k! a k E( k )!

9 d Here x N(, ). Smlarly, E(Y ) k =, E(Y ) k ale k (k )!!. Then for t < /, (.3) X t k E(X Y ) k X t k ( k (k )!!) E ex(tx Y )= ale k! (k)! k= k= X (t ) k (k )!! X / = k = (t ) k ( ) k k! k = k= t 4 Now for fxed z R n, the logarthm of the moment generatng functon of a T X (z)b satsfes log E ex(ta T X (z)b) = ale ale k= log E ex(tz X Y ) ale t z 4 ( t z 4 ) ale t kzk 4 ( t kzk 4 ) t kzk 4 ( t kzk ) for any t < /(kzk ). Here we used the fact that log( x) = X x ale X x = x x 9 log( t z 4 ) for ale x<. Tal Bound of a T X (z)b By the roof of Lemma n [8], we know for fxed z R n, a T X (z)b has tal bound: for x> and fxed a, b, wehave P a T X (z)b kzk x + kzk x ale ex( x); P a T X (z)b ale kzk x + kzk x ale ex( Set x = C( + ), we have (.4) P a T X (z)b kzk C( + )+kzk C( + ) aleex( C( + )). x).

10 For convenence, We denote (.5) T = kzk (C( + )) + kzk C( + ). "-net and the uer bound of kx (z)k. In ths ste, we stll fx z. Weusethe"-net method to derve the uer bound of kx (z)k, whch s gven by kx (z)k = su a T X (z)b ar,br From Lemma.5 n [39], we can fnd an "-net A n the unt shere of R,.e. for all a n the unt shere of R,thereexstsa A such that ka ak ale. Besdes, A ale( + /"). Smlarly, there exsts -net B of the unt ball of R such that B ale( + /"). By (.4), we have (.6) P a T X (z)b T,9a A, b B ale (+/") + ex( C( + )) Now we consder under the event that a T X (z)b ale T,8a A, b B. Suose µ = kx (z)k = max kak =kbk = a T X (z)b, (a,b ) = arg max a,b a T X (z)b, then we can fnd a A, b B such that ka a kale ", kb b kale". Then, µ = a T X (z)b = a T X (z)b + (a a ) T X (z)b + a T X (z)(b b ) alet +(ka a k + kb b k ) kx (z)k alet +"µ Ths means µ ale T/( "). Therefore, when a T X (z)b ale T,8a A, b B, wehavekx (z)k alet/( "). Fnally, we set " =/3, under the event that a T X (z)b ale T = kzk (C( + )) + kzk C( + ), 8a A, b B we have kx (z)k alet/( ") ale 3 kzk C( + )+kzk C( + ) By (.6), the robablty that all the event haen s at least ex( (C log 7)( + )) Ths fnshed the roof of the lemma.

11 Proof of Prooston 7... In the roof, we wll use to reresent P wthout any confuson. The deas of the roof of Prooston 7. follows from [36], [3]. Denote S kr = {X R : rank(x) ale kr, kxk F =}. By Lemma 3. n [3], for any ">, there exsts "-net Skr such that S kr ale(9/")( + +)kr. For gven C,C such that C < /(3 4 ), C >, we set C = C +/(3 4 ), C = C +. We can choose small enough such that /(3 + 4 ) C ale mn, C then by Lemma 7., for any gven A R,wehaveC ale kx (A)k /n ale C wth robablty at least ex( n). Hence, P C ale kx (A)k /n ale C, for all A S r (9/") kr( + +) ex( n) Next, we ll estmate the bound of kx (A)k /n on the whole set S kr rovded that C ale kx (A)k /n ale C for all A S kr.defne ale = nf kx (A)k /n and ale = su kx (A)k /n. AS kr AS kr For any A S kr,thereexstsa S kr such that ka A k F ale ". So kx (A)k /n ale kx (A )k /n+kx (A A )k /n ale C kak+ale ka A k F ale C +ale " kx (A)k /n kx (A )k /n kx (A A )k /n C kak ale ka A k F C ale " whch mean ale = su AS kr kx (A)k F ale C + "ale, ale = nf AS kr kx (A)k F C "ale namely, ale ale C /( "), ale C "ale. We choose C " ale mn, /(3 4 ) C, C C by some calculatons we can see ale C, ale ale C. To sum u, we can choose, " only deendng on C,C,,toensure C ale ale = nf AS kr kx (A)k /n ale su AS kr kx (A)k /n = ale ale C

12 wth robablty at least (9/") kr( + +) ex( n). The last ste s to estmate the robablty above. We choose D 8k log(9/")/, then for n Dr( + ), we have n/ 4 log(9/")kr( + ) log + log(9/")kr( + + ), (9/") kr( + +) e ex( n/). n = ex( n + log + kr( + + ) log(9/")) Fnally, we fnsh the roof of the Theorem by choosng ale /. Proof of Lemma The dea of the roof s orgnated from [4, 3]. We rovde the roof here for the comleteness of the aer. Note = mn(, ), suose for any matrx B, (B) sthe-th largest sngular value of B. By Lemma n [3], we have ka max(r) k + ka max(r) k X =kak ka k = ka ( R)k (A) ( R) rx ( (A) (R)) + X ( (R) (A)) =r+ =ka max(r) k ka max(r) k + kr max(r) k kr max(r) k whch mles (7.3). Proof of Lemma Suose R = A A,thenwehave (.7) kx (R)k /n ale Snce ka k ale kak. By Lemma 7.7, wemusthave(7.3). Suose = mn(, ) and R has the sngular value decomoston, R = P u v T = Udag(~ )V T,then~ max(kr) satsfes k~ max(kr) k ale kr, k~ max(kr) k =k~ max(r) k ( r+ + + kr ) Set alek~ max(r) k +ka max(r) k (k )r kr = max kr, (k~ max(r) k +ka max(r) k (k )r kr )/(kr),

13 then k~ max(kr) k ale, k~ max(kr) k ale kr. Smlarly to the roof of Theorem., aly Lemma 7.6, we can get b () R n,, =,,N such that ~ max(kr) = P N ub () and (7.). Hence, kb () k ale q kb () k kb () k ale q (k~ max(r) k +ka max(r) k (k )r kr ) If = kr, we can otmze over kr n the nequalty, q kb () k ale kr(k~ max(r) k +ka max(r) k (k )r kr ) ale k~ max(r)k +ka max(r) k ; r(k ) f =(k~ max(r) k +ka max(r) k (k )r kr )/(kr), we have 3 (.8) kb () k ale k~ max(r)k +ka max(r) k (k )r kr kr ale k~ max(r)k +ka max(r) k kr. Snce k, we always have (.8). Next, we defne B = Udag(b () )V T,then the rank of B are at most kr, P N B = R max(kr) and kb k F = kb () k. Then (.9) kx (R)k /n kx (R max(kr) )k /n kx (R max(kr) )k /n C kr max(kr) k F C kr max(kr) k F C kr max(kr) k F N X X N kx ( B )k /n C kb k F C kr max(r) k +ka max(r) k kr C kr max(kr) k F C k kr max(kr) k F C ka max(r) k kr, where the last nequalty s due to kr max(kr) k F kr max(r) k F rkrmax(r) k. Therefore, (.3) kr max(kr) k F ale C C / k + ka max(r)k r( kc /C )

14 4 Fnally, kr max(kr) k F =k~ q ale max(kr) k ale q k~ max(kr) k k~ max(kr) k kr (k~ max(r) k r(k ) kr ) ale k~ max(r)k r(k ) ale k~ max(r)k +ka max(r) k r(k ) ale kr max(r)k F k + ka max(r)k r(k ) Therefore, q krk F = kr max(kr) k F + kr max(kr)k F v u ale t krmax(kr) k F + kr max(r)k F k + ka! max(r)k r(k ) s (.3) ale + 4(k ) kr max(kr)k F + ka max(r)k r(k ) ale + kr 8(k ) max(kr) k F + ka max(r)k r(k ) ale C C / 3 k + kc /C + k ka max(r) k r Proof of Lemma The roof of ths theorem s smlar to the roof of Lemma 7.8. Suose R = A A. In ths case we have (.3) kx X (R)k ale nstead of (.7). Besdes, snce ka k alekak and Lemma 7.7, westllhave (7.3). Wth the smlar argument as (.9), we have (.33) krk hr, X X (R) = kx (R)k kx (R)k /n C C ka max(r) k n C kr max(kr) k F kr max(kr) k F k kr + Here (x) + means max(x, ). Besdes, krk ale kr max(r) k +(kr max(r) k +ka max(r) k ) ale ale rkrmax(r) k F + ka max(r) k rkrmax(kr) k F + ka max(r) k

15 Suose x = kr max(kr) k F, y = ka max(r) k / r. Based on the revous two nequaltes, we have When x (.34) n (C C / k)x C y ale r(x + y) k + C y k(c C /, the nequalty above leads to k) n(c C / k) x n(c C / k) C k y + r x r y ale. 5 Note that for second order nequalty ax bx c ale, a>, b, c, we have x ale b+ b +4ac a ale b/a + c/a. Hence we can get an uer bound of x from (.34). x ale ale r n(c C / k) + 4C / ky (C C / k) + r n(c C / k) + 4C / ky (C C / k) + r y n(c C / k) r n(c C / k) + y. Hence whenever x C y k(c C / k) or not, (.35) kr max(kr) k F = x ( ale max C y k(c C / k), 3 r 3 r ale n(c C / k) + 4C / (C C / k) + n(c C / k) + 4C / k k (C C / k) +! ka max(r) k. r Fnally, smlarly to (.3) n Lemma 7.8, we can get the uer bound of krk F, krk F ale + kr 8k 8 max(kr) k F + ka max(r)k r(k ) 4 r(" + ) ale (C C / k) 5 + n kc /C + ka max(r) k + k r! y ) whch fnshed the roof of lemma 7.9.

16 6.7. Proof of Proostons.,.3 and.4.. We frst show Prooston.. Denote X, X : R! R b n c such that (.36) [X ] (B) =( ( ) + () ) T B( ( ) () ), =,, b n c (.37) [X ](B) =( ( ) () ) T B( ( ) + () ), =,, b n c Note that ( ) + () and ( ) () are ndeendent..d. standard normal samles, so both X and X are ROP desgn (see (.6)). By Corollary., we know there exsts unform constant C such that whenever b n c Cr, X satsfes the followng roerty wth robablty at least ex( n ), (.38) 8A {A R : rank(a) ale r}, A = arg mn kbk subject to X (B) =X (A). BR Now we consder the event that (.38) holds. We note that for any symmetrc matrx B, [X ] (B) = ( )T B ( ) ()T B () = ([X ] (B) [X ] (B)) So X (B) =X (A) mlesx (B) =X (A) for symmetrc A and B. Also, snce A s feasble n rogrammng (.3), we have kak mn BS subject to X (B) =X (A) mn BR subject to X (B) =X (A) =kak, So we can conclude that A can be exactly recovered by (.3) gven (.38) holds. In summary, for n 6Cr, wth robablty at least ex( n ), X satsfes (.38), then rogrammng (.3) can recover all A S of rank at most r. Next, we consder Prooston.3. The dea of the roof s smlar to Prooston.. Defne z R b n c such that (.39) z = z z, =,, b n c.

17 7 Then z d N(, ). We shall also ont out two facts, z =ỹ X (A) and X = X + X (defned as (.36), (.37)). By Lemma 7.3, weknow P (kzk /n > ) ale 9 n Hence, P (k zk > n) ale bn/c P (k zk > log n) ale bn/c P (A s NOT n the feasble set of rogrammng (.7)) =P kzk /n >, or k X ( z)k > =P kzk /n >, or k X ( z)k > 4 n + 48 log n alep (kzk /n > )+P X (k zk > n)+p (k zk > log n) + P X k X ( z)k > 4k zk + k zk ale 9 n + bn/c + P X ale 5 n kx ( z)k > k zk +6 k zk + P X kx ( z)k > k zk +6 k zk +4ex( ( log 7)) When A s n the feasble set of rogrammng (.7), we have kâk alekak and (.4) k X (Â A)k = b n c X [X ] (Â A) [X ] (Â A) ale kx (Â A)k aleky X(Â)k + kx (A) yk aleky X(Â)k + kzk ale n (.4) k X X ( Â A)k alek X (ỹ X ( Â))k + k X ( X (A) ỹ)k =k X (ỹ X ( Â))k + k X ( z)k ale Smlarly as the roof to Prooston., by Theorem., there exsts constant D, such that f n Dr, X satsfes RUB of order r wth constants C,C such that C /C < wth robablty at least ex( n ). Now we suose the followng two events haen,

18 8. X satsfes RUB of order r and constants C,C satsfyng C /C <,. A s feasble n the rogrammng (.7). Snce X (B) =X (B) for any symmetrc matrx B, byx satsfes RUB condton, we have X satsfes RUB for symmetrc matrces of order r and constants C, C satsfyng (C )/(C ) <. We note that the roof of Lemmas 7.8 and 7.9 stll aly for X n the symmetrc matrces class, so we can get (.8) based on (.4) and (.4) under those two events haen. Fnally the robablty that these events haen s at least 5/n 4ex( ) ex( n ) for <mn(( log 7), ), whch fnshed the roof of Prooston.3. Fnally we consder Prooston.4. Denote = b/c, r = br/c. By r,, we have r r/3, /3. Defne a sub-class of the class rank-r symmetrc matrces, G = 8 >< A S : A = >: ale B B T,B R ( ), rank(b) ale r 9 >= >; we can see 8A G, [X (A)] = ()T A () = ( (),, () )B( () +,, so n G the SROP model becomes y =( (),, () )B( () +,, () ) T + z, z d N(, () ) T, /4) whch s an ROP model whch we already dscussed n secton. We omt the rest of the roof as t can be followed by the roof of Theorem.4. Proof of Prooston 3... The roof can follow from the roof of Prooston.3 and Theorem 3. once we can rove that n hgh robablty, X (defned n (.36)) satsfes RUB condton wth C,C such that C /C s bounded. Ths can be roved smlarly as Prooston 7., where we only need to edt the roof that we use the followng Lemma. nstead of Lemma 7.. Lemma.. Suose A R s a fxed matrx (not necessarly symmetrc) and X s gven by (.36). () s a set of -dmensonal vectors such

19 that d P, where P s some symmetrc varance sub-gaussan dstrbuton excet Rademacher ± dstrbuton. Then for >, we have! mn 3/ (Var(P )/, ) 3( P ) 4 8 P 4 P kak F ale kx (A)k (.4) bn/c ale 3/ P +8 P +4 P kak F. wth robablty at least ex( bn/c). The roof of Lemma. s n the Aendx rght after ths aragrah. Note that rovded P s symmetrc and wth varance, Var(P ) = f and only P s Rademacher ± and A s dagonal, n whch the lower bound of (.4) becomes meanngless. So we only exclude Rademacher ± dstrbuton from the result. Proof of Lemma... The roof of Lemma. s bascally the same to Lemma 7.. We only need to redo two arts of the roof, where there are major d erences.. Part. Ste. Even moments of ( () + () ) T A( () () ).. Frst, based on P s symmetrc and wth varance, we have EP k+ =, EP k ale k Ex k = k (k )!!.Then we can calculate that 9 E( () + () ) k+ = E( () () ) k+ = = E( () + () ) k kx l= k l = k kx l= Smlarly, E( () E( () ) l E( () ) (k l) ale k (k )!! k k! l l! k l (k l)! = k (k )!! kx l= kx l= k (l )!!((k l) )!! l k = k k (k )!! l () ) k ale k k (k )!!. Next, we can smlarly consder the exanson of E ( () + () )A( () () ) k,wherethe non-zero terms can be wrtten as k Y Y ky A l,j l E( () + () ) s l= j= E( () j () j ) t j

20 Here s + + s = t + + t = k. Ths term can be bounded as k ale k ale k Y ky A l,j l l= ky l= ky l= = 4k ky l= ale 4k ky l= A l,j l A l,j l A l,j l A l,j l E( () + () ) s ( () Y s s + t E () ) t () + () s +t + Y s +t (s +t ) ((s + t ) )!! Y ky = 4k 4 A l,j l l= ((s + t ))! s ale 4k +t (s + t )! ky l= A l,j l Y Y (s +t ) ((s + t )!) ky s ale 4k 4 A +t s l,j!t! l Y Ex s Y Ey t l= t () E s + t ((s +t )!!) s +t s!t! Y (s )!(t )! (s )!!(t )!! Here we assume that x,y d N(, ). The rght hand sde of the nequalty above s exactly the term n the exanson of ( ) 4k E(x T A abs y) k, where A abs s the element-wse absolute value of A. Therefore, we have E () + () T k A () () ale ( ) 4k E[x T A abs y] k Now we follow the same argument of the rest art of Ste n the roof of Lemma 7., we can rove that (.43) E () + () T k A () () ale ( ) 4k ((k )!!) kak k F () s +t. Part. The uer and lower bound of µ = E () + () T A () (). To follow the argument of the roof of Ste 3 n Lemma 7., weneedto derve a new bound for µ = E () + () T A () ().Frst,

21 we denote M = () + () T A () (),then µ =EM ale EM s = E ( () + () ) T A( () () ) s X = E( () + () ) 4 A j ( () j,j s X = ale A j + Var(P ) X 6=j r 3 PkAk F On the other hand, EM =E ( () + () ) T A( () () ) () j ) A ale s X 6=j A j P X A = X 6=j A j + Var(P ) X A mn( Var(P ), )kak F, By (.43), we also have EM 4 ale 9( ) 8 kak k F. By Hölder s nequalty, EM ale (EM) /3 (EM 4 ) /3. Hence, r (EM µ ) 3 mn 3/ (Var(P )/, )kak F EM 4 3( ) 4 To sum u, nstead of Lemma 7., we have the bound of µ as follows, (.44) mn 3/ (Var(P )/, )kak F 3( P ) 4 ale µ ale 3/ PkAk F The rest of the roof follows the roof of Lemma 7. wth modfcatons of constants, whch we do not go nto detals. Proof of Lemma 7... Note that () N(, ()T () ), we can assume that (.45) = ()T () Z, d where Z (N(, )) and Z, () are ndeendent. Based on the defnton of z n (4.3), we have (.46) z = y [X ( )] = () () = ()T () (Z ), =,,n.

22 We also denote Q = C n, Q = C n 4, Q 3 = C 3 log n max alealen. We ll frst consder the former art of (7.8), (7.9) and (7.). Suose Z (N(, )). It s well known that the non-central m-th moment of Z s (m )!!, so we have (.47) E (C Z Z ) C qe Z = C (.48) E (C Z Z ) ale E (C Z) + E (Z ) =3C + (.49) E CZ (Z ) =3C (.5) E CZ (Z ) ale C 4 E (Z) 4 +E (Z ) 4 = 5C 4 +6 Next we consder the random quadratc form of. Suose = (,, ) d N(, ), X, d N(, ), ( ),, ( ) are the egenvalues of. Snce s ostve defnte, we have (.5) E T =tr( )=k k E( T ) =E( X + X j j ) <j = X E 4 + X <j jj E j + X 4 je <j j =( X,j j)+( X ) =k k F + k k Hence, (.5) k k ale E( T ) ale 3k k (.53) X E( T ) 4 = E( ( )X ) 4 = X ( ) j ( ) s ( ) t ( )EX Xj Xs Xt ale ale,j,s,tale X ale,j,s,tale ( ) j ( ) s ( ) t ( )7!! = 5k k 4

23 Then we consder C z and C 4 z.by(.45) and (.46), we have C z = ()T () (C Z Z ), C 4 z = ()T () CZ (Z ), whle () and Z are ndeendent n the equaton above. By (.47)- (.53), we obtan an estmaton of the frst and second moment of these two quanttes as (.54) E C z (C )k k (.55) Var C z ale E C z =E (C Z Z ) E ()T () ale 9C +6 k k 3 (.56) E C 4 z 3C k k Var C 4 z ale E C 4 z (.57) ale 5(5C 4 + 6)k k 4 We note that Q kzk /n and Q kzk /n are the average of n..d. coy of C z and C 4 z. We can mmedately get an estmaton of the mean and varance of Q kzk /n and Q kzk /n based on (.54)-(.57). Fnally, by Chebyshev s nequalty, P (Q alekzk /n) ale Var(Q kzk /n) (E (Q kzk /n)) ale 9C +6 n(c ) P Q ale kzk ale Var Q kzk /n n E Q kzk /n ale 5 5C4 + 6 n(3c ) Snce C 3 > and n 3, we know C 3 log nz Z wth robablty. Suose = arg max ()T (),then P (Q 3 alekzk ) alep max C 3 log n( ()T () )Z ale max ( ()T () )(Z ) + P max C 3 log n( ()T () )Z ale max ( ()T () )( Z ) ale+p max C 3 log n( ()T () )Z ale max ()T () alep C 3 log n ()T () Z ale ()T ( ) alep Z ale = P N(, ) ale C 3 log n ale C3 log n C3 log n

24 4 Then we consder the latter art of (7.8)-(7.). We can do smlar calculatons as the frst art of the roof and get EQ = C E = C E( ()T () )=C k k Var(Q )= C n Var ale C n E 4 = C n E ()T () EZ ale 9C n k k EQ = C EZ E ()T () ale 9C k k Var(Q )= n Var C 4 ale C4 n E 8 = C4 n EZ4 E ()T () 4 5 C 4 ale k k 4 n So by Chebyshev s nequalty, P (Q M C k k ) ale P (Q EQ (M )C k k ) ale P Q M Ck k ale P Q EQ (M 9)Ck k Var(Q ) 5 ale (M 9) C 4 ale k k4 n(m 9) 9 (M ) n whch rovde the latter art of (7.8) and (7.9). Fnally we note that =( ()T () )Z. By Lemma n [8] and the fact that k k = P q P ( ), k k = max ( ), k k F = we have (.58) P =P T alen M 3 and ( )X k k + M 3 log nk k ( ), k k F ale k k k k, k k + M 3 log nk k k k +M 3 log nk k v u ( )(X ) t M3 log n ( ) + M 3 log n max P (Z M 3 log n) ale ex( M 3 log n/) = n M 3.! ( ) A

25 Hence, P C 3 log n C 3 M 3 log n k k + M 3 log nk k ale n M 3, 5 and consequently P C 3 log n max alealen C 3 M 3 log n k k + M 3 log nk k ale n M3+, whch gves the rght sde of (7.). Deartment of Statstcs The Wharton School Unversty of Pennsylvana Phladelha, PA, 94. E-mal: tca@wharton.uenn.edu URL: htt://www-stat.wharton.uenn.edu/ tca/ E-mal: anrzhang@wharton.uenn.edu URL: htt://www-stat.wharton.uenn.edu/ anrzhang/