Week 10 Spring Lecture 19. Estimation of Large Covariance Matrices: Upper bound Observe. is contained in the following parameter space,

Transcription

1 Week 0 Sprig 009 Lecture 9. stiatio of Large Covariace Matrices: Upper boud Observe ; ; : : : ; i.i.d. fro a p-variate Gaussia distributio, N (; pp ). We assue that the covariace atrix pp = ( ij ) i;jp is cotaied i the foowig paraeter space, : jij j M ji jj (+) for a k (; "; M) = () ad ax () =" Theore Uder the assuptio (0), we have og p if ^ C og p + + C : () + + This theore tes us that there is a estiator ^ to obtai the rate. I the ext ecture we wi show this rate ca ot be iproved. This resut iproves the rate og p + i Theore i Bicke ad Levia (008a). Whe = = ad p = e p, their rate is 6, whie the rate i Theore 5 is. The key reaso for such a iproveet is that we reaized atrix estiatio is fudaetay di eret fro vector estiatio! stiatio Procedure: De e ad for a itegers = (~ ij I f i < + ; j < + g) pp S () = p = p ad. We estiate by ^ = k S (k) S (k). (3) We wi set k + for the operator or ad (+) for the robeius or. Techicay it is reativey easier to study this risk upper of the taperig estiator uder the operator or tha the usua badig estiator. Lea We have ^ = (w ij ~ ij ) pp where w ij = k f(k ji jj) + (k ji jj) + g.

2 Note that 8 < whe ji jj k w ij = k f(k ji jj) + (k ji jj) + g = (0; ) whe k < ji jj < k : 0 otherwise. Now we estabish the risk upper boud for the estiator i equatio (3) uder the operator or. We show that the variace part, ad the bias part, thus which ipies by settig ^ ^ C k + og p (4) Ck (5) k + og p C C + k, og p + + k = +. (6) Let C be a geeric costat which ay vary fro pace to pace. We prove the risk upper boud (5) for the bias part rst. It is we kow that the to or of a syetric atrix A = (a ij ) pp is the bouded by its to or, i.e., p kak ax ja ij j. i=;:::;p j= We boud the operator or of the bias part ^ ~ ij = ij, we have by its to or. Sice ^ = ((w ij ) ij ) pp where w ij [0; ] ad is exacty whe ji ^ 4 ax i=;:::;p j:ji jj>k jj k, the 3 j ij j5 M k. Now we estabish (4) which is reativey copicated. The key idea i the proof is to write the whoe atrix as a average of atrices which are su of a arge uber of sa disjoit bock atrices, ad for each sa bock atrix the cassica rado atrix theory ca be appied. The foowig ea shows

3 that the operator or of the rado atrix is cotroed by the axiu of operator ors of p uber of k k rado atrices. Let = (~ ij I f i < + ; j < + g) pp. De e N () = ax. p + Lea 3 Let be de ed as i (3). The 3N () : or each sa rado atrix with = k, we cotro its operator or as foows. Lea 4 There is a costat > 0 such that o P N () > x p5 exp x (7) for a 0 < x < ad p. With Leas 3 ad 4 we are ow ready to show the variace boud (4). By Lea 3 we have h i 9 N (k) = 9 N (k) I N (k) x + I N (k) > x 9 x + I N (k) > x. N (k) Note that kk, which is bouded by a costat, ad. The Cauchy Schwarz iequaity the ipies C x + + C I N (k) > x r r C "x 4 + x# + C P N (k) >. q og p+ Set x = 4. The x is bouded by as!. ro Lea 4 we obtai og p + C + p p5 p 8 e 8 = og p + C : (8) Now we give proofs of auxiiary eas. Proof of Lea : It is easy to see kw ij = # f : fi; jg f; : : : ; + k gg # f : (i; j) f; : : : ; + k gg = (k ji jj) + (k ji jj) + ; 3

4 which takes vaue i [0; k]. Ceary fro the above, kw ij = k for ji jj k. Proof of Lea 3: Without oss of geeraity we assue that p ca be divided by. Set () Sice () j+ S () S () = S (). By (3) = () j+ j < p= : (9) are diagoa bocks of their su over j < p=, we have S () ax () j+ ax () : p 0j < p= This ad (3) ipy the cocusio, sice (k) certai atrix (k) with p k +. ad (k) are a sub-bocks of Proof of Lea 4 : or ay syetric atrix A, we have v T Av u T Au v T Av = (u v) T A (u + v) ku vk kak ku + vk u T Au Let S = be a = et of the uit sphere S i the ucidea distace i R. We have kak u T Au us us = u T Au + kak 3 = u T Au + 3 us 4 kak = which ipies kak 4 us u T Au. Sice we are aowed to pack Card S = = bas of radius =4 ito a + =4 ba i R, voue copariso yieds i.e., Card S = (=4) Card S = (5=4) ; 5. Thus there exist v ; v ; : : : ; v 5 S such that kak 4 j5 v T j Av j ; for a syetric A. This oe step approxiatio arguet is siiar to the proof of Propositio 4. (ii) i hag ad Huag (008). Let ; : : : ; be i.i.d. p-vectors with ( ) ( ) T =. Uder the Gaussia (sub-gaussia) assuptio there exists > 0 such that P v T ( i i )( i i ) T v > x e x= for a x > 0 ad kvk = which ipies tv T ( i i )( i i ) T v < for a t < = ad kvk =, the there exists > 0 such that ( P v T ( i i )( i i ) T ) v > x e x = i= 4

5 for a 0 < x < ad kvk =. (See, e.g., Chapter i Sauis ad Statuevicius (99).) Thus we have o P ax > x P > x p + p + p5 Pfjvj T ( v j; p5 exp x =. )v j j > xg Reark: The proof here works for sub-gaussia assuptio which is sighty ore geera tha Gaussia. 5

6 Lecture 9. stiatio of Large Covariace Matrices: Lower boud (I) Observe ; ; : : : ; i.i.d. fro a p-variate Gaussia distributio, N (; pp ). We assue that the covariace atrix pp = ( ij ) i;jp is cotaied i the foowig paraeter space, o (; "; M) = : j ij j M ji jj (+) for i 6= j ad ax () =". (0) I additio we assue that p e. I this ecture we wi see this assuptio is ecessary to estiate pp cosistety uder the operator or. Theore 5 Uder the assuptio (0), we have if c og p + + c ^ : () I this ecture we wi show if ^ c og p by usig Le Ca s ethod. Next tie we wi appy Assouad s ea to prove the other part of the ower boud if c +. ^ We wi appy Le Ca s ethod to derive a ower boud for iiax risk. Let be a observatio fro a distributio i the coectio fp ; = ; ; : : : ; p gg. Le Ca s ethod, which is based o a two-poit testig arguet, gives a ower boud for the axiu estiatio risk over the paraeter set. More specificay, et L be the oss fuctio. De e r ( 0 ; ) = if t [L (t; 0 ) + L (t; )] ad r i = if p r ( 0 ; ), ad deote P = p P p = P. Lea 6 Let T be a estiator of based o a observatio fro a distributio i the coectio fp ; = ; ; : : : ; p gg, the L (T; ) r i P0 ^ P or q p, et be a diagoa covariace atrix with = og p +, ii = for i 6=, ad et 0 be the idetity atrix. Let = ; ; : : : ; p T N (0; ), ad deote the joit desity of ; : : : ; by f, p with p = ax fp; exp (=)g, which ca be writte as foows Y f = x i Y j x i i;jp;j6= i 6

7 where, = or, is the desity of N 0;. Let = for 0 p ad the oss fuctio L be the squared operator or. It is easy to see d ( 0 ; ) = og p for a p. The the ower boud (??) foows iediatey if there is a costat c > 0 such that P0 ^ P c. () Sice R R q 0 ^ q d = jq0 Jese s iequaity ipies q j d for ay two desities q 0 ad q, ad the q 0 q jq 0 q j d = d q (q0 q ) d = q Hece R q 0 ^ q d q q 0 q d : R q 0 q d =. To estabish equatio (), it thus su ces to show that R p P p = f =f0 d! 0, i.e., p p = f d + p 6=j We ow cacuate R f f j d. or 6= j it is easy to see f f j d = 0. Whe = j; we have Thus f d = p p = p p Y i f f j d! 0. (3) exp x i h ( ) i = = og p =. p f! = d = p = = exp = + dx i f d p og p og p og og p for 0 < <, where the ast step foows fro the iequaity og ( x) x for 0 < x < =. = p p! 0 (4) 7