SUPPLEMENT TO OPTIMAL RATES OF CONVERGENCE FOR SPARSE COVARIANCE MATRIX ESTIMATION. University of Pennsylvania and Yale University

Size: px

Start display at page:

Download "SUPPLEMENT TO OPTIMAL RATES OF CONVERGENCE FOR SPARSE COVARIANCE MATRIX ESTIMATION. University of Pennsylvania and Yale University"

Gwendolyn Evans
5 years ago
Views:

1 Submitted to the Annals of Statistics SUPPLEMENT TO OPTIMAL RATES OF CONVERGENCE FOR SPARSE COVARIANCE MATRIX ESTIMATION By T. Tony Cai and Harrison H. Zhou University of Pennsylvania and Yale University In this supplement we prove the additional technical lemmas used in the proof of Lemma 6.. Proof of Lemma 8 (ii). Jensen s inequality yields that for any two densities q 0 and q with respect to a common dominating measure µ, 2 Å q q 0 q dµ 0 q 2 Ôq0 q Õ 2 Å q 2 q dµ dµ 0 dµ. q q Equation (40) implies Ẽ Ôγ,λ Õ TV 2 PÔ,0,γ,λ Õ, P Ô,,γ,λ Õ c 2 2, where T V ÔP, QÕ denotes the total variation distance between two distributions P and Q, which then yields (58) Ẽ Ôγ,λ Õ TV PÔ,0,γ,λ Õ, P Ô,,γ,λ Õ c2, due to the simple fact ÔAverage Øa i ÙÕ 2 Average a 2 i. Note that the total variation affinity ÐP QÐ TV ÔP,QÕ for any two probability distributions P and Q. Equation (58) immediately implies Ẽ Ôγ,λ Õ PÔ,0,γ,λ Õ P Ô,,γ,λ Õ c 2 0. Thus we have P,0 P, Ẽ Ôγ,λ Õ PÔ,0,γ,λ Õ P Ô,,γ,λ Õ c 2 0 following from Lemma 4. The research of Tony Cai was supported in part by NSF FRG Grant DMS The research of Harrison Zhou was supported in part by NSF Career Award DMS and NSF FRG Grant DMS q

2 2 T. TONY CAI AND HARRISON H. ZHOU 2. Proof of Lemma 0. Write «0 v Ôp Õ Σ Σ 0 v Ôp Õ T 0 Ôp Õ Ôp Õ 0 v Ôp Õ Σ 2 Σ 0 T v 0 Ôp Õ Ôp Õ Ôp Õ where v Ôp Õ Ôv j Õ 2jp satisfies v j 0 for 2 j p r and v j 0 or for p r j p with ÐvÐ 0 k, and v v Ôp Õ j satisfies a 2jp similar property. Without loss of generality we consider only a special case with v j v j, p r j p r k 0, otherwise, p r k J j p r 2k J 0, otherwise. It is easy to see that ² Jǫ 2 n,p, i j q ij ǫ ± 2 n,p, p r i p r k, and p r k J j p r 2k J 0, otherwise. It is clear that the rank of ÔΣ Σ 0 Õ ÔΣ 2 Σ 0 Õ is 2. A straightforward calculation shows that the characteristic polynomial det ÖλI p p ÔΣ Σ 0 Õ ÔΣ 2 Σ 0 Õ λ Jǫ 2 n,p 2λ p 2 which implies ÔΣ Σ 0 Õ ÔΣ 2 Σ 0 Õ has two identical nonzero eigenvalues Jǫ 2 n,p. Note that this special case corresponds to and I r Øj : p r j p r kù I c Øj : p r k J j p r 2k JÙ. Hence, I r I c Øj : p r k J j p r kù with CardÔI r I c Õ J. The general case can be reduced to the special case by matrix permutations.

3 SPARSE COVARIANCE MATRIX ESTIMATION 3 3. Proof of Lemma. Let (59) A ÖI ÔΣ 0 Σ Õ ÔΣ 0 Σ 2 Õ Σ 2 0 I ÔΣ 0 Σ Õ ÔΣ 0 Σ 2 Õ, and Note that R γ,λ λ,λ ½ R γ,λ,λ,λ ½ logdet ÔI AÕ. logdet I ÔΣ 0 Σ Õ ÔΣ 0 Σ 2 Õ Σ 2 0 I ÔΣ 0 Σ Õ ÔΣ 0 Σ 2 Õ (60) logdet ØÖI A ÖI ÔΣ 0 Σ Õ ÔΣ 0 Σ 2 Õ Ù logdet ÖI ÔΣ 0 Σ Õ ÔΣ 0 Σ 2 Õ logdet ÔI AÕ 2log Jǫ 2 n,p R γ,λ,λ,λ ½ where the last equation follows from Lemma 0. NowweestablishEquation(46).Itisimportanttoobservethatrank ÔAÕ 2 due to the simple structure of ÔΣ 0 Σ Õ ÔΣ 0 Σ 2 Õ. Let be an eigenvalue of A. It is easy to see that A Σ 2 0 I Σ0 Σ Σ 0 Σ 2 ß Ô Σ 0 Σ Σ 0 Σ 2 Õ 3 Å 2 «2 3 3 ß 3 Å (6) since Σ Σ 0 Σ Σ 0 2kǫ n,p ß3 and λ min ÔΣ 0 Õ I Σ 0 I Σ 0 2ß3 from Equation (22). Note that log Ô xõ 2 x, for x 6, which implies i.e., Note that and R γ,λ,λ,λ ½ 4 A, n exp 2 Rγ,λ,λ exp Ô2n AÕ.,λ ½ I Σ 0 I Σ 0 2kǫ n,p 3 ÔΣ 0 Σ Õ ÔΣ 0 Σ 2 Õ 3 3.

4 4 T. TONY CAI AND HARRISON H. ZHOU We can write «ô 2 Σ 2 0 I ÔI ÔI Σ 0 ÕÕ 2 I I ÔI Σ 0 Õ k I k ô (62) Ôm 2Õ ÔI Σ 0 Õ m ÔI Σ 0 Õ m0 where Define ô m0 Ôm 2Õ ÔI Σ 0 Õ m ô m0 Ôm 2Õ Å m 3. 3 (63) A ÔI Σ 0 Õ ÔΣ 0 Σ Õ ÔΣ 0 Σ 2 Õ then A ÖI ÔΣ 0 Σ Õ ÔΣ 0 Σ 2 Õ ô m0 Ôm 2Õ ÔI Σ 0 Õ m A 3 3 A 27 8 A 27 8 max ØA, A Ù 3 from Equations (59) and (62). It is then sufficient to show Å 27 (64) Ẽ Ôλ,λ Ẽ ½ ÕJ Ôγ,λ ÕÔλ,λ ½ Õ exp 2 nmax ØA, A Ù 3 2, where A depends on the values of λ,λ ½ and Ôγ,λ Õ. We dropped the indices λ, λ ½ and Ôγ,λ Õ from A to simplify the notations. Let A a ij i,j. Then A max mp j a mj and A max mp i a im. We will show that for every non-negative integer t and every absolute row sum we have «ô Å P a mj 2t ǫn,p kǫ 2 k 2 t n,p j and the same tail bound holds for every absolute column sum, which immediately implies Å P max ØA, A Ù 2t ǫ n,p kǫ 2 k 2 t n,p 2p.

5 SPARSE COVARIANCE MATRIX ESTIMATION 5 For each rowm, definee m Ø,2,...,rÙ Þ Ø,mÙ.Note thatforeach column of λ Em, if the column sum of λ Em is less than or equal to 2k 2, then the other two rows can still freely take values 0 or in this column, because the total sum will still not exceed 2k. Let n λem be the number of columns of λ Em with column sum at least 2k, and define p λem r n λem. Without loss of generality we assume that k 3. Since n λem Ô2k 2Õ r k, the total number of s in the upper triangular matrix by the construction of the parameter set, we thus have n λem r 3 4, which immediately implies p λem r n λem r 4 p 8. Recall that the distribution of Ôγ,λ Õ given Ôλ,λ ½ Õ is uniform over Θ Ôλ,λ ½ Õ. Thus from Lemma 0 we have «ô P a mj 2t ǫn,p kǫ 2 n,p λ E m j k t pλem k t p λem k k 2 for every non-negative integer t as shown in Equation (47), which immediately implies «ô Å P a mj 2t ǫn,p kǫ 2 k 2 t n,p for every t 2. j For any random variable X 0 and constant a 0 it is known that EX P ÔX xõdx P ÔX xõdx P ÔX xõdx x0 xa xa a P ÔX xõdx. t 2β β xa SettingX exp 27 2 nmax ØA, A Ù anda exp 27n where β and p n β as defined in Section, we have Å 27 Ẽ Ôλ,λ Ẽ ½ ÕJ Ôγ,λ ÕÔλ,λ ½ Õ exp 2 nmax ØA, A Ù Å 2β exp 27n β ǫ n,p kǫ 2 n,p Å 27 27nkǫ 3 n,p exp 2 n 2t ǫ n,p kǫ 2 n,p 2p Å 54β exp β c n,pǫn,p 3 q (65) exp t 2β β Å t 2β β ǫ n,p kǫ 2 n,p, k 2 log Ô2pÕ Ôt Õlog k 2 27n Ôt Õkǫ 3 n,p Å t dt dt.

6 6 T. TONY CAI AND HARRISON H. ZHOU É Í logp Note that ǫ n,p υ n, k 2 c n,pǫ q n,p, υ 2 β 54β and Mυ q 3 as defined in Section 3, and c n,p Mn q 2 ÔlogpÕ 3 q 2 from Equation (3). These facts imply Å Å 54β 54β (66) exp β c n,pǫn,p 3 q exp β υ2 Mυ q e 3 3 2, and also (67) 27nkǫ 3 n,p 27Mυ 3 q Å logp β β β Å Å 2β logp β β log k 2. The last two equations yield exp t 2β β log Ô2pÕ Ôt Õlog k 2 27n Ôt Õkǫ 3 n,p dt o ÔÕ since logp. Then Equation (65) is bounded from above by 3 2, which immediately implies (64) and thus establishes Lemma. Remark Under the assumption c n,p Mn q 2 ÔlogpÕ 3 q 2 we obtain a finite upper bound in Equations (66) and (67). It is not clear to us if this assumption can be weakened to c n,p Mn q 2 ÔlogpÕ q 2. Department of Statistics The Wharton School University of Pennsylvania Philadelphia, PA tcai@wharton.upenn.edu URL: tcai Department of Statistics Yale University New Haven, CT huibin.zhou@yale.edu URL: hz68

DISCUSSION OF INFLUENTIAL FEATURE PCA FOR HIGH DIMENSIONAL CLUSTERING. By T. Tony Cai and Linjun Zhang University of Pennsylvania

Submitted to the Annals of Statistics DISCUSSION OF INFLUENTIAL FEATURE PCA FOR HIGH DIMENSIONAL CLUSTERING By T. Tony Cai and Linjun Zhang University of Pennsylvania We would like to congratulate the