Strong consistency of log-likelihood-based information criterion in high-dimensional canonical correlation analysis

Transcription

1 Strog cosistecy of log-likelihood-based iformatio criterio i high-dimesioal caoical correlatio aalysis Ryoya Oda, Hirokazu Yaagihara ad Yasuori Fujikoshi Departmet of Mathematics, Graduate School of Sciece, Hiroshima Uiversity -3- Kagamiyama, Higashi-Hiroshima, Hiroshima , Japa Last Modified: Jauary 7, 209 Abstract We cosider the strog cosistecy of a log-likelihood-based iformatio criterio i a ormality-assumed caoical correlatio aalysis betwee q- ad p-dimesioal radom vectors for a high-dimesioal case such that the sample size ad umber of dimesios p are large but p/ is less tha. I geeral, strog cosistecy is a stricter property tha weak cosistecy; thus, sufficiet coditios for the former do ot always coicide with those for the latter. We derive the sufficiet coditios for the strog cosistecy of this log-likelihood-based iformatio criterio for the high-dimesioal case. It is show that the sufficiet coditios for strog cosistecy of several criteria are the same as those for weak cosistecy obtaied by Yaagihara et al Itroductio Let x = x,..., x q ad y = y,..., y p be q- ad p-dimesioal radom vectors. Ivestigatig relatioships betwee x ad y is cetral to multivariate aalysis. Caoical correlatio aalysis CCA is oe such multivariate method which has bee cosidered widely i the theoretical ad applied peer-reviewed literature, as well as textbooks aimed at uder- ad post-graduate studets see, e.g., Srivastava, 2002, chap. 4.7; Timm, 2002, chap I actual empirical cotexts, there are cases where some of the variables i a give model may be redudat. It is importat to be able to effectively idetify ad remove such variables. This paper focusses o removig redudat variables i x. The problem is regarded as selectig the best subset of x, ad this has hitherto bee ivestigated i may studies e.g., McKay, 977; Fujikoshi, 982, 985; Ogura, 200. As oe approach to model selectio, the method of miimizig a iformatio criterio is well kow. The most widely applied of these criteria is Akaike s 973, 974 iformatio criterio AIC. Fujikoshi 985 applied Akaike s idea to the issue of selectio i CCA. Nishii et al. 988 proposed a geeralized iformatio criterio GIC. Fukui 205 ad Yaagihara et al. 207 cosidered a log-likelihood-based iformatio criterio LLIC. GIC ad LLIC are essetially the same ad are defied by addig a pealty term to Correspodig author. oda.stat@gmail.com

2 a egative twofold maximum log-likelihood. GIC ad LLIC iclude several iformatio criteria as special cases: AIC, a bias-corrected AIC AIC c proposed by Fujikoshi 985, the Bayesia iformatio criterio BIC proposed by Schwarz 978, a cosistet AIC CAIC proposed by Bozdoga 987, the Haa-Qui iformatio criterio HQC proposed by Haa ad Qui 979, ad so o. A importat property to cosider regardig iformatio criteria is their cosistecy. I fact, there are two properties i this respect, that is weak cosistecy ad strog cosistecy. Let ĵ ad j be the best subset idetified by miimizig respectively a iformatio criterio ad the true subset, i.e., the miimum subset icludig the true model. Weak cosistecy meas that the asymptotic probability of selectig the true subset approaches oe, i.e., P ĵ = j, where is the sample size. I the cotext of CCA, Yaagihara et al. 207 obtaied sufficiet coditios for weak cosistecy of LLIC whe the sample size teds to ad the umber of dimesios p may ted to, assumig that the true distributio of the observatio vectors is the multivariate ormal distributio. Moreover, they derived sufficiet coditios for several specific criteria. Relaxig the ormality assumptio, Fukui 205 derived sufficiet coditios for weak cosistecy of LLIC whe both ad p ted to. O the other had, strog cosistecy meas that the probability that the best subset approaches the true subset is oe, i.e., P ĵ j =. Sice each subset is discrete, strog cosistecy esures that there exists 0 N such that for all 0, ĵ = j with probability. Hece, strog cosistecy is stricter tha weak cosistecy for selectig the true subset. Agai i the cotext of CCA, Nishii et al. 988 obtaied the sufficiet coditios for strog cosistecy of GIC whe oly the sample size teds to. However, the coditios for strog cosistecy have ot hitherto bee derived for the case where both ad p ted to. Moreover, sice strog cosistecy is stricter tha weak cosistecy, it is ot curretly kow whether several criteria satisfyig sufficiet coditios for weak cosistecy accordig to Yaagihara et al. cosistet i high-dimesioal cases. 207 are strogly The aim of this paper is to obtai sufficiet coditios for strog cosistecy of LLIC ad several other criteria whe both the sample size ad the umber of dimesios p ted to but p does ot exceed. We assume that the umber of dimesios p is a fuctio of, that is we write p = p, ad use the followig high-dimesioal HD asymptotic framework: p = p,, p c [0,. Based o sufficiet coditios for strog cosistecy of LLIC, we show that the coditios for strog cosistecy of AIC, AIC c, BIC, CAIC, ad HQC are equivalet to those for weak cosistecy put forward by Yaagihara et al The remaider of the paper is orgaized as follows. I sectio 2, we itroduce redudacy models ad LLIC. I sectio 3, we preset our key lemmas ad mai results to derive the sufficiet coditios for strog cosistecy. Techical details are relegated to the Appedix. 2 Preiaries I this sectio, we itroduce redudacy models ad LLIC i the cotext of CCA. Let z = x, y be a q +p-dimesioal radom vector distributed accordig to a q +p-variate ormal 2

3 distributio with E[z] = µ = µ x, µ y, Cov[z] = Σ = Σ xx Σ xy Σ xy Σ yy, where µ x ad µ y are q- ad p-dimesioal mea vectors of x ad y, Σ xx ad Σ yy are q q ad p p covariace matrices of x ad y, ad Σ xy is the q p covariace matrix of x ad y. Suppose that j deotes a subset of ω = {,..., q} cotaiig q j elemets, ad x j deotes the q j -dimesioal radom vector cosistig of x idexed by the elemets of j. For example, if j = {, 2, 4}, the x j cosists of the first, secod, ad fourth elemets of x. Without loss of geerality, we ca express x as x = x j, x j, where x j is a q q j -dimesioal radom vector. The, for a subset j, the covariace matrices Σ xx ad Σ xy are expressed as follows: Σjj Σ j j Σ jy Σ xx =, Σ xy =, Σ j j Σ j j where the sizes of Σ jj, Σ j j, Σ jy, ad Σ jy are q j q j, q j q q j, q j p, ad q q j p. Let z,..., z be idepedet radom vectors from z, ad let z be the sample mea of z,..., z give by z = i= z i ad S be the usual ubiased estimator of Σ give by S = i= z i zz i z. I the same way as Σ, we also partitio S as follows: S jj S j j S jy S xx S xy S = =. S xy S yy Σ jy S j j S j j S jy S jy S jy S yy From Fujikoshi 982, x j is redudat if the followig equatio holds: trσ xx Σ xy Σ yy Σ yx = trσ jj Σ jyσ yy Σ jy. The left-had side i expresses the sum of squares of the caoical correlatio coefficiets betwee x ad y, ad the right-had side expresses the sum of squares of the caoical correlatio coefficiets betwee x j ad y. I particular, we ote that is equivalet see, Fujikoshi, 982 to Σ jy j = O q qj,p, 2 where Σ ab c = Σ ab Σ ac Σ cc Σ cb, ad O q qj,p is the q q j p matrix of zeros. We regard a subset j as the cadidate model such that x j is redudat. Followig Fujikoshi 985, the cadidate model j such that x j is redudat is expressed as j : S W p+q, Σ s.t. trσ xx Σ xy Σ yy Σ yx = trσ jj Σ jyσ yy Σ jy. 3 Let J be a set of cadidate models. We the separate J ito two sets, oe is a set of overspecified models J + ad the other is a set of uderspecified models J, which are defied by J + = {j J trσ xx Σ xy Σ yy Σ yx = trσ jj Σ jyσ yy Σ jy}, J = J \J +. The, the true model or subset j ca be regarded as the smallest overspecified model, i.e., j = arg mi j J+ q j. For simplicity, we write q j as q. A estimator of Σ i 3 is give by ˆΣ j = arg mi Σ F S, Σ subject to trσ xx Σ xy Σ yy Σ yx = trσ jj Σ jyσ yy Σ jy, 3

4 where F S, Σ is the discrepacy fuctio based o Stei s loss fuctio give by F S, Σ = {trσ S log Σ S p + q}. By usig F S, ˆΣ j, LLIC i 3 is defied as LLICj = F S, ˆΣ j + mj = log S yy j S yy ω + mj, where S yy j = S yy S jy S jj S jy, ad mj is the pealty term i 3. For simplicity, we write S yy ω as S yy x. By choosig mj i various quatities, we ca express the followig criteria as special cases of LLIC: p 2 + q 2 + p + q + 2pq j AIC 2 p + q j p q j 2 + q q 2 q j q j 2 p + q AIC c { } p + qp + q + mj = pq q j log BIC { 2 } p + qp + q + pq q j + log CAIC { 2 } p + qp + q + 2 pq q j log log HQC 2 Note that LLIC is the same as GIC whe mj is expressed as the umber of parameters i 3 multiplied by the stregth of the pealty. The best subset ĵ selected by LLIC is give by ĵ = arg mi j J LLICj.. 3 Mai results I this sectio, we give the sufficiet coditios for strog cosistecy of LLIC uder the HD asymptotic framework. First, we preset some lemmas which are required for derivig the strog cosistecy coditios, with proofs provided i the Appedix. Lemma 3. Suppose that p is fixed or p = p. Let ĵ = arg mi j J LLICj, ad let h j,l be some positive costat ot covergig to 0 for j, l J. The, we have l J \{j}, h j,l {LLICl LLICj} τ j,l > 0, a.s. P ĵ j =. From Lemma 3., to obtai sufficiet coditios such that LLIC is strogly cosistet, we may derive the almost sure covergece of h j,l {LLICj LLICj } for all j J \{j }. Hece, we derive the covergece by usig the followig lemma. Lemma 3.2 Let p = p ad let r be a atural umber ot relyig o. Suppose that t ad T 2 are a radom variable ad a r r matrix satisfyig [ E t E[t ] 2k] = Op k 2k k N, E [ T 2 E[T 2 ] 4] = O 2, where A is the Frobeius orm for a matrix A. The for all ε > 0, we have p /2 ε t E[t ] = o, a.s., 4 3/4 ε T 2 E[T 2 ] = o, a.s. 5 4

5 Before givig the strog cosistecy coditios, let us prepare some otatio. j J, let a o-cetrality parameter ad a p q q j matrix be deoted by For a subset δ j = log I p + Γ j Γ j, Γ j = Σ /2 yy ω Σ jy j Σ /2 j j j. 6 As well as S yy x, we write Σ yy ω as Σ yy x. From 2, we observe that δ j = 0 ad Γ j = O q qj,p hold if ad oly if j J +. Hece, we derive the sufficiet coditios i each case of j J + \{j } ad j J. By usig this otatio ad these lemmas, we derive the sufficiet coditios for strog cosistecy of LLIC the proof is give i Appedix C. Theorem 3. GIC is strogly cosistet as p = p,, p/ c [0,, if the followig two coditios are satisfied simultaeously: C :, p/ c p/2 ε C2 : j J,, p/ c where δ j is defied i 6. {q j q p log p + mj mj [ δ j + q j q log p + ] {mj mj } > 0, } > 0 for some ε 0 < ε /2, We ote that the sufficiet coditios for strog cosistecy by Theorem 3. are similar to those for weak cosistecy accordig to Yaagihara et al. 207 uder the HD asymptotic framework. By usig Theorem 3., we derive the sufficiet coditios for strog cosistecy of several criteria. The proof is omitted because it ca be foud i Yaagihara et al Corollary 3. As p = p,, p/ c [0,, the sufficiet coditios for strog cosistecy of several criteria are givig as follows: Whe c = 0, AIC, AIC c, BIC, CAIC, ad HQC are strogly cosistet. 2 Whe c > 0, i AIC is strogly cosistet, if c < c a ad q q j < { 2 c δ j + q j q c } log c, p/ c j J, 7 where c a is the solutio of x log x + 2 = 0. Especially, if δ j, 7 holds. ii AIC c is strogly cosistet if q q j < c2 2 c Especially, if δ j, 8 holds. iii BIC ad CAIC are strogly cosistet if q q j < c Especially, if δ j / log, 9 holds. { c δ j + q j q c } log c, p/ c, p/ c δ j log j J. 8 j J. 9 5

6 iv HQC is strogly cosistet if q q j < 2c, p/ c Especially, if δ j / log log, 0 holds. δ j log log j J. 0 From Corollary 3., uder the HD asymptotic framework we observe that the coditios for strog cosistecy of AIC, AIC c, BIC, CAIC, ad HQC are equivalet to those for weak cosistecy derived by Yaagihara et al Ackowledgmets Ryoya Oda was supported by a Research Fellowship for Youg Scietists from the Japa Society for the Promotio of Sciece. Hirokazu Yaagihara ad Yasuori Fujikoshi were partially supported by Grats-i-Aid for Scietific Research C from the Miistry of Educatio, Sciece, Sports, ad Culture, #8K0345 ad #6K00047, respectively. Appedix A Proof of Lemma 3. From the assumptio of Lemma 3., the followig reductios ca be derived: { = P LLICl LLICj τ j,l h k= m= =m j,l < } k { = P k + τ j,l < LLICl LLICj < } h k= m= =m j,l k + τ j,l { P k + τ j,l < } LLICl LLICj h k= m= =m j,l { } P LLICl LLICj > τ j,l h m= =m j,l { } P LLICl LLICj > 0. h j,l m= =m 6

7 Hece, we have P ĵ j = P = P = P k= m= =m = P =. k= m= =m m= =m l J \{j} This completes the proof of Lemma 3.. { ĵ j < } k { ĵ j } k {ĵ j} l J \{j} m= =m P m= =m {LLICl < LLICj} {LLICl LLICj < 0} B Proof of Lemma 3.2 Let us take a arbitrary ε > 0, ad let k be a atural umber such that k > 2ε. By usig Markov s iequality, for all δ > 0, we have P p /2 ε t E[t ] > δ P 3/4 ε T E[T ] > δ p /2+ε δ 2k E[t E[t ] 2k ] = Op 2kε, 3/4+ε δ 4 E[ T E[T ] 4 ] = O ε. The, sice p = p ad k > 2ε, it holds that = p 2kε < ad = ε <. These equatios ad the Borel-Catelli lemma complete the proof of Lemma 3.2. C Proof of Theorem 3. To prove Theorem 3., we use three lemmas from Yaagihara et al. 207 ad Oda & Yaagihara 209. Before Lemma C. is itroduced, let Q be a matrix satisfyig I = QQ ad Q Q = I, where is the -dimesioal vector of oes. Further, let X = x,..., x, where x i is the i-th idividual from x. The followig lemma is Lemma C. by Yaagihara et al Lemma C. For a subset j J, let E, A j, ad B j be mutually idepedet radom matrices, which are distributed accordig to E N p O,p, I p I, A j N q qj O,q qj, I q qj I, B = Q X = B j, B j N q O,q, Σ xx I, 7

8 where E ad B are idepedet ad do ot rely o j, ad B j : q j. The, we have S yy x = Σ /2 yy xe I P EΣ /2 yy x, S yy j = Σ /2 yy xa j Γ j + E I P j A j Γ j + EΣ /2 yy x, where P = BB B B, P j = B j B j B j B j, ad Γ j is defied i 6. The followig lemma is give by usig 23 ad B.6 i Yaagihara et al Lemma C.2 For a subset j J, let U ad U 2 be idepedet radom matrices distributed accordig to U N q p O q p, I p I q, U 2 N q qj po q qj p, I p I q qj. Further, let W ad W 2 be radom matrices distributed accordig to W W q qj p + q 2q j, I q qj, W 2 W q qj p + q 2q j, I q qj. The, we have where δ j is defied i 6. log S yy j S yy x = δ j + log U U + U 2U 2 U U + log W W 2, The followig lemma is Lemma C.2 i Oda & Yaagihara 209. Lemma C.3 Suppose that N 4k > 0 for k N. Let u ad v be idepedet radom variables distributed accordig to u χ 2 N ad v χ 2 p. The, we have [ v E u p ] 2k = Op k N 2k. N 2 First, we cosider the case of j J + \{j }. The distict elemets of j\j deote a,..., a qj q. Let j 0 = j, j i = j i \{a i } i q j q. The, j qj q = j holds, ad we ca express LLICj LLICj as follows: LLICj LLICj = log S yy j S yy j + mj mj q j q = log S yy j i S yy ji i= + mj mj. C. The, from Lemma C., S yy ji ca be expressed as follows: S yy ji = Σ /2 yy xe I P ji EΣ /2 yy x, C.2 where E N p O,p, I p I, P ji = B ji B j i B ji B j i, B ji N qj i+o,qj i+, Σ ji j i I, ad E is idepedet of B ji. Moreover, by applyig Lemma C. to S yy ji, we have S yy ji = Σ /2 yy xe I P ji EΣ /2 yy x, C.3 8

9 where P ji = B ji B j i B ji B j i, ad B ji is the q j i sub matrix of B ji = B ji, b ji. Let V i, = E I P ji E, V i,2 = E P ji P ji E. C.4 Sice I P ji P ji P ji = O, holds, we observe that V i, ad V i,2 are idepedet, ad V i, W p q j + i 2, I p, V i,2 W p, I p from a property of the Wishart distributio ad Cochra s Theorem see, e.g., Fujikoshi et al., 200, Theorem By usig C.2, C.3, ad C.4, we have S yy ji S yy ji = E I P ji E E I P ji E = E I P ji E + E P ji P ji E E I P ji E = V i, + V i,2. C.5 V i, Sice V i,2 W p, I p, we ca express V i,2 = v i v i, where v i N p 0 p, I p ad v i is idepedet of V i,. The, C.5 is calculated as S yy ji S yy ji = I p + V i, V i,2 = + v iv i, v i = + v i 2 vi v i V i, v i v i. C.6 Let ṽ i = v i 2 ad ũ i = v i v i V i, v i v i. The, from a property of the Wishart distributio see, e.g., Fujikoshi et al., 200, Theorem 2.3.3, we see that ṽ i ad ũ i are idepedet, ad ṽ i χ 2 p ad ũ i χ 2 p q j + i. The, C.6 is expressed as S yy ji S yy ji = ṽi ũ i. From Lemma C.3, by applyig 4 i Lemma 3.2 to the above equatio, for all ε > 0 0 < ε /2, the followig equatio ca be derived: + log S yy j i S yy ji = log = log From the above equatio, we have p p q j + i 3 + op/2+ϵ p + op/2+ε, a.s. log S q j q yy j S yy j = log S yy j i S i= yy ji = q j q log p + op /2+ε, a.s. C.7 Therefore, from C. ad C.7, we ca expad p {LLICj LLICj } as follows: p {LLICj LLICj } = q j q p log p + mj mj + op /2+ε, a.s. C.8 9

10 Next, we cosider the case of j J. By usig Lemma C.2, we have log S yy j S yy x = δ j + log U U + U 2U 2 U U + log W W 2, C.9 where U, U 2, W, ad W 2 are defied i Lemma C.2. Let Ũ = U 2 U 2 /2 {U 2 U U U 2} U 2 U 2 /2. From a property of the Wishart distributio, we observe that Ũ ad U 2 are idepedet ad Ũ W q qj p q j, I q qj. The, C.9 is expressed as log S yy j S yy x = δ j + log I q qj + Ũ U 2 U 2 + log W W 2. C.0 By a simple calculatio, we ca ote that E[ Ũ E[Ũ] 4 ] = O 2, E[ U 2 U 2 E[U 2 U 2] 4 ] = Op 2, ad E[ W E[W ] 4 ] = O 2. Hece, we ca apply 5 i Lemma 3.2 to Ũ, U 2U 2, W, ad W 2. From Taylor expasio, for all δ > 0 0 < δ < /4 the followig equatios ca be derived: log I q qj + Ũ U 2 U 2 = q q j log From C.0-C.2, we have p + o p 3/4+δ + o p 5/4+δ, a.s., C. log W W 2 = o /4+δ, a.s. C.2 log S yy j S yy x = δ j + q q j log + o, a.s. p C.3 Therefore, from C.7 ad C.3, we ca expad {LLICj LLICj } as follows: {LLICj LLICj } = { log S yy j S yy x + log S yy x = δ j + q j q log p } S yy j + mj mj Lemma 3., C.8, ad C.4 complete the proof of Theorem {mj mj } + o, a.s. C.4 Refereces [] Akaike, H Iformatio theory ad a extesio of the maximum likelihood priciple. I 2d Iteratioal Symposium o Iformatio Theory eds. B. N. Petrov & F. Csáki, pp Akadémiai Kiadó, Budapest. [2] Akaike, H A ew look at the statistical model idetificatio. Istitute of Electrical ad Electroics Egieers. Trasactios o Automatic Cotrol AC 9, [3] Bozdoga, H Model selectio ad Akaike s iformatio criterio AIC: the geeral theory ad its aalytical extesios. Psychometrika, 52,

11 [4] Fukui, K Cosistecy of log-likelihood-based iformatio criteria for selectig variables i high-dimesioal caoical correlatio aalysis uder oormality. Hiroshima Math. J., 45, [5] Fujikoshi, Y A test for additioal iformatio i caoical correlatio aalysis. A. Ist. Statist. Math., 34, [6] Fujikoshi, Y Selectio of variables i discrimiat aalysis ad caoical correlatio aalysis. I Multivariate Aalysis VI ed. P. R. Krishaiah, , North-Hollad, Amsterdam. [7] Fujikoshi, Y., Shimizu, R. & Ulyaov, V. V Multivariate Statistics: High- Dimesioal ad Large-Sample Approximatios. Joh Wiley & Sos, Ic., Hoboke, New Jersey. [8] Haa, E. J. & Qui, B. G The determiatio of the order of a autoregressio. J. Roy. Statist. Soc. Ser. B, 26, [9] McKay, R. J Variable selectio i multivariate regressio: a applicatio of simultaeous test procedures. J. Roy. Statist. Soc., Ser. B, 39, [0] Nishii, R., Bai, Z. D. & Krishaiah, P. R Strog cosistecy iformatio criterio for model selectio i multivariate aalysis. Hiroshima Math. J., 8, [] Oda, R. & Yaagihara, H A fast ad cosistet variable selectio method for highdimesioal multivariate liear regressio with a large umber of explaatory variables. TR No. 9, Statistical Research Group, Hiroshima Uiversity. [2] Ogura, T A variable selectio method i pricipal caoical correlatio aalysis. Comput. Statist. Data Aal., 54, [3] Srivastava, M. S Methods of Multivariate Statistics. Joh Wiley & Sos, New York. [4] Schwarz, G Estimatig the dimesio of a model. A. Statist., 6, [5] Timm, N. H Applied Multivariate Aalysis. Spriger-Verlag, New York. [6] Yaagihara, H., Oda, R., Hashiyama, Y. & Fujikoshi, Y High-Dimesioal asymptotic behaviors of differeces betwee the log-determiats of two Wishart matrices. J. Multivariate Aal., 57,