TOWARDS THE GLOBAL SOLUTION OF THE MAXIMAL CORRELATION PROBLEM

Transcription

1 TOWARDS THE GLOBAL SOLUTION OF THE MAXIMAL CORRELATION PROBLEM LEI-HONG ZHANG, LI-ZHI LIAO, AND LI-MING SUN Abstract. The maxmal correlaton problem (MCP) amng at optmzng correlaton between sets of varables plays a very mportant role n many areas of statstcal applcatons. Currently, algorthms for the general MCP stop at solutons of the multvarate egenvalue problem (MEP) for a related matrx A. The MEP s a necessary condton for the global solutons of the MCP. Up to date, there s no algorthm that can guarantee convergence to a global maxmzer of the MCP, whch would have sgnfcant mpact n applcatons. Towards the global solutons of the MCP, we have obtaned four results n the present paper. Frst, the suffcent and necessary condton for global optmalty of the MCP when A s a postve matrx s extended to nclude A beng a nonnegatve rreducble matrx. Secondly, the unqueness of the multvarate egenvalues n the global maxma of the MCP s proved ether when there are only two sets of varables nvolved, or when A s nonnegatve rreducble, regardless of the number of sets of varables. The unqueness of the global soluton of the MCP for the latter case s also proved. These theoretcal achevements lead to our thrd result that ether the Horst-Jacob algorthm or the Gauss-Sedel algorthm converges globally to a global maxmzer of the MCP as long as A s a nonnegatve rreducble matrx. Lastly, some new estmates of the multvarate egenvalues related to the global maxma are obtaned. Key words. multvarate statstcs, canoncal correlaton, maxmal correlaton problem, multvarate egenvalue problem, power method, Gauss-Sedal method, global maxmzer, the nonnegatve rreducble matrx AMS subject classfcatons. 62H20, 15A12, 65F10, 65K05 1. Introducton. Assessng the relatonshp between sets of random varables arses from lots of applcatons, among whch are cluster analyss, data classfcaton, pattern recognton, prncpal component analyss, and bonformatcs. Canoncal correlaton analyss (CCA) s an mportant and effectve tool for that purpose; see for example, [5, 6, 7, 10, 13, 14]. As early as 1935, based on the observaton of many real world examples and phenomena, Hotellng studed how to fnd the lnear combnaton of one set of varables that correlates maxmally wth the lnear combnaton of another set of varables, whch later s known as the maxmal correlaton problem (MCP) [8, 9]. If the optmal lnear combnaton s successfully found, we then have the advantage of usng one set of varables to predct the other. Optmzng correlaton between m > 2 sets of varables can be easly generalzed from the case of two sets of varables, and we recommend [2, Secton 2] for a bref but comprehensve background nformaton about the MCP. For the purpose of dscusson n the present paper, t s enough to only ntroduce the notaton and the optmzaton problem that s related to the MCP. Let A R n n be a symmetrc and postve defnte matrx, and P = {n 1, n 2,, n m } (1.1) be a set of postve ntegers wth m =1 n = n. Partton A and a vector x R n nto block forms accordng to P as follows, A 11 A 12 A 1m A 21 A 22 A 2m A = Rn n, (1.2) A m1 A m2 A mm x = [x 1,...,x m] R n, (1.3) Department of Mathematcs, Hong Kong Baptst Unversty, Kowloon Tong, Kowloon, Hong Kong, P. R. Chna. (longzlh@gmal.com). Department of Mathematcs, Hong Kong Baptst Unversty, Kowloon Tong, Kowloon, Hong Kong, P. R. Chna. (llao@hkbu.edu.hk). Department of Mathematcs, Hong Kong Baptst Unversty, Kowloon Tong, Kowloon, Hong Kong, P. R. Chna. (lmsun@nau.edu.cn). 1

2 wth A j R n nj and x R n, respectvely. The optmal soluton to the MCP then corresponds to the global maxmzer of the followng equalty constraned optmzaton problem: { Maxmze r(x) := x Ax (1.4) subject to x 2 = 1, = 1, 2,, m. Denote the constrant set as M := {x R n x 2 = 1, x R n, for = 1, 2,, m.}. (1.5) It s clear that maxmzng r(x) on M s equvalent to maxmzng x (A + ci [n] )x on M for any c R, where I [n] stands for the dentty matrx of order n. Therefore, no generalty s lost when A s assumed to be postve defnte. The frst order necessary optmal condton for (1.4) s the exstence of real scalars λ 1,..., λ m (the Lagrange multplers) and a vector x R n such that the system of equatons { Ax = Λx, (1.6) x 2 = 1, = 1, 2,, m, s satsfed [2, 14], where Λ := dag{λ 1 I [n1], λ 2 I [n2],, λ m I [nm] }. (1.7) The multvarate egenvalue problem (MEP) s cast exactly as the system of equatons (1.6), and because of the analogy of the role of Λ n the MEP wth the classcal egenvalue, λ 1,..., λ m are usually called the multvarate egenvalues. Rather than beng a necessary condton for the MCP, recently, from an entrely dfferent and non-statstcal settng, the MEP tself fnds applcatons n the perturbaton analyss of lnear dynamcal systems subject to addtve bounded noses [15]. Thnkng the par (x, Λ) that solves (1.6) as the soluton of the MEP, Chu and Watterson [2] dscovered that there are precsely m =1 (2n ) solutons for the generc matrx A whose n egenvalues are dstnct. It can also be verfed from (1.6) that the objectve functon value r(x) at the soluton par (x, Λ) of the MEP s r(x) = m =1 λ, and the global optmal soluton of the MCP corresponds to the one wth the largest m =1 λ. Moreover, t s nterestng to note that when reducng the MEP to the classcal egenvalue problem by settng m = 1, λ 1 becomes the objectve functon value and s exactly the largest egenvalue of A. However, t must be ponted out, on the one hand, that the case m > 1 has essental dfferences from the classcal egenvalue problem. Frst, the MCP wth m = 1 s to maxmze the Raylegh quotent on the sphere, whch has no other local maxmzer except for the global one [3], that s, any local maxmzer s a global maxmzer. When m > 1, dfferent stuaton occurs. It has been observed that both global and local maxma exst (see [2] and [15]). Secondly, even though the global objectve functon value r s unque n both m = 1 and m > 1, there may exst dfferent combnatons of λ 1,...,λ m wth the same sum r = m =1 λ when m > 1 (see Example 1 n Secton 3). Therefore, a very nterestng and mportant queston arsng here s how many combnatons of λ 1,..., λ m, or smply Λ, exst n the global solutons par (x, Λ ). One of our four man contrbutons n ths paper s to show, based on a suffcent condton we establshed, that Λ s unque ether when m = 2, or when A s a nonnegatve rreducble matrx. Moreover, t s further shown that the global maxmzer of the MCP wth a nonnegatve rreducble matrx A s also essentally unque. Ths achevement s a step towards better understandng the MCP and the MEP, and may gude the desgn of the numercal algorthm for solvng the MCP successfully. Durng our nvestgaton, on the other hand, we also see many analoges between the classcal egenvalue problem wth the MEP, especally when A s a nonnegatve rreducble matrx. A 2

3 correspondng Collatz-Welandt formula for the MEP s establshed and many nce analoges are summarzed n Secton 5. When turnng to the computatonal ssue, we are dsapponted to note that up to date, no algorthm s able to solve the MCP for the general case. All teratve algorthms avalable are manly centered around satsfyng the necessary condton (1.6), the MEP. The earlest teraton, the Horst-Jacob algorthm [6], whch s summarzed as Algorthm 1 below, s of Jacob-type recurrence structure and s proved to converge monotoncally to a soluton of the MEP n [2]. An mprovement, the Gauss-Sedel algorthm [2] summarzed as Algorthm 2, s developed by adoptng the Gauss- Sedel-type teraton and s proved to converge monotoncally to a soluton of the MEP n [16] recently. However, numercal experments show that for ether algorthm, the computed soluton depends closely on the startng pont, and f the startng pont s not specally selected, there s a hgh probablty that the solutons found are local maxma [16]. Wthout the global maxmzer, the maxmal correlaton would not be establshed, makng the statstcal predcton less relable. Algorthm 1 The Horst-Jacob algorthm for the MEP [6]. Gven x (0) R n, for k = 0, 1,, do for = 1, 2,, m do y (k) λ (k) x (k+1) end for end for := m j=1 A jx (k) j := y (k) 2 := y(k) λ (k) Algorthm 2 The Gauss-Sedel algorthm for the MEP [2]. Gven x (0) R n, for k = 0, 1,, do for = 1, 2,, m do y (k) λ (k) x (k+1) end for end for := 1 j=1 A jx (k+1) j := y (k) 2 := y(k) λ (k) + m j= A jx (k) j In order to obtan the global maxmzer for the MCP, efforts were made along two dfferent lnes n the lterature. The frst effort s to establsh the global optmal condtons for the MCP. The frst pertanng result perhaps was made n [5] where t was shown that f m = 2 or f m > 2 wth A beng a postve matrx, a soluton x to the MEP s a global soluton for the MCP f and only f A Λ s negatve sem-defnte. In general, even for m = 3, example was gven n [5] to llustrate the complcated stuaton (see also Example 1 n Secton 3). Most recently, [16] further establshes a necessary global optmal condton for the general case. In partcular, t was shown that f x s a global maxmzer, the correspondng multvarate egenvalue λ, for = 1, 2,, m, s not less than the largest egenvalue of A,.e., λ σ 1 (A ), and hence, the matrx D Λ s negatve sem-defnte, where D = dag {A 11,...,A mm } R n n, (1.8) 3

4 s the block dagonal matrx of A. Our second contrbuton n ths paper s also made along ths lne by extendng the global optmal condton for the postve matrx to the nonnegatve rreducble matrx. In partcular, we show that when A s a nonnegatve rreducble matrx, the negatve semdefnteness of A Λ s stll a necessary and suffcent condton for a soluton par (x, Λ ) of the MEP to be a global soluton for the MCP. The mportance of ths global optmal condton s that, when A s nonnegatve rreducble, we are able to prove that ether the Horst-Jacob algorthm or the Gauss-Sedel algorthm converges globally to a global maxmzer of the MCP. Ths s our thrd contrbuton of ths paper. The other effort towards the global solutons of the MCP was made n order to establsh some estmaton for the global soluton, whch would help to solve the MCP successfully. The estmaton, λ σ 1 (A ), for = 1, 2,, m, establshed n [16] s also along ths lne. Ths result then leads to settng up an effectve startng pont strategy for both the Horst-Jacob algorthm and the Gauss-Sedel algorthm. Startng from the ntal pont x (0) = [(x (0) 1 ), (x (0) 2 ),, (x (0) m ) ] R n, where x (0) R n s the unt egenvector correspondng to the largest egenvalue of A R n n, both algorthms demonstrate better numercal performance n teraton numbers and n boostng up sgnfcantly the probablty of fndng a global maxmzer of (1.4) as well. As far as the estmate λ σ 1 (A ) tself s concerned, however, t s apparently a relaton between the multvarate egenvalues wth only the block dagonal matrces A. Therefore, some new estmates can also be expected by pckng up the nformaton n the off-dagonal block A j R n nj, j. Our last contrbuton of ths paper s then to establsh some new estmates for λ by utlzng the nformaton of A as much as possble. The paper s organzed as follows. In the next secton, we wll extend the global optmal condton of the MCP from the postve matrx [5] to the nonnegatve rreducble matrx A, by establshng a necessary and suffcent condton. In Secton 3, we wll dscuss the unqueness of the multvarate egenvalues and show that Λ s unque for the global solutons of the MCP (1.4) when m = 2, or when A s a nonnegatve rreducble matrx. The unqueness of the global maxmzer for the latter case s also proved. In Secton 4, we shall prove the global convergence of ether the Horst- Jacob algorthm or the Gauss-Sedel algorthm to a global maxmzer whenever A s a nonnegatve rreducble matrx. Secton 5 s dedcated to the comparson between the classcal egenvalue problem and the MEP when A s nonnegatve rreducble. In Secton 6, we shall manly develop some new estmates of multvarate egenvalues related to the global maxma of the MCP. The last secton then concludes the paper by pontng out some further research drectons. 2. Global optmalty for the MCP wth nonnegatve rreducble A. Up to now, two teratve algorthms (the Horst-Jacob algorthm and the Gauss-Sedel algorthm) have been proved to converge monotoncally to solutons of the MEP. However, none of these algorthms can guarantee that the soluton found s a global maxmzer of the MCP. Therefore, whenever a soluton of the MEP s n hand, we should rely on some global optmal condtons to check the optmalty for MCP so that the statstcal predcton s relable. The frst effort of establshng the global optmal condtons was made by Hanaf and Ten Berge [5] and we frst restate ther results as follows. Theorem 2.1. Suppose (x, Λ ) s soluton par to the MEP (1.6). For m = 2, or m > 2 wth a postve matrx A, x s a global maxmzer of the MCP (1.4) f and only f A Λ s negatve sem-defnte. To generalze ths theorem, we provde the defnton of the nonnegatve rreducble matrx together wth some useful lemmas. Defnton 2.2. B R n n s sad to be a reducble matrx when there exsts a permutaton matrx P R n n such that ( ) P B1 B BP = 2, where B 0 B 1 and B 3 are both square. 3 4

5 Otherwse B s sad to be an rreducble matrx. Lemma 2.3. Suppose B R n n s an rreducble matrx, then for any dagonal matrx Υ R n n, B + Υ s also rreducble. Proof. Suppose by contradcton that B + Υ s reducble whch mples that a permutaton matrx P exsts such that ( ) P (B + Υ)P = P B1 B BP + Υ = 2, where Υ s dagonal. 0 B 3 Therefore, P BP = ( ) ( ) B1 B 2 B1 Υ Υ = 1 B 2, 0 B 3 0 B 3 Υ 3 mplyng B s reducble by Defnton 2.2. Ths contradcton completes the proof. The followng theorem s very useful n analyzng the global optmalty when A s a nonnegatve rreducble matrx. ρ(b) here represents the spectral radus of a matrx B. Theorem 2.4. (The Perron-Frobenus Theorem, [12]) Let B R n n be a nonnegatve rreducble matrx. Then the followng statements are vald. (1) ρ(b) > 0. (2) ρ(b) s an egenvalue of B. (3) There exsts a postve egenvector of B correspondng to the egenvalue ρ(b). (4) The egenvalue ρ(b) s smple. Wth the help of ths crucal theorem, we have another mportant lemma. Lemma 2.5. Suppose B R n n s symmetrc and nonnegatve rreducble. Then for any dagonal matrx Υ = dag{υ 11, Υ 22,, Υ nn } R n n, there s a unque unt postve egenvector correspondng to the largest egenvalue σ 1 (B+Υ) of B+Υ. Moreover, the largest egenvalue σ 1 (B+Υ) s a smple egenvalue. Proof. Let ζ := max{ Υ, ρ(b + Υ)}. Obvously, the matrx B+Υ+2ζI [n] s nonnegatve rreducble by Lemma 2.3, and ρ(b+υ+2ζi [n] ) = σ 1 (B +Υ)+2ζ. By Theorem 2.4, ρ(b +Υ+2ζI [n] ) s a smple egenvalue, so s σ 1 (B +Υ) for B +Υ; moreover, there s a unque unt postve egenvector b correspondng to ρ(b + Υ + 2ζI [n] ), that s, (B + Υ + 2ζ)b = (B + Υ)b + 2ζb = σ 1 (B + Υ)b + 2ζb, or (B + Υ)b = σ 1 (B + Υ)b. On the other hand, f b s a unt postve egenvector assocated wth σ 1 (B + Υ), then b = b by the unqueness of a unt postve egenvector for B + Υ + 2ζI [n]. Ths proves our clam. Wth the ad of these results, we are able to establsh the global optmal condton for the MCP wth a nonnegatve rreducble matrx. Theorem 2.6. Suppose A R n n s symmetrc postve defnte and nonnegatve rreducble, and suppose (x, Λ ) s a soluton par to the MEP (1.6). Then x s a global maxmzer of the MCP (1.4) f and only f A Λ s negatve sem-defnte. Proof. It has been proved n [5] that the negatve sem-defnteness of A Λ s a suffcent global optmal condton for the MCP (1.4). For the necessary part, we frst denote x as the vector whose elements are the absolute values of x s. Observng that x M and r( x ) = x A x x Ax = r(x ), 5

6 we know x s also a global maxmzer of the MCP (1.4), and thereby, there s a dagonal matrx, say Ω, such that A x = Ω x, Ω = dag{ω 1 I [n1], ω 2 I [n2],, ω m I [nm] } R n n. (2.1) Observe that Ax = Λ x yelds n a j x (j) = Λ x (), = 1, 2,, n, j=1 where x (j) R represents the j-th element of x to dstnct the block vector x j Rnj. Therefore, n n a j x (j) a j x (j) = Λ x (), = 1, 2,, n, (2.2) j=1 j=1 snce λ σ 1(A ) > 0; see [16]. Consequently, t follows from (2.1) and (2.2) that Ω x = A x e Λ x, or (Ω Λ ) x e 0. (2.3) Here for two vectors a and c R n, the symbol a e c means a c for all = 1, 2,, n. The nequalty (2.3) together wth x 2 = 1 for = 1, 2,, m, mmedately mples ω λ, = 1, 2,, m. (2.4) On the other hand, t follows that m =1 λ = m =1 ω = r(x ), whch together wth (2.4) leads to Λ = Ω. Now, f A Λ s not negatve sem-defnte, then the largest egenvalue σ 1 (A Λ ) > 0. By Lemma 2.5, there s a unque unt postve egenvector b correspondng to σ 1 (A Λ ). Snce x and b are two egenvectors correspondng to dfferent egenvalues 0 and σ 1 (A Λ ) > 0, respectvely, t must follow that x b = 0. Ths s a contradcton because b > e 0, x e 0 and x 0. Therefore, we clam that A Λ s negatve sem-defnte and complete the proof. Theorem 2.6 and the arguments n ts proof lead to an mportant global convergence result of ether Algorthm 1 or Algorthm 2, whch we shall dscuss n more detals n Secton Unqueness of the multvarate egenvalues. For the MCP (1.4), t s clear that the global maxmzer x s not unque snce ±x both reach r(x ). However, we wll show n ths secton that when m = 2 or when A s a nonnegatve rreducble matrx, the correspondng dagonal matrx Λ s unque. Because of ths unqueness, we then can call Λ the domnant multvarate egenvalue of the MEP for these two cases. We provde a suffcent condton for the unqueness of Λ frst. Theorem 3.1. If there s a global maxmzer x of the MCP such that the related matrx A Λ s negatve sem-defnte, then any global maxmzer of the MCP s related to the unque multvarate egenvalues. Proof. Suppose (x, Λ) s another arbtrary soluton par to the MEP such that r(x) = r(x ). Note that x (A Λ )x = r(x) r(x ) = 0, whch together wth the negatve sem-defnteness of A Λ mples that x s also an egenvector correspondng to the largest egenvalue σ 1 (A Λ ) = 0,.e., (A Λ )x = 0, or Ax = Λ x. 6

7 Moreover, t s clear that Ax = Λx. Therefore, we have (Λ Λ )x = 0. Snce x 0, for = 1, 2, m, t must follow that Λ = Λ, and completes the proof. Based on the global optmalty for the case m = 2 and the case when A s nonnegatve rreducble, the unqueness of Λ n the global maxma of the MCP turns out to be evdent. Moreover, the followng corollary s another drect consequence of Theorem 3.1. Corollary 3.2. If there are two global soluton pars (x, Λ) and (x, Λ ) of the MCP wth Λ Λ, then nether A Λ nor A Λ s negatve sem-defnte. It s nterestng to note that n general, nevertheless, the unqueness of Λ cannot be guaranteed. A smple example s presented for demonstraton. Example 1. Consder the example when m = 3, P = {1, 1, 1} and A = R 3 3. (3.1) As a soluton par (x, Λ) to the MEP, λ for = 1, 2, 3 must satsfy λ 1 = 3 + x2 x 1 + x3 x 1, λ 2 = 4 + x1 x 2 x3 x 2, λ 3 = 5 + x1 x 3 x2 x 3. It then becomes very clear that max r(x) = max (12 + 2x x 3 2 x 3 ) = 14, x M x M x 1 x 1 x 2 for whch two global maxma are x = (1, 1, 1) and x = (1, 1, 1), wth the correspondng Λ = dag{5, 4, 5} and Λ = dag{3, 4, 7}, respectvely. Obvously, Λ Λ and both A Λ and A Λ are not negatve sem-defnte. Ths s also a smple example to llustrate that the global optmalty for the case m = 2 or m > 2 wth a nonnegatve rreducble matrx A s not vald for the general case any more. By randomly choosng 10 7 startng ponts wth elements from the unform [ 0.5, 0.5] dstrbuton for the Horst-Jacob algorthm as the replacement of the exhaustve search, the example gven n [5] only numercally shows the optmalty condton, Theorem 2.1, s volated, whereas Example 1 clearly llustrates ths. On the other hand, the unqueness of the global maxmzer of the MCP s also not guaranteed even for the nonnegatve matrx wth m = 2. Ths pont can be quckly verfed from the smple example where A = dag{i [n1], I [n2] } = I [n1+n2] for whch any x M becomes a global maxmzer. However, the case when A s nonnegatve rreducble enjoys addtonal nterestng propertes and deserves extra observaton. Theorem 3.3. Suppose A s nonnegatve rreducble, then there are only two soluton pars, (x, Λ ) and ( x, Λ ) for the MEP, such that x and x are the global maxma for the MCP (1.4). Moreover, one of x and x must be a postve vector. Proof. Suppose (x, Λ ) s an arbtrary global soluton for the MCP. Accordng to Lemma 2.5 and Theorem 2.6, there s a unque postve egenvector wth norm m, correspondng to the smple egenvalue σ 1 (A Λ ) = 0 of the related matrx A Λ. Moreover, accordng to Theorem 3.1, Λ s unque n all global maxma of the MCP. Ths shows that x and x are the only two global solutons and one of them s a postve vector. 7

8 An nterestng open queston s whether Theorem 3.1 s also a necessary condton. If ths s true, we then have another necessary and suffcent condton to descrbe the global soluton of the MCP. Our current results establshed seem to support ths conjecture because the unqueness of Λ and the negatve sem-defnteness of A Λ happen smultaneously n ether the case m 2 or the case m > 2 wth a nonnegatve rreducble matrx A. Further research s needed to clarfy the relatonshp between the unqueness of Λ and the negatve sem-defnteness of A Λ. 4. Convergence to the global maxmzer. Suppose now a current soluton par (x, Λ) for the MEP s n hand. For m = 2, [5] has already specfed a numercal scheme to construct a better pont x M than x for the MCP,.e., r( x) > r(x), n case A Λ has a postve egenvalue. For the case when A s a nonnegatve rreducble matrx, a smlar result s certanly very demandng. Ths secton s dedcated to ths task. As a very nterestng and surprsng result, we can show that both the Horst-Jacob algorthm and the Gauss-Sedel algorthm are able to globally converge to a global maxmzer of the MCP whenever A s nonnegatve rreducble. Ths mportant convergence result turns out to be obvous after establshng the followng theorem. Theorem 4.1. Suppose A s a nonnegatve rreducble matrx, and x s a soluton for the MEP. If x s stll a soluton to the MEP, then x s a global maxmzer for the MCP (1.4). Proof. The proof s agan conducted by contradcton. If ( x, Ω) s a soluton par for the MEP, whle x s not a global maxmzer for the MCP, then by Theorem 2.6 and Lemma 2.5, t follows that the largest egenvalue σ 1 (A Ω) > 0 wth a postve unt egenvector b > e 0, that s, Moreover, t s noted that (4.1) and (4.2) then mply the followng relaton, whch s certanly a contradcton snce (A Ω)b = σ 1 (A Ω)b. (4.1) (A Ω) x = 0. (4.2) 0 = x (A Ω)b = σ 1 (A Ω) x b, σ 1 (A Ω) > 0, and x b > 0. Therefore, σ 1 (A Ω) must be zero, whch by Theorem 2.6 agan shows that x s a global maxmzer of the MCP and fnshes the proof. Based on Theorem 4.1 and Theorem 3.3, consequently, we can establsh an alternatve global optmal condton to Theorem 2.6, whch s obvously much more convenent and easer to check. Theorem 4.2. Suppose A R n n s symmetrc postve defnte and nonnegatve rreducble, and suppose x s a soluton to the MEP (1.6). Then x s a global maxmzer of the MCP (1.4) f and only f all elements of x are of the same sgn. Proof. The necessary part s already ensured by Theorem 3.3. For the suffcent part, we only need to observe that x s stll a soluton to the MEP whenever all elements of x are of the same sgn. Then Theorem 4.1 ensures the result. Remark 4.3. When A s nonnegatve rreducble, Theorem 3.3 not only shows the unqueness of Λ, but also the essental unqueness of the global maxmzer for the MCP (1.4), and we assume x > e 0; Theorem 4.2 along wth Theorem 3.3 further clams that there are no nonnegatve solutons for the MEP except for the global maxmzer x > e 0 of the MCP. By observng the detaled teraton n the Horst-Jacob algorthm (Algorthm 1), and the Gauss- Sedel algorthm (Algorthm 2), t s clear that f the startng pont x (0) e 0, the successve teratons 8

9 x (k) e 0 for k = 1, 2,. When A s generc, convergence of {x (k) } to a soluton of the MEP s guaranteed for ether algorthm ([2] and [16]), and thereby, convergence to a global soluton of the MCP when A s addtonally assumed to be nonnegatve rreducble then becomes clear. Wthout the generc assumpton of A, convergng to a global maxmzer reles on the unqueness of the postve global soluton x > e 0 as we shall see n the next theorem. Theorem 4.4. Suppose A s a nonnegatve rreducble matrx. Then for any startng pont x (0) e 0, ether the Horst-Jacob algorthm or the Gauss-Sedel algorthm converges to a global maxmzer x > e 0 of the MCP (1.4). Proof. Suppose {(x (k), Λ (k) )} s generated from ether the Horst-Jacob algorthm or the Gauss- Sedel algorthm. From the convergence proofs of both these algorthms (see [2] and [16]), t follows that any convergent subsequence, say {(x (kj), Λ (kj) )}, of {(x (k), Λ (k) )} converges to a correspondng soluton (x, Λ), of the MEP. Note that x (kj) e 0 mplyng x e 0. Theorem 4.2 and Theorem 3.3 therefore, ensure x = x and Λ = Λ. Ths shows {(x (k), Λ (k) )} converges to (x, Λ ). 5. The MEP versus the classcal egenvalue problem for nonnegatve matrces. In the prevous sectons, we have already seen the crucal role of the Perron-Frobenus Theorem n analyzng the MEP for nonnegatve rreducble matrces. In ths secton, we shall further summarze some nterestng analoges between the MEP and the classcal egenvalue problem when A s a nonnegatve rreducble matrx. It s well known that Perron (1907) dscovered that the spectral radus ρ(a) s a smple egenvalue for a postve matrx A wth a unque unt postve egenvector, the so-called Perron vector p > e 0, and Frobenus (1912) contrbuted substantal extensons of Perron s results to cover the case of nonnegatve matrces. Moreover, an alternatve characterzaton of the spectral radus ρ(a) and the Perron vector s the so-called Collatz-Welandt Formula, whch possesses the followng form n the context of nonnegatve matrces [11]. Theorem 5.1. If A s nonnegatve rreducble, then ρ(a) = max x N mn 1 j n,x(j) =0 [Ax] j x(j), where N = {x x e 0 and x 0}, (5.1) where [Ax] j and x(j) denote the j-th elements of Ax and x respectvely. For the MEP, smlar results also hold. Frst, the necessary and suffcent global optmal condton establshed for the postve matrx can be completely extended to the nonnegatve matrx. Second, the (unque) domnant multvarate egenvalue Λ has a unque unt postve egenvector x > e 0 for nonnegatve matrces. Thrd, the Collatz-Welandt Formula (5.1) also fnds a counterpart n the MEP and can be generalzed n the followng way. A straghtforward generalzaton s lke (5.2), whch takes the feature that the objectve functon value r(x) at a soluton of the MEP s the sum of the correspondng multvarate egenvalues. max x N m mn 1 j n,x (j) =0 =1 where A = [A 1, A 2,, A m ] R n n, = 1, 2,, m. [A x] j x (j), (5.2) However, we need to pont out a subtle flaw n (5.2) n that ts maxmum would possbly be nfnte. An example s enough to demonstrate ths flaw. Example 2. Consder the example when m = 2, P = {1, 1} and ( ) 1 1 A = 2 1 R 2 2. (5.3) 2 2 9

10 It s easy to verfy that n ths case max x N m =1 mn 1 j n,x (j) =0 [A x] j x (j) = max x N Ths flaw, nonetheless, can be quckly fxed by usng x(1) (3 + 2x(2) + x(2) 2x(1) ) +. N = {x x e 0, and x 2 = 1, = 1, 2,, m.} = N M (5.4) nstead of N, and the correspondng Collatz-Welandt Formula for the MEP can be descrbed as follows. Theorem 5.2. If A s nonnegatve rreducble, then the global maxmum r of the MCP (1.4) can be expressed as where N s gven by (5.4). r = m =1 Proof. For any x N, we denote Then t follows that and hence λ = max x N r (x) = m =1 mn 1 j n,x (j) =0 [A x] j mn 1 j n,x (j) =0 x (j). r (x)x e A x, [A x] j x (j), (5.5) r (x) x A x, and m m r (x) x A x = x Ax r. =1 =1 On the other hand, Theorem 3.3 ensures that x N acheves the maxmum,.e., m =1 r (x ) = r, and our clam (5.5) follows drectly. To sum up, we lst the detaled correspondences between the classcal egenvalue problem and the MEP for nonnegatve rreducble matrces n Table Estmaton of multvarate egenvalues. Let (x, Λ ) be a typcal soluton par to the MEP where Λ = dag{λ 1 I[n1],...,λ m I[nm] } and x s a global maxmzer for the MCP. We focus on the estmaton of λ, = 1, 2,, m, n ths secton. Recall that n [16], a lower bound λ σ 1 (A ), = 1, 2,, m, (6.1) has been establshed. Whle (6.1) has already shown ts mportance n settng up an effectve startng pont strategy for the Horst-Jacob algorthm and the Gauss-Sedel algorthm, the estmate tself manly uses the nformaton n the dagonal block matrces A s, gnorng the contrbuton of the off-dagonal block matrces A j s. To establsh some new estmates n terms of the nformaton n the off-dagonal block matrces of A, we frst establsh a general result n Theorem 6.1 whch s related wth the partton of the postve defnte matrx A. 10

11 Table 5.1 Correspondences between the classcal egenvalue problem and the MEP for nonnegatve rreducble matrces. The classcal egenvalue problem The MEP ρ(a) Λ The Perron vector, p > e 0, s the unque unt postve egenvector. x > e 0 s the unque postve soluton of the MEP. Ap = ρ(a)p A ρ(a)i [n] s negatve sem-defnte. σ 1(A ρ(a)i [n] ) = 0 s a smple egenvalue. ρ(a) = max x 2 =1 x Ax Ax = Λ x A Λ s negatve sem-defnte. σ 1(A Λ ) = 0 s a smple egenvalue. m =1 λ = maxx M x Ax The Collatz-Welandt Formula (5.1). The Collatz-Welandt Formula (5.5). Theorem 6.1. Suppose A R n n s symmetrc postve sem-defnte and has a partton (1.2), then max,j A j ( A 2 + A jj 2 ), (6.2) A j 2 A 2 A jj 2, ( j), (6.3) A j 2 = max A 2. (6.4) Proof. For (6.2), we consder a vector (we assume > j wthout loss of generalty.) v = (0,, 0, v, 0,, 0,v j, 0,, 0) R n, where v R n and v j R nj are the left and rght sngular vectors of A j correspondng to ts largest sngular value σ 1 (A j ) = A j 2,.e., Note that whch yelds (6.2). v A j v j = A j 2. 0 v Av = v A v + v j A jj v j 2v A j v j A 2 + A jj 2 2 A j 2, For (6.3) wth j, let us consder another vector v = (0,, 0, tv, 0,, 0,v j, 0,, 0) R n, t R, wth the same defntons of v and v j. It then follows that Therefore, one must have 0 v Av = t 2 v A v + v j A jjv j + 2tv A jv j t 2 A 2 + A jj 2 + 2t A j 2, t R. 4 A j A 2 A jj 2 0, 11

12 whch leads to (6.3). Fnally, (6.4) s straghtforward from (6.3). Ths theorem s a drect generalzaton of Theorem [4] and the followng corollary provdes one of ts straghtforward applcatons for estmatng λ. Corollary 6.2. Suppose x s a global maxmzer of the MCP, then the correspondng multvarate egenvalues satsfy max,j {λ, λ j } A j 2,, j = 1, 2,, m. (6.5) Proof. For any, j = 1, 2,, m, (6.1) and (6.2) yelds and consequently the estmaton (6.5). λ + λ j A 2 + A jj 2 2 A j 2, Furthermore, Theorem 6.1 s also very helpful n establshng the followng estmaton. Theorem 6.3. Suppose A s symmetrc postve defnte and (x, Λ ) s a soluton par to the MEP n whch x s a global maxmzer for the MCP. If A Λ s negatve sem-defnte, then r(x ) 1 m 1 [σ n (A ) + σ nj (A jj ) + 2 A j 2 ], (6.6) >j A j 2 2 (λ σ n (A ))(λ j σ nj (A jj )), j, (6.7) λ σ 1(A ) [ j (λ j σ n j (A jj )) 1 2 ] 2, = 1, 2,, m. (6.8) Proof. For (6.6), we note that Λ A s postve sem-defnte and λ I[n] A 2 = λ σ n (A ), = 1, 2,, m, (6.9) and t follows from Theorem 6.1 that 2 A j 2 λ σ n (A ) + λ j σ nj (A jj ),, j = 1, 2,, m; or λ + λ j 2 A j 2 + σ n (A ) + σ nj (A jj ),, j = 1, 2,, m. Therefore, (m 1)r(x ) = (λ + λ j) j 2 + σ n (A ) + σ nj (A jj )], >j >j[2 A whch s (6.6). The nequalty (6.7) s straghtforward based on Theorem 6.1 and (6.9). For (6.8), f λ I[n] A s sngular, t s trvally true. Therefore, we consder only the nonsngular case. For any j, the submatrx of (Λ A) ( λ I [n] A A j A j λ j I[nj] A jj ) R (n+nj) (n+nj), 12

13 s postve sem-defnte. Applyng Proposton n [1], we can express A j as A j = (λ I [n] A ) 1 2 Kj (λ ji [nj] A jj ) 1 2, (6.10) where K j R n nj satsfes K j 2 1. Thus from (6.10) and Ax = Λ x, one has λ x = A j x j + A x = j j (λ I [n] A ) 1 2 Kj (λ ji [nj] A jj ) 1 2 x j + A x, and hence (λ I [n] A )x = j (λ I [n] A ) 1 2 Kj (λ ji [nj] A jj ) 1 2 x j. The nonsngularty of (λ I[n] A ) further gves rse to x = (λ I [n] A ) 1 2 K j (λ ji [nj] A jj ) 1 2 x j, j and 1 = (λ I [n] A ) 1 2 K j (λ ji [nj] A jj ) 1 2 x j 2 j (λ σ 1(A )) 1 2 (λ j σ n j (A jj )) 1 2, j whch s exactly (6.8). It s clear, based on the global optmal condton, that Theorem 6.3 s applcable for both the case m = 2 and the case m > 2 wth a nonnegatve rreducble matrx A. In partcular, (6.8) nterestngly provdes an upper bound for the estmaton (6.1). In the rest of ths secton, we wll focus on the case m = 2. Frstly, f m = 2, (6.8) reads as λ 1 σ 1(A 11 ) λ 2 σ n 2 (A 22 ), (6.11) λ 2 σ 1 (A 22 ) λ 1 σ n1 (A 11 ). (6.12) Thus, f σ 1 (A 11 ) σ n2 (A 22 ) (or σ 1 (A 22 ) σ n1 (A 11 )), we have λ 1 λ 2 (or λ 2 λ 1 ). Moreover, a refnement of (6.11) and (6.12) can be made as we shall see n the followng theorem. Theorem 6.4. Suppose x s a global maxmzer for the MCP wth m = 2. Then the correspondng multvarate egenvalues λ 1 and λ 2 satsfy (λ 1 σ 1 (A 11 ))(λ 2 σ 1 (A 22 )) A (λ 1 σ n1 (A 11 ))(λ 2 σ n2 (A 22 )). (6.13) Proof. The second nequalty s already clear from (6.7). For the frst nequalty, f ether λ 1I [n1] A 11 or λ 2 I[n2] A 22 s sngular, t s trvally true. We then assume they are both nonsngular. Note that ( Λ (λ A = 1 I [n1] A 11 ) 1 2 (λ 2I [n2] A 22 ) 1 2 ) ( (λ E 1 I [n1] A 11 ) 1 2 (λ 2I [n2] A 22 ) 1 2 ), 13

14 where ( I [n 1] C E := C I [n2] ) R n n, and C := (λ 1 I[n1] A 11 ) 1 2 A12 (λ 2 I[n2] A 22 ) 1 2 R n 1 n 2. The postve sem-defnteness of Λ A mples the postve sem-defnteness of E whose egenvalues are exactly 1 σ 1 (C) 1 σ q (C) 1 = = σ q (C) 1 + σ 1 (C) wth n 1 n 2 ones n the mddle, where q := mn{n 1, n 2 } and σ (C) here represents the -th largest sngular value of C. Thus t must follow that 1 = σ 1 (C), and hence, 1 = (λ 1 I[n1] A 11 ) 1 2 A12 (λ 2 I[n2] A 22 ) A 12 2 (λ 1 σ 1(A 11 )) 1 2 (λ 2 σ 1 (A 22 )) 1 2, whch s exactly the frst nequalty n (6.13). 7. Concludng remarks. In ths paper, towards the global solutons of the MCP, whch s extremely desred from a practcal pont of vew, we put our man efforts nto the global optmalty, the unqueness and the estmaton of multvarate egenvalues, λ 1, λ 2,, λ m. The theoretcal achevements of the global optmal condtons and the unqueness of Λ for the MCP wth a nonnegatve rreducble matrx A not only offer us a much more clear pcture of the MCP and the related MEP, but also helps us to establsh the global convergence of ether the Horst-Jacob algorthm and the Gauss-Sedel algorthm to a global maxmzer. By comparng the classcal theorem for nonnegatve matrces wth the MEP wth nonnegatve rreducble matrces, we have observed many nce smlartes. The estmaton establshed n Secton 6, lastly, pcks up the nformaton n the off-dagonal matrces of A, and certanly moves us closer to the global soluton of the MCP. However, some problems stll reman open. The unqueness of Λ and the global optmalty are nterestng ssues that are worthy to be generalzed. Moreover, effcent numercal methods for the MCP wth a general matrx A should stll be pursued n the future. The success of the Horst-Jacob algorthm and the Gauss-Sedel algorthm for a nonnegatve rreducble matrx A reles on the nce propertes of the global maxmzer of the MCP and the core engne n choosng the startng ponts. Ths success agan shows the power of some specal startng pont strategy n fndng the global soluton of the MCP, and encourages us to nvestgate further for other cases. Acknowledgements. Specal thanks to Professor Moody T. Chu for ntroducng and suggestng the maxmal correlaton and multvarate egenvalue problems to the frst author durng hs vst to the North Carolna State Unversty. REFERENCES [1] R. Bhata, Postve defnte matrces, Prnceton Unversty Press, [2] M. T. Chu and J. L. Watterson, On a multvarate egenvalue problem: I. Algebrac theory and power method, SIAM J. Sc. Comput., vol. 14(1993), pp [3] G. H. Golub and L.-Z. Lao, Contnuous methods for extreme and nteror egenvalue problems, Lnear Alg. Appl., 415 (2006), [4] G. H. Golub and C. F. Van Loan, Matrx computatons, 3rd ed., Johns Hopkns Unversty Press, Baltmore, MD, [5] M. Hanaf and J. M. F. Ten Berge, Global optmalty of the successve Maxbet algorthm, Psychometrka, 68(2003), pp [6] P. Horst, Relatons among m sets of measures, Psychometrka, 26(1961), pp [7] P. Horst, Factor analyss of data matrces, Holt Rnehart and Wnston, New York, [8] H. Hotellng, The most predctable crteron, J. Educ. Pyschol., 26(1935), pp [9] H. Hotellng, Relatons between two sets of varates, Bometrka, 28(1936), pp [10] J. R. Kettenrng, Canoncal analyss of several sets of varables, Bometrka, 58(1971), pp [11] C. D. Meyer, Matrx Analyss and Appled Lnear Algebra, SIAM, Phladelpha,

15 [12] C. R. Rao and M. B. Rao, Matrx algebra and ts applcatons to statstcs and econometrcs, World Scentfc, [13] J. M. F. Ten Berge, Generalzed approaches to the MAXBET problem and the MAXDIFF problem, wth applcatons to canoncal correlatons, Psychometrka, 53 (1988), pp [14] J. R. Van de Geer, Lnear relatons among k sets of varables, Psychometrka, 49(1984), pp [15] T. L. Van Noorden and J. Barkmejer, The multvarate egenvalue problem: a new applcaton, theory and a subspace accelerated power method, Unverstet Utrecht, preprnt, [16] L.-H. Zhang and M. T. Chu, On a multvarate egenvalue problem: II. Global solutons and the Gauss-Sedel method, submtted to SIAM Journal on Scentfc Computng for publcaton. Avalable: mtchu/research/papers/readme.html 15