CRITICAL POINT ANALYSIS OF JOINT DIAGONALIZATION CRITERIA. Gen Hori and Jonathan H. Manton

Transcription

1 CRITICAL POINT ANALYSIS OF JOINT DIAGONALIZATION CRITERIA Gen Hor and Jonathan H Manton Bran Scence Insttute, RIKEN, Satama , Japan hor@bspbranrkengojp Department of Electrcal and Electronc Engneerng, Unversty of Melbourne, Vctora, 3010, Australa jon@eemuozau ABSTRACT The stablty and senstvty of jont dagonalzaton crtera are analyzed usng the Hessan of the crtera at ther crtcal ponts The senstvty of some known jont dagonalzaton crtera s shown to be weak when they are appled to the matrces wth closely placed egenvalues In such a stuaton, a large devaton n the dagonalzer cause a slght change n the crteron whch makes the estmaton of the dagonalzer dffcult To overcome the problem, a new jont dagonalzaton crteron s ntroduced The crteron s lnear wth respect to the target matrces and has ablty to cancel the effect of closely placed egenvalues 1 INTRODUCTION Jont dagonalzaton s a problem of fndngamatrx whch smlarly transform a set of gven matrces as dagonal as possble smultaneously It has applcatons n ndependent component analyss where typcal targets are the fourth order cumulant matrces[3] and the delayed correlaton matrces[7] of prewhtened observed sgnalsaswellasnotherresearchareassuchasstructured egenvalue problem[2] A jont dagonalzaton algorthm s a combnaton of a jont dagonalty crteronandanoptmzatonmethodusedformnmzng the crteron A jont dagonalty crteron s a functon defned on the space of the dagonalzer whch takes ts mnmum when the gven matrces are jontly dagonalzed The present paper surveys three jont dagonalty crtera n Secton 2 Typcal optmzaton methods appled to jont dagonalty crtera nclude the Jacob method, gradent method and the Newton method The purpose of the present paper s to analyze the stablty of the crtcal ponts of jont dagonalzaton crtera, pont out that known crtera have some problem when t s appled to the jont dagonalzaton of matrces wth closely placed egenvalues, and ntroduce a new crteron (defnednsecton23)tocrcumvent the problem The stablty analyss of the crtcal ponts usng the Hessan of the crteron reveals that, when the gven matrces have closely placed egenvalues, the senstvty of the jont dagonalzaton crteron serously goes down, that s, a large devaton n the dagonalzer causes an extremely small amount of change n the crteron, whch s a problem n the context of ndependent component analyss where the man purpose s not the calculaton of the egenvalues of gven matrces but the estmaton of the dagonalzer The effect of the closeness of the egenvalues to the senstvty of the wdely used off crteron s quadratc To crcumvent the problem, we propose to use a new jont dagonalty crteron (defned nsecton23) Thecr- teron s parameterzed by constant dagonal matrces ofthesameszewththetargetmatrcesandtssenstvty can be controlled by the dagonal elements of the constant dagonal matrces On the other hand, unless the dagonal elements are approprately adjusted, the crtcal ponts of the crteron become unstable, that s, the teratve mnmzaton of the crteron does not converge to the desred matrx Snce the approprate adjustment of the dagonal elements can not be done wthout knowng the orders of the egenvalues of the target matrces, we have to start the optmzaton wth some other crteron lke off and swtch to the mnmzaton of the new crteron after the target matrces come close to dagonal matrces so that they expose the order of ther egenvalues The rest of the present paper s organzed as follows Secton 21 and 22 survey two known jont dagonalzaton crtera and Secton 23 ntroduces the new crteron Secton 3 analyzes the crtcal ponts of the two know crtera and the new crteron to examne the stablty and the senstvty of them Secton 4 demonstrates the advantage of the new crteron usng smulaton Secton 5 contans concludng remarks

2 2 JOINT DIAGONALIZATION CRITERIA Consder a set of K matrces {A 1,A 2,,A K }, A k M(n, C) Jont dagonalzaton of the set of matrces s a problem of fndng a matrx U whch makes U A 1 U, U A 2 U,,U A K U as dagonal as possble smultaneously where U s typcally a untary matrx If all the matrces of the set commute each other then they can be dagonalzed smultaneously If they do not then they can not be dagonalzed smultaneously but nstead we can try to fnd a untary matrx whch mnmzes a jont dagonalty crteron defned to solve specfc problems We call the former and the latter exact jont dagonalzaton and approxmate jont dagonalzaton respectvely A dagonalty measure of a sngle matrx s a functon defned on the set of matrces A whch meets the followng condton, ψ(a) mnψ(a) A s dagonal A A A jont dagonalty crteron of a set of matrces s then defned usng the dagonalty measure of each matrx, φ(u) ψ(u A k U) Once a jont dagonalty crteron s defned, a jont dagonalzaton problem can be approached usng teratve optmzaton methods such as the Jacob method, the gradent method and the Newton method The rest of the secton brefly surveys two known jont dagonalty crtera and ntroduces a new crteron 21 Sum of squares of off-dagonals The sum of squares of the off-dagonal elements, ψ 1 (A) off(a) X a j 2, 6j usually called off, s a wdely used measure of dagonalty of a matrx Based on off, a jont dagonalty crteron s defned as φ 1 (U) off(u A k U) (1) The crteron s easy to understand as well as theoretcally reasonable when t s appled to the jont dagonalzaton of the fourth order cumulant matrces[3] Several optmzaton methods have been appled to the crteron ncludng the extended Jacob method[1][3] and the Newton method[10] 22 Log lkelhood crteron Pham[5] ntroduced a dagonalty measure of a postve defnte Hermtan matrx A defned as ψ 2 (A) log det dag(a) log det(a), motvatedbythemaxmumlkelhoodmethodusedn the common prncpal component estmaton[6] The maxmum lkelhood estmaton n the problem amounts to mnmzaton of the jont dagonalty crteron φ 2 (U) n k (log det dag(u A k U) log det(u A k U)) (2) where n k s a postve number representng the sze of observatons from the dstrbuton wth a covarance matrx A k The crteron has an nvarance wth respect to a scale change whch s not shared wth off Pham[5] consdered the case wthout orthogonalty constrant and developed a Jacob-lke method for the optmzaton of the crteron In the followng secton, we consder the case wth orthogonalty constrant where the second term of the crteron becomes constant 23 Lnear crteron To overcome the shortcomngs of the prevous two crtera whch wll be dscussed n the followng sectons, we ntroduce a new jont dagonalty crteron based on the dagonalty measure of a matrx defned as ψ 3 (A) trca c a, 1 where C dag(c 1,,c n ) s a constant dagonal matrx Ths dagonalty measure s related the work by Brockett[8] n whch t was shown that the gradent flow on the adjont orbt mnmzng the measure globally converges to a dagonal matrx whose dagonal elements are ordered smlarly to the dagonal elements of the constant matrx C Based on the dagonalty measure, we ntroduce a lnear jont dagonalty crteron defned as φ 3 (U) trc k (U A k U), (3) where C k dag(c (k) 1,,c(k) n ),k 1, 2,,K are constant dagonal matrces As we wll see n the analyss n the followng secton, a crtcal pont of the jont dagonalty crteron s stable only f the dagonal elements of U A k U and the dagonal elements of C k are smlarly ordered for all k Because t s generally mpossble to know the order of the dagonal elements of

3 U A k U at crtcal ponts n advance, we need to start the optmzaton wth some other crteron lke off and swtch to the lnear crteron when the target matrces become close to dagonal so that ther dagonal elements expose ther order at crtcal ponts Because any choce of the dagonal matrces C k whose dagonal elements are approprately ordered makes the crtcal ponts stable, we have room to control the senstvty of the crteron by the dagonal elements of C k 3 CRITICAL POINT ANALYSIS We analyze the stablty and senstvty at the crtcal ponts of the three jont dagonalty crtera defned n the prevous secton, restrctng our attenton to the exact problems of real matrces To calculate the Hessan of the crtera, we use the followng coordnate system defned on the orthogonal group Let E j denote an n n matrx whose (, j)-th element s 1 and all the other elements are 0 s Suppose that X j s a skew symmetrc matrx defned as X j E j E j (<j), then a plane rotaton s expressed n an exponental of the skew symmetrc matrx X j, 1 cos ϑ sn ϑ e ϑx j sn ϑ cos ϑ 1 We ntroduce a local coordnate system, θ (θ 12, θ 13,,θ 1n, θ 23,,θ 2n,,θ n 1,n ), on the orthogonal group around the dentty defned as U(θ) e θ12x12 e θ13x13 e θ11n e θ23x23 e θ22n e θn 1,n 1,n Inthecoordnatesystem,thedenttymatrxI s expressed as θ 0 Let us put A(θ) U(θ) A 0 U(θ), then the frst and second order dervatves of A(θ) at A 0 are calculated as j A(θ) θ0 [A 0,X j ], j kl A(θ) θ0 [[A 0,X j ],X kl ], where j denotes the dfferentaton wth respect to θ j, j θ j Note that the order of X j and X kl n the rght hand sde of the formula of the second order dervatve s the same as the order appeared n the defnton of U(θ) and does not depend on the order of dfferentaton We consder an obvous exact jont dagonalzaton problemofasetofmatrces, where {Λ 1, Λ 2,,Λ K }, Λ k dag(λ (k) 1, λ(k) 2,,λ(k) n ), k 1, 2,,K, and examne the stablty and the senstvty of jont dagonalty crtera around the crtcal pont U I From a vewpont of the stablty analyss, we do not loose generalty by ths settng To calculate the Hessan of a dagonalty crteron at dagonal ponts, we put A k (θ) U(θ) Λ k U(θ), k 1,,K Then the frst and second order dervatves of A k (θ) at Λ k are calculated as j A k (θ) θ0 [Λ k,x j ] (4) (λ (k) λ (k) j )(E j + E j ), j kl A k (θ) θ0 [[Λ k,x j ],X kl ] (5) 2(λ (k) λ (k) j )(E E jj ) ( k, j l) (λ (k) λ (k) j )(E jl + E lj ) ( k, j 6 l) (λ (k) λ (k) j )(E k + E k ) ( 6 k, j l) (λ (k) λ (k) j )(E l + E l ) (k j, < l) 0 (otherwse) The Hessan of the off jont dagonalty crteron (1) at U I s calculated as follows Mnmzng off s equvalent to maxmzng the sum of square of the dagonal elements under an orthogonal transform Therefore the negatve of the sum of square of dagonal elements s equvalent to off from the vewpont of the calculaton of the Hessan Then we have j kl φ 1 (U(θ)) θ0 j kl (a (r) pp ) 2 2(a (r) pp j kl a (r) pp + j a (r) pp kl a (r) pp )

4 where a (k) j denotes the (, j)-th element of the k-th matrx A k (θ) Usng (4) and (5), we obtan j kl φ 1 (U(θ)) θ0 4 (λ (r) λ (r) j ) 2 ( k, j l) r1 0 (otherwse) The obtaned Hessan s already dagonalzed and ts n(n 1)/2 dagonal elements are gven as 4 (λ (k) λ (k) j ) 2 (<j) From ths we see that all the dagonal elements of the Hessan are postve and the crtcal pont U I s stable when each Λ k has dstnct dagonal elements whle the senstvty of the crteron becomes weak when λ (r) and λ (r) j are close for the most of r for some par (, j) For the log lkelhood crteron (2), the Hessan s calculated as follows Snce the second term of the dagonalty crteron s nvarant under an orthogonal transform, the crteron s equvalent to ψ 2 (A) log det dag(a) log a pp p1 Then the Hessan s obtaned as j kl φ 2 (U(θ)) θ0 n r j kl log(a (r) pp ) 2 r1 a (r) pp j kl a (r) pp j a (r) pp kl a (r) pp n r (a (r) pp ) 2 (λ (r) n r λ (r) λ (r) j ) 2 λ (r) j ( k, j l) 0 (otherwse) The obtaned Hessan s also dagonalzed n ths case Note that the log lkelhood crteron s for the postve defnte Hermtan matrces and all the dagonal elements of Λ k are assumed to be postve We then see that the crtcal ponts of the log lkelhood crteron have the stablty and the senstvty smlar to those of the off crteron For the lnear jont dagonalty crteron (3), the Hessan s calculated as j kl φ 3 (U(θ)) θ0 2 (c (r) r1 c (r) p j kl a (r) pp c (r) j )(λ (r) λ (r) j ) ( k, j l) 0 (otherwse) The obtaned Hessan s also dagonalzed n ths case We see that the condton for the crtcal pont U I to be stable s that C k and Λ k have dstnct dagonal elements and they are smlarly ordered for all k Whle the dagonal elements of C k need to meet the above condton to make the crteron stable, they can be used to control the senstvty of the crteron For example, f λ (r) and λ (r) j are close for the most of r for some par (, j) and ths spols the senstvty of the crteron, then we can set c (r) and c (r) j relatvely apart to cancel the effect of the closeness and recover the senstvty 4 SIMULATION We compare the performance of the three jont dagonalty crtera φ 1 (U), φ 2 (U) andφ 3 (U) usng the followng two jont dagonalzaton problems, Problem (A) - wthout closely placed egenvalues {dag(1, 2, 3, 4, 5), dag(5, 4, 3, 2, 1)}, Problem (B) - wth closely placed egenvalues {dag(1, 2, 3, 4, 41), dag(5, 4, 3, 2, 19)} Obvously, the optmal solutons of those jont dagonalzaton problems are permutaton matrces, ncludng the dentty matrx We smulate gradent equatons of the three jont dagonalty crtera for the two jont dagonalzaton problem startng from an orthogonal matrx very close to the dentty matrx (whch must converge to the dentty matrx) usng the Euler method For the lnear crtera φ 3 (U), we compare the followng two optons of constant matrces, and C 1 dag(1, 2, 3, 4, 5),C 2 dag(5, 4, 3, 2, 1), C 1 dag(1, 2, 3, 4, 14),C 2 dag(14, 13, 12, 11, 1) From the consderaton n the prevous secton, t s expected that the former suffers the senstvty reducton for Problem (B) whle the latter cancels the effect of the closely placed egenvalues Fgs1-4 depct the tme-evoluton of the off crteron φ 1 (U) and the devaton from the dentty matrx d(u, I) log(u) for the gradent equatons of the jont dagonalty crtera To measure the convergence of the target matrces, we use φ 1 (U) n common for the sake of comparson The convergence of the dagonalzer s measured by d(u, I) Where φ 1 (U) ssmall enough but d(u, I) s not, the target matrces are converged to dagonal matrces but the dagonalzer s not converged to the optmal soluton

5 Fgs1-3 share the tendency that the convergence of both φ 1 (U) andd(u, I) become dull for Problem (B), comparng to Problem (A) Especally, the convergence of the dagonalzer s totally spoled by the the effect of the closely placed egenvalues On the other hand, n Fg4, the convergence of both φ 1 (U) andd(u, I), and hence of the dagonalzer, are not affected by the closely placed egenvalues Fgure 1 Gradent equaton of φ 1 (U) Fgure 2 Gradent equaton of φ 2 (U) Fgure 3 Gradent equaton of φ 3 (U) (C 1 dag(1, 2, 3, 4, 5),C 2 dag(5, 4, 3, 2, 1)) 5 CONCLUDING REMARKS The stablty and senstvty of two known jont dagonalty crtera were analyzed usng the Hessan and t was ponted out that ther senstvty s weakened when the target matrces have closely placed egenvalues To crcumvent the problem, we ntroduced the lnear crteron and consdered ts stablty condton and senstvty control The consderaton was valdated by smulaton of the gradent equatons of the jont dagonalty crtera To add to that the lnear crteron has ablty to cancel the effect of closely placed egenvalues, ts lnearty makes the jont dagonalzaton problem more tractable n many aspects of optmzaton methods For example, when the lnear crteron s optmzed usng the Jacob teraton, the optmal rotaton angle s gven by a much smpler formula (almost the same formula of the rotaton angle for a sngle matrx) comparng to the formula for the off crteron obtaned by Cardoso and Souloumac[4] On the other hand, the lnear crteron s only for the fnal phase of teraton and not for the global optmzaton We have to start wth an optmzaton of some other crteron and swtch to the lnear crteron when the target matrces come close to dagonal matrces Fnally, we note that there s a unfed way of lookng at the three jont dagonalty crtera referred to n the present paper Let us extend the constant matrces C k n the defnton of φ 3 (U) to tme varyng matrces and consder the followng two cases, C k dag(a (k) 11,a(k) 22,,a(k) 33 ), C k dag(1/a (k) 11, 1/a(k) 22,,1/a(k) 33 ), Fgure 4 Gradent equaton of φ 3 (U) (C 1 dag(1, 2, 3, 4, 14),C 2 dag(14, 13, 12, 11, 1)) then the former and the latter correspond to the off crteron φ 1 (U) and the log lkelhood crteron φ 2 (U) respectvely From ths vewpont, swtchng jont dagonalzaton crtera can be regarded as swtchng the matrces C k and extended to more flexble schedulng of C k

6 6 REFERENCES [1] A Bunse-Gerstner, R Byers, and V Mehrmann Numercal methods for smultaneous dagonalzaton SIAM J Mat Anal Appl, 14: , 1993 [2] A Bunse-Gerstner, R Byers, and V Mehrmann A chart of numercal methods for structured egenvalue problems SIAM J Matrx Anal Appl, 13: , 1992 [3] J-F Cardoso and A Souloumac Blnd beamformng for non-gaussan sgnals IEE Proc F, 140: , 1993 [4] J-F Cardoso and A Souloumac Jacob angles for smultaneous dagonalzaton SIAM J Mat Anal Appl, 17: , 1996 [5] DT Pham Jont approxmate dagonalzaton of postve defnte Hermtan matrces SIAM J Matrx Anal Appl, 22: , 2001 [6] BN Flury Common prncpal components n kgroups J Amer Statst Assoc, 79: , 1984 [7] L Molgedey and HG Schuster Separaton of a mxture of ndependent sgnals usng tme delayed correlatons PhysRevLett, 72: , 1994 [8] RW Brockett Dynamcal systems that sort lsts, dagonalze matrces and solve lnear programmng problems Lnear Algebra Appl, 146:79 91, 1991 [9] G Hor A new approach to jont dagonalzaton In Proc 2nd Intl Workshop on Independent Component Analyss and Blnd Sgnal Separaton (ICA2000), pages [10] M Nkpour, JH Manton, and G Hor Algorthms on stefel manfold for jont dagonalsaton In ProcofICASSP2002, pages , 2002 [11] JH Manton Optmsaton algorthms explotng untary constrants IEEE Trans Sgnal Processng, 50: , 2002 [12] A Edelman, TA Aras, and ST Smth The geometry of algorthms wth orthogonalty constrants SIAM J Matrx Anal Appl, 20: , 1998