/16/$ IEEE 1728

Transcription

1 Extension of the Semi-Algebraic Framework for Approximate CP Decompositions via Simultaneous Matrix Diagonalization to the Efficient Calculation of Coupled CP Decompositions Kristina Naskovska and Martin Haardt Communications Research Laboratory Ilmenau University of Technology P. O. Box 00565, D Ilmenau, Germany { kristina.naskovska, martin.haardt }@tu-ilmenau.de Abstract Several combined signal processing applications such as the joint processing of EEG and MEG data can benefit from coupled tensor decompositions, for instance, the coupled CP (Canonical Polyadic decomposition. The coupled CP decomposition jointly decomposes tensors that have at least one factor matrix in common. The (Semi-Algebraic framework for approximate CP decomposition via SImultaneaous matrix diagonalization framework is an efficient tool for the calculation of the CP decomposition based on matrix diagonalizations. It provides a semi-algebraic solution for the CP decomposition even in ill-posed scenarios, e.g., if the columns of a factor matrix are highly correlated. Moreover, the framework provides an adjustable complexity-accuracy trade-off. In this paper, we present an extension of the framework to the efficient calculation of coupled CP decompositions and show its advantages compared to the traditional solution via alternating least squares (ALS and other state of the art algorithms. Index Terms Coupled, CP, PARAFAC, tensor decomposition, semialgebraic framework,, simultaneous diagonalization I. INTRODUCTION Tensors provide a useful tool for the analysis of multidimensional data. A comprehensive review of tensor concepts is provided in. Tensors have a very broad range of applications such as compressed sensing, processing of big data, blind source separation and many more. Often a tensor should be decomposed into the minimum number of rank one components. This decomposition is know as PARAFAC (PARallel FACtors, CANDECOMP (Canonical Decomposition, or CP (CANDECOMP/PARAFAC. The CP decomposition is often calculated via the iterative multilinear-als (Alternating Least Square algorithm. ALS based algorithms require a lot of iterations to calculate the CP decomposition and there is no convergence guarantee. Moreover, ALS based algorithms are less accurate in ill-conditioned scenarios, for instance, if the columns of the factor matrices are highly correlated. There are already many ALS based algorithms for calculating the CP decomposition such as the ones presented in 3 and 4 that either introduce constraints to reduce the number of iterations or are based on line search, respectively. Alternatively, semi-algebraic solutions have been proposed in the literature based on SMDs (Simultaneous Matrix Diagonalizations. Such examples include 5, 6, 7, 8 and 9. The framework 8 calculates all possible SMDs and then selects the best available solution in a final step via appropriate heuristics. However, recent combined signal processing application such as joint processing of EEG and MEG data can benefit from coupled tensor decompositions, such as a coupled CP decomposition. The coupled CP decomposition jointly decomposes tensors that have at least one factor matrix in common. In order to jointly decompose tensors the existing algorithms have to be modified. Therefore, in this paper we propose an extension of the framework for the calculation of coupled CP decompositions and compare it to the coupled ALS. We use the following notation. Scalars are denoted either as capital or lower-case italic letters, A, a. Vectors and matrices, are denoted as bold-face capital and lower-case letters, a, A, respectively. Finally, tensors are represented by bold-face calligraphic letters A. The following superscripts, T, H,, and + denote transposition, Hermitian transposition, matrix inversion and Moore-Penrose pseudo matrix inversion, respectively. The outer product, Kronecker product and Khatri-Rao product are denoted as,, and, respectively. The operators. F and. H denote the Frobenius norm and the Higher order norm, respectively. Moreover, an n-mode product between a tensor A C I I... I N and a matrix B C J In is defined as A n B, for n =,,... N 0. A super-diagonal or identity N- way tensor of dimensions R R... R is denoted as I N,R.. TENSOR DECOMPOSITIONS For simplicity, in our derivation we will take into account two low rank tensors X ( 0 CM M ( ( M 3, 0 CM M ( ( M 3. Moreover, the two tensors have only one common mode and that is the first mode. Therefore, the two tensors have the first factor matrix as a common one. The CP decomposition of the low rank tensors X ( 0 and 0 is defined as X ( 0 = I 3,R A B ( 3 C (, ( 0 = I 3,R A B ( 3 C ( ( where the tensor rank of both tensors is equal to R. The CP decomposition is essentially unique under mild conditions, which means that the factor matrices (i.e., A, B (, B (, C (, and C (, can be identified up to a permutation and scaling ambiguity. Another, tensor decomposition which is much easier to calculate is the HOSVD (Higher Order Singular Value Decomposition 0. The HOSVD of the tensors X ( 0 and 0 is given by X ( 0 = S ( U U ( 3 U ( 3 (3 0 = S ( U U ( 3 U ( 3 (4 M ( where, S ( C M M ( ( M 3 and S ( C M M ( 3 are the core tensors. The matrices U C M M, U C M M and U 3 C M 3 M 3 are unitary matrices /6/$ IEEE 78 Asilomar 06

2 Moreover, the truncated HOSVD is defined as X ( 0 = S s,( U s 0 = S s,( U s U s,( 3 U s,( 3 (5 U s,( 3 U s,( 3 (6 where S s,( and S s,( C R R R are the truncated core tensors and the loading matrices U s C M R, U s, C M R and 3 R have unitary columns and span the column space U s, 3 C M of the n-mode unfolding of X 0, for n =,, 3 and i =,, respectively. Note that the matrices U s and A span the same column space of X ( 0 (. Due to the fact that the tensors X ( 0 and 0 have the factor matrix A in common the unitary matrix U s should be the same for both HOSVDs in equations (5 and (6. In practice we can only observe a noise corrupted version of the low rank tensor X = X 0 + N, where N contains uncorrelated zero mean circularly symmetric complex Gaussian noise. Hence, we have to calculate a rank R approximation of X X S s, U s s, U 3 U s, 3. (7 Note that (7 holds exactly in the absence of noise and if R is the true rank of the tensor X. For the following derivations we assume that this is true and hence write equalities. In the presence of noise, all the following relations still hold approximately. I. COUPLED FRAMEWORK In this section we derive the coupled framework for two tensors of order three and tensor rank R denoted by X, i =,, which have the first factor matrix in common. An extension to tensors of order N is straightforward by using the concepts of general unfoldings as described in 9. Moreover, an extension to multiple common matrices is also straightforward. Our goal is to jointly provide an estimate of the factor matrices for both tensors. The framework starts by computing the truncated HOSVD. Since the first factor matrix is common for both tensors, the column space of the corresponding one mode unfolding is calculated jointly, and independently for the rest of the modes (i.e, m =, 3 via the following SVDs (Singular Value Decompositions. X ( ( s ( = U Σs V sh, X (m = U s, m Σ s, m V m s,h, m =, 3; i =,. Inserting equations (5 and (6 into ( and (, we get ( X = S s, 3 U s, 3 U s s, U (8 = 3 T 3 }{{} C I3,R 3 (U s, }{{ T (U s, T }}{{} A B (9 (U s The equations (8 and (9 represent the fundamental link between the HOSVD and the CP decomposition, and the coupling between the two tensors. The invertible matrices T, T, and T 3 of size R R diagonalize the core tensors S s,, for i =,, respectively, as previously shown in 6 and 8. Therefore, after multiplying equations (8 and (9 by U sh U s,h we obtain the following tensors S 3 = C T T i =, (0 R R M where S 3 = S s, 3 U s, 3 C 3 and C = I 3,R 3 C R R M C 3. The visualization of equation (0 Fig. : Diagonalization of the tensor S 3, i =,. is given in Fig.. Equation (0 represents a non-symmetric SMD, while in this paper we recommend to diagonalize the core tensors via symmetric SMDs, for instance. The extension of the framework based on non-symmetric SMDs was presented in. However, instead of non-symmetric SMDs we recommend to use symmetric SMDs so that the coupling between the two tensors can be better exploited. Therefore, we convert the non-symmetric SMD problem into a symmetric SMD. In order to do so one of the diagonalization matrices has to be eliminated. Hence, as shown in 6 we multiply equation (0 by one pivoting slice from the right and left hand side, respectively. S rhs, = S S 3,p i ( = T diag(c (k, :./C (p i, : T ( ( T = S 3,p i S (3 diag(c (k, :./C (p i, : T (4 where S and C (k, :, are the k-th slice of the tensors S 3 and C, respectively. Moreover, C (k, : represents the k-th row of the factor matrix C. Furthermore, p i can be any arbitrary pivoting slice, p i {,,..., M 3 }. However, since this slice has to be inverted, the best choice is to choose the slice with the smallest conditioning number. Note that a different pivoting slice p i can be chosen for the different tensors. Equation ( represents two symmetric SMDs, for each of the two tensors S ( 3 and S ( 3. Moreover, the two SMDs have the same diagonalization matrices, which means that we can concatenate the two equations and solve one diagonalization problem instead. Hence, S rhs,( S rhs,( diag(c ( (k, :./C ( (p, : = T diag(c ( (k, :./C ( T (p, : (5 is a coupled symmetric SMD, which allows as to diagonalize both core tensors jointly. From the coupled SMD, we can estimate the first factor matrix as  I = U s T guaranteeing that even in a noisy scenario the common mode will have the same factor matrix estimate for the tensors X ( and. Next, from the diagonal elements of the diagonalized tensor the factor matrices Ĉ ( I and Ĉ ( I are estimated 8. Finally, based on a LS (Least Squares solution using the corresponding estimates of the other two factor matrices the last factor matrices can be estimated, ˆB ( I and ˆB ( I. Note that equation (4 does not depend on the common mode. Therefore the two SMDs cannot be combined and they have to be solved separately. Similarly to the coupled SMD, an estimate of the matrices ˆB (, ˆB (, Ĉ ( and Ĉ ( can be provided. The common 79

3 factor matrix is estimated from the following joint LS problem  = X ( ( ( ( ˆB ( Ĉ ( T ( ˆB ( Ĉ ( T +. Up to this point we have diagonalized the tensors along the third mode as depicted in Fig., but the rest of the modes can also be used in order to obtain more estimates as explained in 8. Another two sets of estimates can be obtained by diagonalizing the tensors along the second mode based on the following SMDs and S rhs,(,k S rhs,(,k = T diag(b ( (k, :./B ( (p, : diag(b ( (k, :./B ( (p, : T (6,k 3 diag(b (k, :./B (p i, : T 3. (7 The estimates obtained from (6 are given by  I = U s T from the transform matrix, ˆB ( I, ˆB ( I from the diagonal elements of the diagonalized tensor, and Ĉ ( I and Ĉ ( I based on a LS solution using the corresponding estimates of the other two factor matrices. Moreover, from (7 the following estimates are obtained. The factor matrices Ĉ ( IV = U s, 3 T 3 and Ĉ ( IV = U s, 3 T 3 are obtained from the transform matrices. Moreover, ˆB ( IV and ˆB ( IV are obtained from the diagonal elements of the diagonalized tensor and  IV is estimated based on the following joint LS problem. (  IV = X ( ( ( ˆB ( T ( IV Ĉ( IV ˆB ( T + IV Ĉ(. IV Finally, the following SMDs are defined for the tensors diagonalization along the first mode. S rhs,,k diag(a (k, :./A (p i, : T,k 3 diag(a (k, :./A (p i, : T 3 The coupled mode is in the diagonal elements of the diagonalized tensor, therefore a joint SMD cannot be calculated. From the four SMDs presented above, four different estimates of the coupled mode are obtained. The additional estimates obtained from the diagonalization along the first mode are summarized in Table I. Transform Matrix V V Ĉ V Ĉ V Diagonalized Tensor  V  VI  V  VI LS Ĉ V Ĉ VI VI V TABLE I: Estimates of the factor matrices obtained from the diagonalization along the first mode. To summarize, the coupled framework for two tensors of order three with N c common modes, i.e., N c =,, 3, will result in 3 + (3 N c sets of estimates of the factor matrices. For the scenario that we have presented in this paper, two tensors of order three with one mode in common, 8 different sets of estimates can be obtained with the coupled framework. As a comparison the original framework calculates 6 sets of estimates, 8. The two additional sets are obtained in the case of the diagonalization along the coupled mode. The estimate of the common mode that comes from tensor X ( can be considered as a possible solution for the tensor. However, when using the common factor matrix that is estimated from another tensor and for calculating the joint LS the permutation and scaling ambiguity has to be taken into account. The estimates that are based on different SMDs have an arbitrary permutation, which can be eliminated via a comparison if one estimate is taken as a reference. For simplicity, the final estimate is selected via the BM (Best Matching scheme, 8. The BM solves all the SMDs and the final estimate is the one that leads to the lowest reconstruction error after calculating all possible combinations. The reconstruction error is calculated according to ˆX X H RSE = X. (8 H Different heuristics that lead to different complexity-accuracy tradeoffs, have been presented in 8. They lead to a comparable performance at a significantly reduced computational complexity and are also applicable to the coupled framework described here. IV. COUPLED ALTERNATING LEAST SQUARES We want to compare the performance of the coupled framework with another algorithm for the coupled CP decomposition. Therefore, we summarize a very simple extension of ALS to coupled ALS. Similar to ALS, the coupled ALS also takes into account all unfoldings of the tensor and iteratively updates each of the factor matrices starting from a random initialization. Based on the three unfoldings for the given tensors of order three X ( and the estimates of the factor matrices can be defined as follows. For the coupled mode the u-th update of the corresponding factor matrix is jointly calculated from (  u = X ( ( ( ˆB ( T ( u Ĉ( u B ( T + u Ĉ( u. Moreover, the u-th update for the other two factor matrices is given by ( (Ĉ T + u = X ( u  u ( Ĉ u = X (3 (Âu Ĉ T + u. V. SIMULATIONS RESULTS In this section the proposed extension of for coupled CP decompositions, denoted as C-, is compared to the original framework (, coupled ALS denoted as, coupled CPD by unconstrained nonlinear optimization and coupled / symmetric CPD by nonlinear least squares from 3 denoted as and, respectively. In each case we have computed Monte Carlo simulation using 000 realizations. For simulation purposes two different tensors with first common mode and tensor rank R have been designed. Each of the tensors is generated according to the CP decomposition. X ( 0 = I 3,R A B ( 3 C ( 0 = I 3,R A B ( 3 C ( where the factor matrices A, B, and C have i.i.d. zero mean Gaussian distributed random entries with variance one, if not otherwise stated. Moreover, for some simulation scenarios we want the tensors to have correlated factor matrices, therefore we add correlation via A A R(ρ R(ρ = ( ρ I R,R + ρ R R R, 730

4 where R(ρ is the correlation matrix with correlation factor ρ and R R denotes a matrix of ones. Finally, the synthetic data is generated by adding i.i.d. zero mean Gaussian noise with variance σn. In the simulation results the (Total relative Mean Square Factor Error ˆF n P F n F = E min P M PD (R F n F ˆF n=â, ˆB,Ĉ is used as an accuracy measure, where M PD(R is a set of permuted diagonal matrices of size R R that resolve the permutation ambiguity of the CP decomposition and F n is equal to A, B or C C- C Fig. 3: of the for complex-valued via tensors with dimensions 4 8 7, tensor rank R = 3 and factor matrices with mutually correlated columns, SNR = 45 db X C- X C- X X very difficult to calculate. From the Fig. 3 it is noticeable that C- ALS fails in most of the attempts to decompose the given tensors. However, the and the C- frameworks are still able to decompose the tensors. Moreover, the C- algorithm shows a better performance than the and NLS algorithms Fig. : of the for real-valued tensors with dimensions , tensor rank R = 4 and factor matrices with mutually correlated columns, SNR = 30 db. First, we compare the performance of the C-,, C- ALS, CCDP NLS and MINF for two real-valued tensors, X ( and of size The two tensors have the first factor matrix in common, and additionally the common factor matrix has collinear columns with correlation factor ρ = The (Complementary Cumulative Distribution Function of the for SNR = 30 db is presented in Fig.. We present the of the error since we are also interested in the convergence of the algorithms in addition to the mean error. Moreover, the mean value of the error for the SNR = 30 db and for each curve is presented as a vertical line. From Fig., it is easy to observe that the C- outperforms the rest of the algorithms, however there is no performance gain compared to the original framework. Next, in Fig. 3 the of the is presented for two tensors of size with a common first factor matrix. For this scenario the common factor matrix is chosen as A from ( A = C ( = C ( = (9 This factor matrix is ill-conditioned and has highly correlated columns, and the CP decomposition containing this factor matrix is C- C- Fig. 4: of the for complex-valued tensors with dimensions 7 8 4, tensor rank R = 3 and factor matrices with mutually correlated columns, SNR = 45 db. Similarly, in Fig. 4 we compare the performance of above discussed algorithms for an ill-conditioned scenario, where the third factor matrices are chosen as C and C from (9. The two tensors have the dimensions and they still have the first mode in common. In the scenario where the ill-conditioned factor matrix is not the common mode, we are able to observe an accuracy gain compared to the uncoupled framework as depicted in Fig. 4. Moreover, since the framework is able to estimate the factor matrices even in a degenerate case, when the rank of the tensor R exceeds the tensor size in one of the modes, we have also simulated such a scenario. The tensors are of size with rank R = 4, hence the two tensors are degenerate in mode two, but still have the first factor matrix in common. The of the for SNR = 30 db is visualized in Fig. 5. Moreover, in this figure we show the performance of the C- algorithm plus one iteration, denoted as C-+x. In this case the C- 73

5 C- C- C- + x C- + x Fig. 5: of the for complex-valued tensors with dimensions 7 3 4, tensor rank R = 4 and factor matrices with mutually correlated columns, SNR = 30 db. outperforms the framework. If they converge, the, NLS and MINF provide a more accurate estimate, but in some cases they do not converge at all. Therefore, the mean error is larger than for the frameworks. Furthermore, already a single iteration of improves the accuracy of the C-SESCI framework additionally. Furthermore, in 4 was suggested that if two tensors with different noise variances are coupled, requires normalization with respect to the different SNRs. To investigate this effect, we assume the following scenario. Two tensors, X ( and with tensor rank R = 3 of sizes and first mode in common have the following SNRs, SNR = 40 db and SNR = 0 : 60 db. The resulting as function of SNR is depicted in Fig. 6. Based on these results it can be concluded that C- does not require normalization of the noise variance (in contrast to C- C- C- normalized C- normalized normalized normalized SNR Fig. 6: as a function of SNR for complex-valued tensors with dimensions 3 8 7, tensor rank R = 3, SNR = 40 db, SNR = 0 : 60 db. VI. CONCLUSIONS In this paper, we present an extension of the framework to the efficient computation of the coupled CP decomposition. For tensors of order three, the coupled framework results in 3 + (3 N c sets of estimates of the factor matrices, where N c =,, 3 is the number of common modes. The final estimate can be selected based on different heuristics as discussed in 8 that lead to different complexity-accuracy trade-offs of the coupled framework. The coupled framework exploits the fact that the tensors have at least one factor matrix in common and guarantees that even in noisy scenarios the common mode will have the same factor matrix for the different tensors. We have compared the coupled framework with the original framework as well as with other state of the art algorithms and shown that it outperforms these algorithms. Moreover, we have shown that it provides a better accuracy than the original framework in challenging scenarios. C- performs coupled SMDs and joint LS estimates which lead to improved accuracy when those solutions are chosen as final estimates. Furthermore, even for tensors with different SNRs, normalization is not required which makes the framework significantly more robust than coupled ALS. Extensions to coupled matrix-tensor decompositions are straightforward. REFERENCES T. Kolda and B. Bader, Tensor decompositions and applications, SIAM Review, vol. 5, pp , 009. A. Cichocki, D. Mandic, A. Phan, C. Caiafa, G. Zhou, Q. Zhao, and L. de Lathauwer, Tensor decompositions for signal processing applications: From two-way to multiway, IEEE Signal Processing Magazine, vol. 3, pp , R. Bro, N. Sidiropoulos, and G. B. Giannakis, Least squares algorithm for separating trilinear mixtures. in Proc. Int. Workshop on Independent Component Analysis and Blind Signal Separation, pp. 5, January M. Rajih, P. Comon, and R. Harshman, Enhanced line search: A novel method to accelerate PARAFAC, SIAM Journal on Matrix Analysis Appl., vol. 30, pp. 48 7, September L. de Lathauwer, Parallel factor analysis by means of simultaneous matrix decompositions, Proc. First IEEE Int. Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP 005, pp. 5 8, December F. Roemer and M. Haardt, A closed-form solution for parallel factor (PARAFAC analysis, Proc. IEEE Int. Con. on Acoustics, Speech and Sig. Proc. (ICASSP 008, pp , April X. Luciani and L. Albera, Semi-algebraic canonical decomposition of multi-way arrays and joint eigenvalue decomposition, IEEE Int. Con. on Acoustics, Speech and Sig. Proc. (ICASSP 0, pp , May 0. 8 F. Roemer and M. Haardt, A semi-algebraic framework for approximate CP decomposition via simultaneous matrix diagonalization (, Signal Processing, vol. 93, pp , September F. Roemer, C. Schroeter, and M. Haardt, A semi-algebraic framework for approximate CP decompositions via joint matrix diagonalization and generalized unfoldings, in Proc. of the 46th Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, pp , November 0. 0 L. D. Lathauwer, B. D. Moor, and J. Vandewalle, A multilinear singular value decomposition, SIAM J. Matrix Anal. Appl. (SIMAX, vol., pp , 000. T. Fu and X. Gao, Simultaneous diagonalization with similarity transformation for non-defective matrices, in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 006, vol. 4, pp , May 006. K. Naskovska, M. Haardt, P. Tichavsky, G. Chabriel, and J. Barrere, Extension of the semi-algebraic framework for approximate CP decomposition via non-symmetric simultaneous matrix diagonalization, in Proc. IEEE Int. Conference on Acoustics, Speech and Signal Processing (ICASSP, March N. Vervliet, O. Debals, M. B. L. Sorber, and L. de Lathauwer, Tensorlab, Release 3.0, KU Leuven, March J. Cohen, R. Farias, and P. Comon, Joint tensor compression for coupled canonical polyadic decompositions, in Proc. 4th European Signal Processing Conference (EUSIPCO 06, August