Subtracting a best rank-1 approximation may increase tensor rank

Transcription

1 Subtracting a best rank- approximation may increase tensor rank Alwin Stegeman, Pierre Comon To cite this version: Alwin Stegeman, Pierre Comon. Subtracting a best rank- approximation may increase tensor rank. European Signal Processing Conference, Aug 009, Glasgow, United Kingdom. pp.5/08/009, 009. <hal > HAL Id: hal Submitted on 6 Nov 009 HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

2 SUBTRACTING A BEST RANK- APPROXIMATION MAY INCREASE TENSOR RANK Alwin Stegeman Pierre Comon Heymans Institute for Psychological Research, University of Groningen, Grote Kruisstraat /, 97 TS Groningen, The Netherlands phone: , fax: , a.w.stegeman@rug.nl Laboratoire IS, UMR6070 CNRS, UNS, 000, route des Lucioles, BP, 0690 Sophia Antipolis Cedex, France phone: , fax: , pcomon@unice.fr ABSTRACT Is has been shown that a best rank-r approximation of an order-k tensor may not exist when R and k. This poses a serious problem to data analysts using Candecomp/Parafac and related models. It has been observed numerically that, generally, this issue cannot be solved by consecutively computing and substracting best rank- approximations. The reason for this is that subtracting a best rank- approximation generally does not decrease tensor rank. In this paper, we provide a mathematical treatment of this property for real-valued tensors, with symmetric tensors as a special case. Regardless of the symmetry, we show that for generic tensors (which have rank or ), subtracting a best rank- approximation will result in a tensor that has rank and lies on the boundary between the rank- and rank- sets. Hence, for a typical tensor of rank, subtracting a best rank- approximation has increased the tensor rank. Keywords: tensor rank, low-rank approximation, tensor decomposition, multi-way, Candecomp, Parafac. AMS subject classifications: 5A0, 5A, 5A69, 49M7, 6H5.. INTRODUCTION Tensors of order d are defined on the outer product of d linear spaces, S l, l d. Once bases of spaces S l are fixed, they can be represented by d-way arrays. For simplicity, tensors are usually assimilated with their array representation. We assume throughout the following notation: bold italic uppercase for tensors e.g., bold uppercase for matrices e.g. T, bold lowercase for vectors e.g. a, calligraphic for sets e.g. S, and plain font for scalars e.g. X i jk, T i j or a i, will be distinguished thanks to their font. Let be a rd order tensor defined on the tensor product S S S. If a change of bases is performed in the spaces S,S,S by invertible matrices S,T,U, then the tensor representation is transformed into def (S,T,U) () whose coordinates are given by X i jk pqr S ip T jq U kr X pqr. This is known as the multi-lnearity property enjoyed by ten- This work has been partially supported by contract ANR-06-BLAN Decotes. A. Stegeman is supported by the Dutch Organisation for Scientific Research (NWO), VENI grant and VIDI grant sors. Matrices, which can be associated with linear operators, are tensors of order. The rank of a tensor is defined as the smallest number of outer product tensors whose sum equals, i.e. the smallest R such that R a r b r c r. () r Hence a rank- tensor is the outer product of vectors a,b,c and has entries X i jk a i b j c k. The decomposition of a tensor into a sum of outer products of vectors and the corresponding notion of tensor rank were first introduced and studied by 4 5. The multilinear rank of a rd order tensor is a triplet (r,r,r ), where r i denotes the rank of the set of mode-i vectors. A mode-i vector is obtained by varying the ith index and keeping other indices fixed. Usefulness. Tensors play a wider and wider role in numerous application areas including blind techniques for Telecommunications 0 8, Arithmetic Complexity 0 8 7, or Data Analysis. For instance, Independent Component Analysis was originally introduced for symmetric tensors whose rank did not exceed dimension 4. Now, it has become possible to estimate more factors than the dimension 6 9. In some applications, tensors may be symmetric only in some modes 7, or may not be symmetric nor have equal dimensions 9. In most of these applications, the decomposition of a tensor into a sum of rank- terms is relevant, since tensors entering the models to fit have a reduced rank. Matrix algebra is insufficient. The manipulation of tensors remains difficult, because of major differences between their properties when we go from second order to higher. Several of these differences have already been underlined in the past 6, e.g. (i) tensor rank often exceeds dimensions, (ii) tensor rank can be different in real and complex fields, (iii) maximal tensor rank is not generic, and is still unknown in general, (iv) computing the rank of a tensor is very difficult, (v) a tensor may not have a best low-rank approximate It has been observed numerically in 7, section 7 that a best or good rank-r approximation cannot be obtained by consecutively computing and substracting R best rank- approximations (which always exist). The reason for this is that subtracting a best rank- approximation generally does not decrease tensor rank. Hence, the deflation technique practiced for matrices (via the Singular Value Decomposition)

3 cannot generally be extended to higher-order tensors. A special case where this deflation technique works is when the tensor is diagonalizable by orthonormal multilinear transformation; see 7, section 7. In this paper, we provide a mathematical treatment of the (in)validity of a rank- deflation procedure for higher-order tensors. We consider tensors over the real field, with symmetric tensors as a special case. First, however, we discuss the problem of finding a best rank- approximate to a rd order tensor. The proofs of our results will be available in a forthcoming full-length version of this paper.. BEST RANK- APPROXIMATION Finding the best rank- approximate consists of minimizing the criterion Ψ a b c () with respect to vectors a, b and c. The solution will likely depend on the norm, and we shall restrict to the Frobenius norm: i jk X i jk. Obviously there is a scale indeterminacy in this problem, and we could impose two of these three vectors to be unit norm. We shall not do that here because the presentation would be slightly longer. Let l denote the summation over the lth index (that is the contraction operator in the lth space). For instance, the product AB T between two matrices can be written as A B; if is a rd order tensor, a is a matrix, and a b is a vector. And let us rewrite criterion () as: Ψ a b c+ a b c. (4) Proceeding as in 5, gradients with respect to the three vectors can be obtained: dψ a b c+a b c dψ b a c+b a c dψ c a b+c a b Concerning the uniqueness of a best rank- approximate, one may ask the following question: are there tensors for which the solution defined by a b c b c and dψ b dψ c 0 is not unique up to scale? We exhibit in this section a family of such tensors. If we plug the expression of a back in the equation of stationary values of b, we get that ( c) ( c) b λ b, where λ a b c 4, which means that b is an eigenvector of the matrix ( c) ( c). If the latter matrix is proportional to the identity for any c, then any b is an eigenvector. Analogously, substituting the expression for a into dψ c 0, we get that ( b) ( b) c µ c, where µ a b 4 c. If ( b) ( b) is also proportional to the identity for any b, then it follows that any (a,b,c) with a given by dψ a 0 is a stationary point. Substituting the expression of a into the criterion (4) then yields a criterion function in (b,c) for which any (b,c) is a stationary point. Hence, the function is constant and any (b, c) is a minimizer. This yields the following proposition Proposition If a tensor is such that the matrix ( c) is orthogonal for any vector c, and ( b) is orthogonal for any vector b, then has infinitely many best rank- approximates. In accordance with the usual practice, we shall represent a p p tensor with two p p matrix slices, X and X, as X X. 0 0 EXAMPLE. Let 0 0. Then for any choice of non zero vector c, the matrix ( c), obtained by linear combination of the above two matrix slices, is orthogonal. Also, for any non zero vector b, the matrix ( b) is orthogonal. Hence has infinitely many rank- approximates. Most tensors have multiple locally best rank- approximates, with one of them being better than the others (i.e., a unique best rank- approximate), as pointed out in Section. Examples will illustrate this fact. Remark. The tensor in Example has rank. Ten Berge et al. 9 showed that has no best rank- approximation, the infimum of over of rank at most being. A more general result was obtained by De Silva and Lim who showed that no tensor of rank has a best rank- approximation. Stegeman showed that any sequence of rank- approximations (n) for which (n) converges to the infimum of, features diverging components. It is shown in that the stationary points of the symmetric best rank- approximation problem are obtained as the roots of a rd degree polynomial.. BEST RANK- SUBTRACTION From now on, we restrict our discussion to tensors in the real field. De Silva and Lim, Section 7 showed that tensors (over the real field) can be transformed by invertible multilinear matrix multiplications into eight distinct canonical forms. This partitions the space R into eight distinct orbits under the action of invertible transformations in each of the modes. Table lists the canonical forms for each orbit as well as their rank and multilinear rank. These quantities are invariant under the transformations defining an orbit. This kind of classification is better known for symmetric tensors or multivariate polynomials 5. Recall the following result stated by De Silva and Lim : Lemma Let be a tensor with matrix slices X and X. (i) If X X or X X has real eigenvalues and is diagonalizable, then is in orbit G. (ii) If X X or X X has two identical real eigenvalues with only one associated eigenvector, then is in orbit D. (iii) If X X has complex eigenvalues, then is in orbit G. We shall use this lemma to verify the orbit of -dimensional rd order tensors. EXAMPLE. Consider the tensor (5) Since X X has complex eigenvalues, is in orbit G. It can be verified that has a unique best rank- approximation (6)

4 When is subtracted from we end up in orbit D, since , (7) can be transformed to the canonical form of orbit D by swapping rows within each slice. Canonical Tensor Multilinear form rank rank D 0 : 0 (0,0,0) D : (,,) D : (,,) D 0 0 : (,,) D : (,,) G : (,,) 0 0 D : (,,) 0 0 G : 0 0 (,,) Table : Canonical forms of tensors for the eight orbits under the action of invertible multilinear matrix multiplications over the real field. The letters D and G stand for degenerate (zero volume set in the 8-dimensional space of tensors) and typical (positive volume set), respectively. For tensors in the orbits of Table, we would like to know in which orbit is contained, where is a best rank- approximation of. We have the following result for the degenerate orbits of ranks and. Proposition Let be a tensor, and let be a best rank- approximation of. (i) If is in orbit D, then is in orbit D 0. (ii) If is in orbit D, D, or D, then is in orbit D. For in orbit G or D, the tensor is not restricted to a single orbit. EXAMPLE. For the canonical tensor of orbit G in Table, it can be seen that is in D. On the other hand, consider (8) For this tensor, X X has two distinct real eigenvalues. Hence, by Lemma, the tensor is in G. It can be shown that has a unique best rank- approximation such that equals the canonical tensor of orbit D in Table. EXAMPLE 4. Next, consider tensors in orbit D : , , Subtracting the best rank- approximation from these tensors amounts to replacing the element by zero. Hence, is in orbit D, D, and D, respectively. On the other hand, it can be verified numerically or analytically that for equal to the canonical tensor of orbit D in Table, is also in orbit D. Moreover, numerical experiments show that for a generic in orbit D, we have in orbit D as well. This suggests the following Conjecture 4 If is in D and is the best rank- approximate of, then almost all tensors are in D. Tensors given in Examples or were both in orbit G, and we have seen that is in orbit D in Example. For Example, this can be proven for any of the infinite best rank- approximates of. Numerically and analytically, we have not found any in orbit G for which is not in orbit D. We have no deterministic result for tensors in orbits G and G, but we still have the following result, verified almost everywhere (hence the word generic ): Proposition 5 Let be a generic tensor, and be a best rank- approximation of. Then almost all tensors are in orbit D. Hence, for typical tensors in orbit G, subtracting a best rank- approximate increases the rank to. For typical tensors in orbit G, subtracting a best rank- approximate does not affect the rank. However, some non typical tensors of rank may have a different behavior, as now shown. Proposition 6 Let be a rank- tensor with diagonal slices, and let be a best rank- approximation of. Then is in orbit D. EXAMPLE 5. Let a 0 e 0 0 d 0 h. Then ( ) ( ) ( ) ( ) ( ) ( ) a 0 0 d 0 0 e + h Then it can be seen that (I,I,T) Á, where I denotes the identity matrix and Á the diagonal tensor tensor with ones on its diagonal: Á a d, and T e h This shows that is is orbit G, and Proposition 6 implies that this is an exception to Proposition 5. Proposition 5 states that such exceptions form a set of null measure. 4. SYMMETRIC TENSORS A tensor is symmetric if its entries are invariant under arbitrary permutations of its indices. There is a bijection between the space of symmetric I I I tensors and the space of homogeneous polynomials of degree in I variables. A symmetric I I I tensor can be associated with the polynomial p(s,...,s I ) x i jk s i s j s k. (9) i jk

5 The symmetric rank of an order- symmetric tensor is the minimal number R such that 6: R a r a r a r. (0) r The orbits of symmetric tensors are given in Table. canonical form polynomial sym. rank D 0 : D : G : D : G : s s + s s s s + s s + s Table : Canonical forms of symmetric tensors and associated polynomials, for the three orbits under the action of invertible multilinear transformations over the real field. The letters D and G stand for degenerate (zero volume set in the 4-dimensional space of symmetric tensors) and typical (positive volume set), respectively. The symmetric rank of symmetric tensors of dimension can be obtained from the Sylvester Theorem, at any order. This Theorem is formulated below in the case of third order tensors. Theorem 7 (Sylvester) A symmetric tensor with associated polynomial p(s,s ) γ s + γ s s + γ s s + γ 0 s, () has a symmetric rank-r decomposition (0) if and only if there exists a vector g (g 0,...,g R ) T such that γ 0... γ R γ... γ R+ g 0, ().. γ R... γ and if the polynomial q(s,s ) g R s R + g R s R s + + g s s R + g 0 s R has R distinct real roots. EXAMPLE 6. Using Sylvester s Theorem, one can obtain the following decomposition for the representative of orbit D given in Table : 6s s (s + s ) +( s + s ) s. () In other words, the associated tensor can be written a + b c, where a, T, b, T and c 0, T. We have the following analogue of Lemma to verify the orbit of symmetric tensors of dimension. Lemma 8 Let be a symmetric tensor with matrix slices X and X. (i) If X X or X X has distinct real eigenvalues, then is in orbit G. (ii) If X X or X X has two identical real eigenvalues, then is in orbit D. (iii) If X X has complex eigenvalues, then is in orbit G. Next, we present an example of a symmetric tensor in orbit G, that has a unique best symmetric rank- approximation, such that is in orbit D. EXAMPLE 7. Let We have 0 0 X X 0. (4), (5) which has complex eigenvalues. Hence, by Lemma 8 (iii) the tensor is in orbit G. Next, we compute the best symmetric rank- approximation to, which has the form x x x x x x x x x x x x x x. (6) After some manipulations, it can be shown that the minimum of is obtained for x x /4, that is 4. (7) By subtraction, we obtain 4 and Z Z 0, (8), (9) which has a double eigenvalue. Hence, by Lemma 8 (ii) the tensor is in orbit D. In our next example, the symmetric tensor is in orbit G, and has a unique best symmetric rank- approximation, such that is in orbit D. EXAMPLE 8. Let. (0) We have X X 0 4, () which has real and distinct eigenvalues. Hence, by Lemma 8 (i) the tensor is in orbit G. Next, we compute the best symmetric rank- approximation to. It can be shown that the minimum of is obtained for. ()

6 By subtraction, we obtain and Z Z 0, (), (4) which has a double eigenvalue. Hence, by Lemma 8 (ii) the tensor is in orbit D. Finally, we have the following analogue of Proposition 5. Proposition 9 Let be a generic symmetric tensor, and be a best rank- approximation of. Then almost all tensors are in orbit D. Hence, for typical symmetric tensors with symmetric rank, subtracting a best rank- approximate increases the symmetric rank to. For typical symmetric tensors with symmetric rank, subtracting a best rank- approximate does not affect the symmetric rank. REFERENCES P. BÜRGISSER, M. CLAUSEN, and M. A. SHOKROLLAHI. Algebraic Complexity Theory, volume 5. Springer, 997. J. F. CARDOSO. Blind signal separation: statistical principles. Proc. of the IEEE, 90:009 05, Oct special issue, R.W. Liu and L. Tong eds. A. CICHOCKI and S.-I. AMARI. Adaptive Blind Signal and Image Processing. Wiley, New York, P. COMON. Independent Component Analysis. In J.-L. Lacoume, editor, Higher Order Statistics, pages 9 8. Elsevier, Amsterdam, London, P. COMON. Tensor decompositions. In J. G. McWhirter and I. K. Proudler, editors, Mathematics in Signal Processing V, pages 4. Clarendon Press, Oxford, UK, P. COMON, G. GOLUB, L.-H. LIM, and B. MOUR- RAIN. Symmetric tensors and symmetric tensor rank. SIAM J. Matrix Ana. Appl., 0():54 79, P. COMON and M. RAJIH. Blind identification of under-determined mixtures based on the characteristic function. Signal Proc., 86(9):7 8, Sept A. L. F. de ALMEIDA, G. FAVIER, and J. C. M. MOTA. Parafac-based unified tensor modeling for wireless communication systems. Signal Pro., 87():7 5, Feb L. de Lathauwer. A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization. SIAM J. Matrix Ana. Appl., 8:64 666, L. de LATHAUWER and J. CASTAING. Tensor-based techniques for the blind separation of DS-CDMA signals. Signal Proc., 87(): 6, Feb L. de Lathauwer, B. de Moor, and J. Vandewalle. On the best rank- and rank-(r,r,...,r n ) approximation of higher-order tensors. SIAM J. Matrix Ana. Appl., :4 4, 000. V. de SILVA and L.-H. LIM. Tensor rank and the illposedness of the best low-rank approximation problem. SIAM J. Matrix Ana. Appl., 0():084 7, 008. D. L. DONOHO and X. HUO. Uncertainty principles and ideal atomic decompositions. IEEE Trans. Inf. Theory, 47(7):845 86, Nov F. Hitchcock. The expression of a tensor or a polyadic as a sum of products. Journal of Mathematics and Physics, 6:64 89, F. Hitchcock. Multiple invariants and generalized rank of a p-way matrix or tensor. Journal of Mathematics and Physics, 7:9 70, T. JIANG and N. SIDIROPOULOS. Kruskal s permutation lemma and the identification of CANDE- COMP/PARAFAC and bilinear models. IEEE Trans. Sig. Proc., 5(9):65 66, Sept E. Kofidis and P. Regalia. On the best rank- approximation of higher-order supersymmetric tensors. SIAM J. Matrix Ana. App., :86 884, W. P. KRIJNEN, T. K. DIJKSTRA, and A. STEGE- MAN. On the non-existence of optimal solutions and the occurrence of degeneracy in the candecomp/parafac model. Psychometrika, 7:4 49, P. Kroonenberg. Applied Multiway Data Analysis. Wiley Series in Probability and Statistics, J. B. KRUSKAL. Three-way arrays: Rank and uniqueness of trilinear decompositions. Linear Algebra and Appl., 8:95 8, 977. N. D. SIDIROPOULOS, G. B. GIANNAKIS, and R. BRO. Blind PARAFAC receivers for DS-CDMA systems. IEEE Trans. on Sig. Proc., 48():80 8, Mar A. SMILDE, R. BRO, and P. GELADI. Multi-Way Analysis. Wiley, 004. A. STEGEMAN. Degeneracy in Candecomp/Parafac explained for p p arrays of rank p+ or higher. Psychometrika, 7():48 50, A. STEGEMAN. Degeneracy in Candecomp/Parafac explained for several three-slices arrays with a twovalued typical rank. Psychometrika, 7:60 69, A. STEGEMAN. Low-rank approximation of generic pxqx arrays and diverging components in the Candecomp/Parafac model. SIAM J. Matrix Ana. Appl., 0(): , A. STEGEMAN and L. de LATHAUWER. A method to avoid diverging components in the Candecomp/Parafac model for generic IxJx arrays. SIAM J. Matrix Ana. Appl., 0:64 68, A. STEGEMAN and N. D. SIDIROPOULOS. On kruskal s condition for the candecomp/parafac decomposition. Linear Algebra and Applications, 40:540 55, V. STRASSEN. Rank and optimal computation of generic tensors. Linear Algebra Appl., 5: , July J. M. F. ten BERGE, H. A. L. KIERS, and J. de LEEUW. Explicit Candecomp/Parafac solutions for a contrived xx array of rank three. Psychome-

7 trika, 5: , 988.