arxiv: v2 [math.sp] 23 May 2018

Transcription

1 ON THE LARGEST MULTILINEAR SINGULAR VALUES OF HIGHER-ORDER TENSORS IGNAT DOMANOV, ALWIN STEGEMAN, AND LIEVEN DE LATHAUWER Abstract. Let σ n denote the largest mode-n multilinear singular value of an I 1 I N tensor T. We prove that arxiv: v [math.sp] 3 May 018 (1) σ σ n 1 + σ n σ N (N ) T + σ n, n = 1,..., N, where denotes the Frobenius norm. We also show that at least for the cubic tensors the inverse problem always has a solution. Namely, for each σ 1,..., σ N that satisfy (1) and the trivial inequalities σ 1 1 T,..., σ I N 1 T, there always exists an I I tensor whose largest multilinear I singular values are equal to σ 1,..., σ N. For N = 3 we show that if the equality σ 1 + σ = T + σ 3 in (1) holds, then T is necessarily equal to a sum of multilinear rank-(l 1, 1, L 1 ) and multilinear rank-(1, L, L ) tensors and we give a complete description of all its multilinear singular values. We establish a connection with honeycombs and eigenvalues of the sum of two Hermitian matrices. This seems to give at least a partial explanation of why results on the joint distribution of multilinear singular values are scarce. Key words. multilinear singular value decomposition, multilinear rank, singular value decomposition, tensor AMS subject classifications. 15A69, 15A3 1. Introduction. Throughout the paper denotes the Frobenius norm of a vector, matrix, or tensor and the superscripts T, H, and denote transpose, hermitian transpose, and conjugation, respectively. We also use the empty sum/product convention, i.e., if m > n, then n ( ) = 0 and n ( ) = 1. m Let T C I1 I N. A mode-n fiber of T is a column vector obtained by fixing indices i 1,..., i n 1, i n+1,..., i N. A matrix T (n) C In I1 In 1In+1...I N formed by all mode-n fibers is called a mode-n matrix unfolding (aka flattening or matricization) of T. For notational convenience we assume that the columns of T (n) are ordered such that () the (i n, 1 + N k n m (i k 1) I l )th entry of T (n) = the (i 1,..., i N )th entry of T. l=1 l n For instance, if N = 3, i.e., T C I1 I I3, then () implies that T (1) = [T 1... T I3 ] C I1 II3, Submitted to the editors DATE. Funding: This work was funded by (1) Research Council KU Leuven: C1 project c16/15/059- nd; () F.W.O.: project G N, G N; (3) the Belgian Federal Science Policy Office: IUAP P7 (DYSCO II, Dynamical systems, control and optimization, ); (4) EU: The research leading to these results has received funding from the European Research Council under the European Union s Seventh Framework Programme (FP7/ ) / ERC Advanced Grant: BIOTENSORS (no ). This paper reflects only the authors views and the Union is not liable for any use that may be made of the contained information Group Science, Engineering and Technology, KU Leuven - Kulak, E. Sabbelaan 53, 8500 Kortrijk, Belgium and Dept. of Electrical Engineering ESAT/STADIUS KU Leuven, Kasteelpark Arenberg 10, bus 446, B-3001 Leuven-Heverlee, Belgium (ignat.domanov@kuleuven.be, alwin.stegeman@kuleuven.be, lieven.delathauwer@kuleuven.be). 1

2 IGNAT DOMANOV, ALWIN STEGEMAN, AND LIEVEN DE LATHAUWER T () = [T T 1... T T ] C I I1I3, T (3) = [vec(t 1 )... vec(t I3 )] T C I3 I1I, where T 1,..., T I3 C I1 I denote the frontal slices of T. Tensor T C I1 I N is all-orthogonal if the matrices T (1) T H (1),..., T (N)T H (N) are diagonal. The MultiLinear (ML) Singular Value Decomposition (SVD) (aka Higher- Order SVD) is a factorization of T into the product of an all-orthogonal tensor S C I1 I N and N unitary matrices U 1 C I1 I1,..., U N C I N I N, (3) T = S 1U 1 U... NU N, where n denotes the n-mode product of S and U n. Rather than giving the formal definition of n, for which we refer the reader to [3, 4, 13], we present N equivalent matricized versions of (3): (4) T (n) = U n S (n) (U N U n+1 U n 1 U 1 ) T, n = 1,..., N, where denotes the Kronecker product. For N =, i.e., for T = T 1 C I1 I, the MLSVD reduces, up to trivial indeterminacies, to the classical SVD of a matrix, T (1) = T 1 = USV H, where U = U 1, S = S (1), and V = U 1. It is known [4] that MLSVD always exists and that its uniqueness properties are similar to those of the matrix SVD. The MLSVD has many applications in signal processing, data analysis, and machine learning (see, for instance, the overview papers [13, Subsection 4.4], [17]). Here we just mention that as Principal Component Analysis (PCA) can be done by SVD of a data matrix, MLPCA can be done by MLSVD of a data tensor [5, 15, 18]. The singular values of T (n), are called the mode-n singular values of T. Since S (1) S H (1),..., S (N)S H (N) are diagonal, it follows from (4) that the ML singular values of T coincide with the ML singular values of S, which are just the Frobenius norms of the rows of S (1),..., S (N). Throughout the paper, σ n denotes the largest singular value of T (n). In the matrix case, i.e., for N =, the description of MLSVD is trivial. Indeed, the singular values of T (1) = T 1 and T () = T T 1 coincide and T (3) = vec(t 1 ) T has a single singular value T. Thus, the singular values of T (1) completely define the singular values of T () and T (3). In particular, the set of triplets (σ 1, σ, σ 3 ) coincides with the set {(x, x, y) : y x 0} R 3 whose Lebesgue measure is zero. The situation for tensors is much more complicated. It is clear that in the general case N, the sets of the mode-1,..., mode-n singular values are not independent either. The study of topological properties of the set of ML singular values of real tensors has been initiated only recently in [8] and [7]. In particular, it has been shown in [8] and [7] that, as in the matrix case, some configurations of ML singular values are not possible but, nevertheless, at least for n n tensors the set of ML singular values has a positive Lebesgue measure. In this paper we study possible configurations for the largest ML singular values, i.e., for σ 1,..., σ N. Our results are valid for real and complex tensors. The following theorem presents simple necessary conditions for σ 1, σ, and σ 3 to be the largest ML singular values of a third-order tensor. For instance, it implies that a norm-1 tensor whose largest ML singular values are equal to 0.9, 0.9, and 0.7 does not exist.

3 ON THE LARGEST MULTILINEAR SINGULAR VALUES OF HIGHER-ORDER TENSORS 3 Theorem 1. Let σ 1, σ, and σ 3 I 1 I tensor T. Then denote the largest ML singular values of an (5) (6) σ1 + σ T + σ3, σ1 + σ3 T + σ, σ + σ3 T + σ1, σ 1 1 T, σ 1 T, σ 3 1 T. I1 I I3 Figure 1 shows four typical shapes of the set {(σ 1, σ, σ 3) : σ 1, σ, σ 3 satisfy (5) (6)} (WLOG, we assumed that I 1 I ). Z N Z N σ 3 Z 1 σ 3 X X Y Y σ 1 X 1 S Y 1 σ 1 X 1 S Y 1 O σ O σ (a) I 1 < I < (b) I 1 = I < Z N Z N σ 3 σ 1 Z 1 S X Y 1 Y σ 3 σ 1 S X Y O σ O σ (c) I 1 < I = (d) I 1 = I = = I Fig. 1. The typical shapes of the set {(σ1, σ, σ 3 ) : σ 1, σ, σ 3 satisfy (5) (6)} for I 1 I (drawn for I 1 =, I = 3, = 5 and T = 1). Plot (a) is the case where all dimensions of a tensor are distinct. The points S, X 1, X, Y 1, Y, Z 1, Z and N have coordinates ( 1, 1, 1 ), I 1 I ( , 1, 1 ), (1, 1, 1 ), ( 1, , 1 ), ( 1, 1, 1 ), ( 1, 1, ), ( 1, 1, 1), and I I I I I 1 I 1 I 1 I 1 I 1 I I 1 I I 1 I 1 (1, 1, 1), respectively. Plots (b) (c) are the cases where a tensor has exactly two equal dimensions, the points Z 1 and Z were merged into one point Z and the points X 1 and X were merged into one point X. Plot (d) is the case where all three dimensions of a tensor are equal to each other, I 1 = I = = I. In this case, the points Y 1 and Y were merged into one point Y, so S, X, Y, and Z have the coordinates ( 1 I, 1 I, 1 I ), (1, 1 I, 1 I ), ( 1 I, 1, 1 I ), and ( 1 I, 1, 1), respectively. By Corollary 3, any I point (σ1, σ, σ 3 ) of the polyhedron SXY ZN in plot (d) is feasible, i.e., there exists a norm-1 tensor T C I I I whose squared largest multilinear singular values are σ1, σ, and σ 3. The volume of SXY ZN equals half of the volume of the cube, i.e., 1 (1 1 I )3.

4 4 IGNAT DOMANOV, ALWIN STEGEMAN, AND LIEVEN DE LATHAUWER One can easily verify that if σ 1, σ and σ 3 satisfy (5) (6) for I 1 = I = = and T = 1, then σ 1, σ and σ 3 are the largest ML singular values of the tensor T with mode-1 matrix unfolding T (1) = [T 1 T ] = σ 1 +σ +σ σ 1 σ σ 3 1+σ 3 σ 1 σ 1+σ σ 1 σ 3 0 The proof of the following result relies on a similar explicit construction of an I 1 I tensor T. Theorem. Let I 1 I and σ 1, σ, σ 3 satisfy (5) and the following three inequalities (7) σ 1 1 I1 T,. (8) (9) (I I 1 )σ 1 + (I 1 I I )σ 3 + (1 I ) T 0, (I I 1 )σ 1 + (I 1 I I )σ + (1 I ) T 0. Then there exists an I 1 I tensor T such that 1. all entries of T are non-negative;. T is all-orthogonal; 3. the largest ML singular values of T are equal to σ 1, σ and σ 3. Conditions (6) and (7) (9) mean that the point (σ 1, σ, σ 3) belongs to the trihedral angle SX 1 Y 1 Z 1 and S X Y Z, respectively, where S has coordinates ( 1 I 1, 1 I 1, 1 I 1 ). The gap between the necessary conditions in Theorem 1 and the sufficient conditions in Theorem, i.e., the set (10) {(σ 1, σ, σ 3) : (5) (6) hold and at least one of (7) (9) does not hold}, is shown in Figure c. One can easily verify that the gap is empty only for I 1 = I =. Corollary 3. Let σ 1, σ and σ 3 satisfy (5) (6) for I 1 = I = = I. Then there exists an I I I tensor T such that 1. all entries of T are non-negative;. T is all-orthogonal; 3. the largest ML singular values of T are equal to σ 1, σ and σ 3. Thus, the conditions in Theorem 1 are not only necessary but also sufficient for σ 1, σ, and σ 3 to be feasible largest ML singular values of a cubic third-order tensor. Figure 1d shows the set of feasible triplets (σ1, σ, σ3) of an I I I tensor. We do not have a complete view on the feasibility of points in (10). In Section 3 we obtain particular results on the (non)feasibility of the points S( 1 I 1, 1 I, 1 ), X 1 (1 1 I + 1 3, 1 I, 1 ), and Y 1 ( 1 I 1, 1 1 I 1 + 1, 1 ). Namely, we show that if I 1 < I and = I 1 I 1, then the point S is not feasible and if = I 1 I, then the point S is feasible but the points X 1 and Y 1 not. It worth mentioning a link with scaled all-orthonormal tensors introduced recently in [6]. Tensor T C I1 I N is scaled all-orthonormal [6, Definition ] if at least N 1 of the N matrices T (1) T H (1),..., T (N)T H (N) are multiples of the identity matrix. It is 1 clear that if the largest mode-n singular value of a norm-1 tensor is In, then all moden singular values are also 1 In. Thus, feasibility of a point belonging to the segment

5 ON THE LARGEST MULTILINEAR SINGULAR VALUES OF HIGHER-ORDER TENSORS 5 N N Z Z σ 3 Z 1 X 1 Y σ 3 S X Y σ 1 S Y 1 σ 1 O σ O σ (a) σ 1, σ, σ 3 satisfy (5) (6) (b) σ 1, σ, σ 3 satisfy (5) and (7) (9) N Z σ 3 Z 1 X 1 S Y σ 1 S Y 1 O σ (c) the set in eq. (10) Fig.. Gap between the necessary conditions in Theorem 1 and the sufficient conditions in Theorem for I 1 < I < (drawn for I 1 =, I = 5, = 7 and T = 1). The point S has coordinates ( 1, 1, 1 ). The set in plot (c) is the difference of the set in plot (a) and the set in I 1 I 1 I 1 plot (b). SX 1 (resp. SY 1 or SZ 1 ) is equivalent to the existence of a norm-1 I 1 I tensor T such that T () T H () = 1 I I I, T (3) T H (3) = 1 I I3 (resp. T (1) T H (1) = 1 I 1 I I1, T (3) T H (3) = 1 I I3 or T (1) T H (1) = 1 I 1 I I1, T () T H () = 1 I I I ), i.e., to the existence of a scaled all-orthonormal tensor T. The following results generalize Theorem 1 and Corollary 3 for Nth-order tensors. Theorem 4. Let σ 1,..., σ N denote the largest ML singular values of an I 1 I N tensor T. Then (11) (1) σ1 + + σn 1 + σn σn (N ) T + σn, n = 1,..., N, T σ 1 1 T,..., T σ N 1 T. I1 IN

6 6 IGNAT DOMANOV, ALWIN STEGEMAN, AND LIEVEN DE LATHAUWER Theorem 5. Let σ 1,..., σ N satisfy (11) (1) for I 1 = = I N = I. Then there exists an I I tensor T such that 1. all entries of T are non-negative;. T is all-orthogonal; 3. the largest ML singular values of T are equal to σ 1,..., σ N. Thus, the conditions in Theorem 4 are not only necessary but also sufficient for σ 1,..., σ N to be feasible largest ML singular values of an I I tensor. This result was independently proved for real tensors in [16]. Theorems 1,, 4, and 5 are proved in Section. It is natural to ask what happens if some inequalities in (5) are replaced by equalities. Obviously, the three equalities in (5) hold if and only if σ 1 = σ = σ 3 = T, implying that T (1), T (), and T (3) are rank-1 matrices. Hence all the remaining ML singular values of T are zero. Similarly, the two equalities σ1 + σ = T + σ3 and σ1 + σ3 = T + σ are equivalent to σ 1 = T and σ = σ 3, implying that rank(t (1) ) = 1 and rank(t () ) = rank(t (3) ) =: L, i.e., T is an ML rank-(1, L, L) tensor, where L min(i, ). It is clear that in this case the remaining nonzero mode- and mode-3 singular values of T also coincide and may take any positive values whose squares sum up to T σ. In Section 4 we characterize the tensors T for which the single equality σ1 + σ = T + σ3 holds. We show that T is necessarily equal to a sum of ML rank-(l 1, 1, L 1 ) and ML rank-(1, L, L ) tensors and give a complete description of all its ML singular values. The description relies on a problem posed by H. Weyl in 191: given the eigenvalues of two n n Hermitian matrices A and B, what are all the possible eigenvalues of A + B? The following answer was conjectured by A. Horn in 196 [9] and has been proved through the development of the theory of honeycombs in [10, 11] (see also [, 1]). Let If λ i ( ) denote the ith largest eigenvalue of a Hermitian matrix. (13) α i = λ i (A), β i = λ i (B), γ i = λ i (A + B), then α i, β i, and γ i satisfy the trivial equality (14) γ γ n = α α n + β β n and the list of linear inequalities (15) k K γ k i I α i + j J β j, (I, J, K) T n r, 1 r n 1, where I = {i 1,..., i r }, J = {j 1,..., j r }, K = {k 1,..., k r } are subsets of {1,..., n} and Tr n denotes a particular finite set of triplets (I, J, K). (The construction of Tr n is given in Appendix A.) The inverse statement also holds: if α i, β i, and γ i satisfy (14) and (15), then there exist n n Hermitian matrices A, B, and C such that (13) holds. We have the following results. Theorem 6. Let σ 1 + σ = T + σ 3. Then T is a sum of ML rank-(l 1, 1, L 1 ) and ML rank-(1, L, L ) tensors, where L 1 min(i 1, ) and L min(i 1, ).

7 ON THE LARGEST MULTILINEAR SINGULAR VALUES OF HIGHER-ORDER TENSORS 7 Theorem 7. Let σ 1 + σ = T + σ 3. Then the values σ 1 = σ 11 σ 1 σ 1I1 0, σ = σ 1 σ σ I 0, σ 3 = σ 31 σ 3 σ 3I3 0, are the mode-1, mode-, and mode-3 singular values of an I 1 I tensor T, respectively, if and only if σ σ 1I 1 = σ σ I = σ σ 3 = T, σ 1i = 0 for i > min(i 1, ), σ i = 0 for i > min(i, ), and (14) and (15) hold for { { σ1i+1 α i =, i min(i 1, ) σi+1, β i =, i min(i, ) (16), γ i = σ3i+1, 0, otherwise 0, otherwise and n = 1. Example 8. If n =, then T1 = {(i, j, k) : k = i + j 1, 1 i, j, k } = {(1, 1, 1), (1,, ), (, 1, )} (see Appendix A). By Horn s conjecture, the equality γ 1 + γ = α 1 + α + β 1 + β together with the inequalities (also known as the Weyl inequalities) (17) γ 1 α 1 + β 1, γ α 1 + β, γ α + β 1, characterize the values α 1, α, β 1, β, γ 1, γ that can be eigenvalues of Hermitian matrices A, B, and A + B. Let σ 11 + σ 1 = T + σ 31. From Theorem 7 and (17) it follows that the values σ 11 σ 1 σ 13 0, σ 1 σ σ 3 0, and σ 31 σ 3 σ 33 0, are the mode-1, mode-, and mode-3 singular values, respectively, of a tensor T if and only if σ 11 + σ 1 + σ 13 = σ 1 + σ + σ 3 = σ 31 + σ 3 + σ 33 = T, σ 3 σ 1 + σ, σ 33 σ 1 + σ 3, σ 33 σ 13 + σ. Horn s Conjecture has recently also been linked to singular values of matrix unfoldings in the Tensor Train format [14].. Proofs of Theorems 1,, 4, and 5. The following lemma will be used in the proof of Theorem 1. Lemma 9. Let H = (H ij ) I3 i,j=1 CI3I1 I3I1 be a positive semidefinite matrix consisting of the blocks H ij C I1 I1. Then (18) λ max (H H I3 ) + λ max (H) tr(h) + λ max (Φ(H)). where Φ(H) denotes the matrix with the entries (Φ(H)) ij = tr(h ij ) and λ max ( ) denotes the largest eigenvalue of a matrix. Proof. To get an idea of the proof we refer the reader to the mathoverflow page [1] where the case = was discussed. Here we present a formal proof for. Let

8 8 IGNAT DOMANOV, ALWIN STEGEMAN, AND LIEVEN DE LATHAUWER H = R w r wr H, where w r are orthogonal and w r = [w1r T... wi T 3r ]T with w kr C I3. First, we rewrite (18) in terms of w kr, 1 k, 1 r R. WLOG, we can assume that w 1 = max w r. Hence, r (19) λ max (H) = w 1 = It is clear that Hence H ij = w k1. R w ir wjr, H 1 i, j. (0) λ max (H H I3 ) = max Since H = R w r wr H, it follows that x =1 (1) tr(h) = Since it follows that () where (H kk x, x) = max R w r. ( R ) Φ(H) ij = tr(h ij ) = tr w ir wjr H = Φ(H) = x =1 R wjrw H ir = R w R 1rw T 1r... w1rw T r..... = wi T 3r w 1r... wi T 3r w I 3r w R T 1r. [ ] R w1r... wi 3r = Wr T Wr, w T r W r := [w 1r... w I3r] C I1 I3. Now we prove (18). By (19), (0), the Cauchy inequality, and (1), R (w kr, x). w T irw jr, (3) λ max (H) + λ max (H H I3 ) = [ I3 ] R w 1 + max (w k1, x) + (w kr, x) x =1 r= [ I3 ] [ R I3 ] w 1 + max (w k1, x) + w r = tr(h) + max (w k1, x). x =1 x =1 r=

9 ON THE LARGEST MULTILINEAR SINGULAR VALUES OF HIGHER-ORDER TENSORS 9 To complete the proof of (18) we should show that [ I3 ] (w k1, x) λ max (Φ(H)). max x =1 This can be done as follows [ I3 (4) max x =1 (w k1, x) ] = max x =1 [ I3 ] x H w k1 wk1x H ( = λ max W1 W1 H ) = ( ( ) R ) λ max W H 1 W 1 λmax Wr H W r = λ max (Φ(H) ) = λ max (Φ(H)). Now we are ready to prove Theorem 1. Proof of Theorem 1. The three inequalities in (6) are obvious. We prove that σ1 + σ T + σ3. The proofs of the inequalities σ1 + σ3 T + σ and σ + σ3 T + σ1 can be obtained in a similar way. By definition of ML singular values, σ 1 = λ max (T (1) T H (1) ) = λ max(t 1 T H T I3 T H ), σ = λ max (T H () T ()) = λ max (T T () T () ) = λ max(h), where H = T T () T () = T 1 T H 1... T 1 T H T I3 T H 1... T I3 T H Since vec(t i ) T (vec(t j ) T ) H = tr(t i T H j ), it follows that where σ 3 = λ max (T (3) T H (3) ) = λ max(φ(h)), Φ(H) = tr(t 1 T H 1 )... tr(t 1 T H ) tr(t I3 T H 1 )... tr(t I3 T H ) Since T = tr(h), the inequality σ 1 + σ T + σ 3 is equivalent to λ max (T 1 T H T I3 T H ) + λ max (H) tr(h) + λ max (Φ(H)), which holds by Lemma 9. Proof of Theorem. The proof consists of three steps. In the first step we construct all-orthogonal and non-negative I 1 I tensors S, X, Y, Z, and N whose squared largest ML singular values are the coordinates of S ( 1 I 1, 1 I 1, 1 I 1 ), X (1, 1 I, 1 I ), Y ( 1 I 1, 1, 1 I 1 ), Z ( 1 I 1, 1 I 1, 1), and N(1, 1, 1), respectively (see Figure b). Then we show that because of the zero patterns of S, X, Y, Z, and N, the tensor (5) T = ( t S S + t X X + t Y Y + t Z Z + t N N ) 1, is all-orthogonal for any non-negative values t S, t X, t Y, t Z, t N. The superscripts and 1 in (5) denote the entrywise operations. Finally, in the third step, we

10 10 IGNAT DOMANOV, ALWIN STEGEMAN, AND LIEVEN DE LATHAUWER find non-negative values t S, t X, t Y, t Z, t N such that T is norm-1 tensor whose squared largest ML singular values are equal to σ1, σ, and σ3. Step 1. Let π denote the cyclic permutation π : 1 I 1 I The tensors S, X, Y, and Z are defined by { 1 I1 S,ijk =, if j = π (i) and 1 i, k I 1, 0, otherwise, 1 I, if j = π (i), i = 1, and 1 k I 1, X,ijk = 1 I, if i = 1 and I 1 < j = k I, 0, otherwise, { 1 Y,ijk = I1, if j = π (i), j = 1, and 1 k I 1, 0, otherwise, { 1 Z,ijk = I1, if j = π (i), k = 1, and 1 i I 1, 0, otherwise, and the tensor N, by definition, has only one nonzero entry, N 111 = 1. For instance, if I 1 = I = =, then the first matrix unfoldings of S, X, Y, Z, and N have the form S,(1) = 1 [ ] , X ,(1) = 1 [ ] , Y,(1) = 1 [ ] , Z ,(1) = 1 [ ] , N (1) = [ ] Step. It is clear that the (i, j, k)th entry of a linear combination of S, X, Y, Z, and N may be nonzero only if j = π (i) and 1 i, k I 1 or i = 1 and I 1 < j = k I. The same is also true for T defined in (5). One can easily check that each column of T (1), T (), and T (3) contains at most one nonzero entry, implying that T is allorthogonal tensor. Step 3. From the construction of the all-orthogonal tensors S, X, Y, Z, and N it follows that their largest ML singular values are equal to the Frobenius norms of the first rows of their matrix unfoldings. Thus, the same property should also hold for T whenever the values t S, t X, t Y, t Z, and t N are non-negative. Now the result follows from the fact that the polyhedron in Figure b is the convex hull of the points S, X, Y, Z, and N. We can also write the values of t S, t X, t Y, t Z, and t N explicitly. We set f(σ 1, σ, σ 3) := (I 1 I + I I 1 )σ 1 + (I 1 1)I σ + (I 1 1)I σ 3 + ( I 1 I I ). If (σ 1, σ, σ 3) belongs to the tetrahedron X Y Z N, i.e., f(σ 1, σ, σ 3) 0, then I t X = (I 1) (1 + σ 1 σ σ3), I 1 t Y = (I 1 1) (1 + σ σ1 σ3), I 1 t Z = (I 1 1) (1 + σ 3 σ1 σ),

11 ON THE LARGEST MULTILINEAR SINGULAR VALUES OF HIGHER-ORDER TENSORS11 t N = 1 t X t Y t Z = f(σ 1, σ, σ 3) (I 1 1)(I 1), t S = 0. If (σ 1, σ, σ 3) belongs to the tetrahedron X Y Z S, i.e., f(σ 1, σ, σ 3) 0, then t X = I 1 I 1 1 (σ 1 1 I 1 ), t Y = I 1 I 1 1 (σ 1 ) + (I I 1 )I 1 I 1 (I1 1)I (σ1 1 ), I 1 t Z = I 1 I 1 1 (σ 3 1 ) + (I I 1 )I 1 I 1 (I1 1)I (σ1 1 ), I 1 t S = 1 t X t Y t Z = f(σ 1, σ, σ 3)I 1 I (I 1 1), t N = 0. Proof of Theorem 4. The inequalities in (1) are obvious. We prove that (6) σ σ N 1 (N ) T + σ N. The proofs of the remaining N 1 inequalities in (11) can be obtained in a similar way. The proof of (6) consists of two steps. In the first step we reshape T into thirdorder tensors T [1],..., T [N ] and compute their matrix unfoldings. In this step we will make use of () for N = 3. For the reader s convenience and for a future reference here we write a third-order version of () explicitly: if X C I J K, then for all values of indices i, j, and k (7) the (i, j + (k 1)J)th entry of X (1) = the (j, i + (k 1)I)th entry of X () = the (k, i + (j 1)I)th entry of X (3) = the (i, j, k)th entry of X. In the second step, we apply the first inequality in (5) to each tensor T [n], then we sum up the obtained inequalities and show that the result coincides with inequality (6). Step 1. Let n {1,..., N }. A third-order tensor T [n] C I1 In In+1 In+ I N is constructed as follows: the (i 1 + n (i k 1) k= l=1 I l, i n+1, i n+ + N k=n+3 (i k 1) l=n+ I l )th entry of T [n] is equal to the (i 1,..., i N )th entry of T. Now we apply (7) for X = T [n] and i = i 1 + n (i k 1) k= l=1 I l, j = i n+1, k = i n+ + N k=n+3 (i k 1) l=n+ I l.

12 1 IGNAT DOMANOV, ALWIN STEGEMAN, AND LIEVEN DE LATHAUWER After simple algebraic manipulations, we obtain that (8) the (i 1 + n (i k 1) k= the (i n+1, 1 + the (i n+ + N k= k n+1 N k=n+3 l=1 (i k 1) (i k 1) I l, i n+1 + l=1 l n+1 l=n+ the (i 1,..., i N )th entry of T. N k=n+ (i k 1) l=n+1 I l )th entry of T [n] () = n+1 I l, i 1 + k= (i k 1) l=1 I l )th entry of T [n] (1) = I l )th entry of T [n] (3) = Step. From (8) and () it follows that (9) (30) (31) T [1] (1) = T (1), T [n] () = T (n+1), 1 n N, T [N ] (3) = T (N). Comparing the expressions of T [n] (1) and T[n] (3) in (8), we obtain that (3) T [n] (3) = ( T [n+1] (1) By Theorem 1, for every n {1,..., N } ) T, 1 n N 3. (33) σ max(t [n] (1) ) + σ max(t [n] () ) T [n] + σ max(t [n] (3) ) = T + σ max(t [n] (3) ), where σ max ( ) denotes the largest singular value of a matrix. Substituting (9) (3) into (33) we obtain σ1 + σ T + σmax(t [1] (3) ) = T + σmax(t [] (1)), n = 1, σmax(t [] (1) ) + σ 3 T + σmax(t [] (3) ) = T + σmax(t [3] (1)), n =,. σ max(t [N 3] (1) ) + σ N T + σ max(t [N 3] (3) ) = T + σ max(t [N ] (1) ), n = N 3, σ max(t [N ] (1) ) + σ N 1 T + σ max(t [N ] (3) ) = T + σ N, n = N. Summing up the above inequalities and canceling identical terms on the left- and right-hand side we obtain (6). Proof of Theorem 5. It can be checked that a polyhedron described by the inequalities in (11) (1) is a convex hull of N N points { } 1 (34) V = {(α 1,..., α N ), α n I, 1 and at least two of α-s are equal to 1 I }. To show that each point of the polyhedron is feasible we proceed as in the proof of Theorem.

13 ON THE LARGEST MULTILINEAR SINGULAR VALUES OF HIGHER-ORDER TENSORS13 First, for each (α 1,..., α N ) V we construct an all-orthogonal and non-negative I I tensor P α1,...,α N whose squared largest ML singular values are α 1,..., α N. Let π denote the cyclic permutation π : 1 I I 1 1. The tensor P 1 I,..., 1 I is defined by P 1 I,..., 1 I i 1,...,i N = {I N 1, if i = π i3+ +i N N+ (i 1 ), 0, otherwise, and the tensor P 1,...,1, by definition, has only one nonzero entry, P1,..., = 1. Let (α 1,..., α N ) V \ {( 1 I,..., 1 I ), (1,..., 1)} and j 1,..., j k denote all indices such that α j1 = = α jk = 1. Then the tensor P α1,...,α N is defined by P α1,...,α N i 1,...,i N = {I N 1 k, if i = π i3+ +i N N+ (i 1 ) and i j1 = = i jk = 1, 0, otherwise. For instance, if N = 4 and I =, then the first matrix unfolding of P 1 I,..., 1 I is given by P 1 I,..., 1 I (1) = 1 [ ] and the first matrix unfoldings of the remaining tensors P α1,...,α N can be obtained from P 1 I,..., 1 I (1) by rescaling and introducing more zeros. It is clear that the (i 1,..., i N )th entry of a linear combination of P 1 I,..., 1 I,..., P 1,...,1 may be nonzero only if i = π i3+ +i N N+ (i 1 ). The same is also true for T defined by T = (α 1,...,α N ) V t α1,...,α N P α1,...,α N where, as before, the superscripts and 1 denote the entrywise operations. One can easily check that each column of T (1),..., T (N) contains at most one nonzero entry, implying that T is all-orthogonal tensor. Finally, from the construction of the all-orthogonal tensors P α1,...,α N it follows that their largest ML singular values are equal to the Frobenius norms of the first rows of their matrix unfoldings. Thus, the same property should also hold for T whenever the values t α1,...,α N are non-negative. Now the result follows from the fact that the polyhedron described by the inequalities in (11) (1) is a convex hull of points in V. Note that in the proof of Theorem 5 the constructed tensor T has squared singular values in the nth mode equal to σn, 1 I 1 (1 σ 1 n),..., I 1 (1 σ n), i.e., the I 1 smallest singular values in the nth mode are equal. 3. Results on feasibility and non-feasibility of the points S, X 1, and Y 1. Throughout this subsection we assume that T is a norm-1 tensor. In the following example we show that it may happen that S is the only feasible point in the plane through the points S, X 1, and Y 1, i.e., the plane σ 3 = 1. Example 10. Let = I 1 I and T C I1 I I3. Assume that σ 3 = 1. Then T H (3) T (3) = 1 I I3. Since T (3) is a square matrix, it follows that T (3) is a scalar 1,

14 14 IGNAT DOMANOV, ALWIN STEGEMAN, AND LIEVEN DE LATHAUWER multiple of a unitary matrix, T (3) = 1 I3 U. One can easily verify (see [6, p. 65]), that T H (1) T (1) = 1 I 1 I I1 and T H () T () = 1 I I I. Hence, σ 1 = 1 I 1 and σ = 1 I. Thus, the points X 1 and Y 1 are not feasible. From Example 10 it follows that the point S is feasible if I 1 =, I = 3, and = 6. The point S is also feasible if I 1 =, I = 3, and = 4. Indeed, let T be an 3 4 tensor with mode-3 matrix unfolding T (3) = Then one can also easily verify that T (1) T H (1) = 1 I, T () T H () = 1 3, and T (3) T H (3) = 1 4 I 4. The following result implies that in the intermediate case I 1 =, I = 3, and = 5 the point S is not feasible. Theorem 11. Let = I 1 I 1, T C I1 I I3, and T (3) T H (3) = 1 I I3. Then the following statements hold: (i) if T (1) T H (1) = 1 I 1 I I1, then I 1 I ; (ii) if T () T H () = 1 I I I, then I I 1 ; (iii) if the point S is feasible, then I 1 = I. Proof. (i) Let T (3) = [t 1... t I1I ]. Then the identity T (1) T H (1) = 1 I 1 I I1 is equivalent to the system (35) t H i 1 t i + t H I 1+i 1 t I1+i + + t H I 1(I 1)+i 1 t I1(I 1)+i = 0, t i1 + t I1+i t I1(I 1)+i 1 = 1 I 1, 1 i 1 < i I 1. Since T (3) T H (3) = 1 I I3, the matrix T (3) C I3 I1I can be extended to a unitary matrix [ ] T(3) C I1I I1I, where a C I1I is a vector such that T (3) a = 0 a T and a = 1. Hence, or [ T H (3) a ] [ T (3) a T ] = 1 I II I 3 3 (36) t H i t j + ā i a j = 0 for i j and t i + a i = 1, 1 i < j I 1 I. From (35) (36) it follows that ā i1 a i + ā I1+i 1 a I1+i + + ā I1(I 1)+i 1 a I1(I 1)+i = 0, a i1 + a I1+i a I1(I 1)+i 1 = 1 I 1, 1 i 1 < i I 1. Thus, the vectors [a i a I1+i... a I1(I 1)+i] T C I, 1 i I 1 are nonzero and mutually orthogonal. Hence, I 1 I. (ii) The proof is similar to the proof of (i). (iii) Since S is feasible, it follows that T (1) T H (1) = 1 I 1 I I1 and T () T H () = 1 I I I. Hence, by (i) and (ii), I 1 = I.

15 ON THE LARGEST MULTILINEAR SINGULAR VALUES OF HIGHER-ORDER TENSORS15 4. The case of at least one equality in (5). The following two lemmas will be used in the proof of Theorem 7. Lemma 1. Let H and Φ(H) be as in Lemma 9. Then the equality in (18) holds if and only if H can be factorized as (37) H = [vec(w 1 ) G x][vec(w 1 ) G x] H, where (i) W 1 C I1 I3 and x is a principal eigenvector of W 1 W H 1, i.e., W 1 W H 1 x = λ max (W 1 W H 1 )x, x = 1; (ii) the matrix G = [g... g R ] C I3 (R 1) has orthogonal columns; (iii) G T W H 1 x = 0; (iv) λ max (W H 1 W 1 ) = λ max (W H 1 W 1 + G G T ). Moreover, if (37) and (i) (iv) hold, then (38) (39) (40) σ( H kk ) = σ(w 1 W H 1 + G xx H ), σ(h) = { W 1, g,..., g R, 0,..., 0}, σ(φ(h)) = σ(w1 H W 1 + G G T ), where σ( ) denotes the spectrum of a matrix. Proof. The proof essentially relies on the proof of Lemma 9 so we use the same notations and conventions as in the proof of Lemma 9. Derivation of (38) (40). Assume that 37 and (i) (iv) hold. Then Hence H = R vec(w r ) vec(w r ) H, where W r = xgr T for r =,..., R. H kk = R R W r Wr H = W 1 W1 H + xgr T grx H = W 1 W1 H + G xx H, r= which implies (38). By (ii), (iii), and the convention x = 1 in (i), the vectors vec(w r ) are mutually orthogonal, which implies (39). Finally, by (), Φ(H) = R R Wr T Wr = W1 T W1 + g r x T x gr H = W1 T W1 + GG H, which implies (40). Sufficiency. By (i) and (38), λ max ( H kk ) = λ max (W 1 W H 1 ) + G.

16 16 IGNAT DOMANOV, ALWIN STEGEMAN, AND LIEVEN DE LATHAUWER By (iv) and (ii), W 1 λ max (W H 1 W 1 ) λ max (G G T ) = max r R g r. Thus, by (39), λ max (H) = W 1 and tr(h) = W 1 + G. By (iv) and (40), λ max (Φ(H)) = λ max (W1 H W 1 ). Thus, the left- and right-hand sides of (18) are equal to λ max (W 1 W1 H ) + W 1 + G. Necessity. It is clear that the equality in (18) holds if and only it holds in (3) and (4). So we replace the inequality signs in (3) and (4) with an equality sign. From the first line of (4) it follows that x satisfies (i). By the Cauchy inequality, the equality R R (w kr, x) = w r in (3) would imply that for some c kr C. Hence, r= r= w kr = c kr x, k = 1,...,, r =,..., R. (41) w r = [w T 1r... w T r] T = [c 1r... c I3r] T x = g r x, r =,..., R. Since H = R w r wr H, it follows that H = [w 1... w R ][w 1... w R ] H = [w 1 g x... g R x][w 1 g x... g R x] H, which coincides with (37). The mutual orthogonality of w,..., w R and the orthogonality of w 1 to w,..., w R implies (ii) and (iii), respectively. By (41), W r = xgr T for r =,..., R. Hence, the equality ( ( ) R ) λ max W H 1 W 1 = λmax Wr H W r in (4) would imply (iv): λ max ( W H 1 W 1 ) = λmax (W H 1 W 1 + ) R grx H xgr T = λ max (W1 H W 1 + G G T ). Lemma 13. (i) Let W 1, G, and x satisfy conditions (i) (iv) of Lemma 1, H be defined as in (37) and L := λ max (W H 1 W 1 ). Then there exist ( 1) ( 1) positive semidefinite matrices A and B such that (4) rank(a) min(i 1, ) 1, rank(b) = R 1, L λ max (A + B) and (43) σ( H kk ) = {L + tr(b), λ 1 (A),..., λ min(i1,) 1(A), 0,..., 0}, }{{} I 1

17 ON THE LARGEST MULTILINEAR SINGULAR VALUES OF HIGHER-ORDER TENSORS17 (44) (45) σ(h) = {L + tr(a), λ 1 (B),..., λ R 1 (B), 0,..., 0}, }{{} I 1 R σ(φ(h)) = {L} σ(a + B). (ii) Let a positive value L and ( 1) ( 1) positive semidefinite matrices A and B satisfy (4). Then there exists a matrix H of form (37) such that (43) (45) hold. Proof. (i) Let p be a principal eigenvector of W 1 W1 H, i.e., W1 H W 1 p = Lp, p = 1. Then, by (iv), G G T p = 0. Let U p be an unitary matrix whose first column is p. Then (46) U H p W H 1 W 1 U p = [ ] L 0, U H 0 A p G G T U p = [ ] 0 0, 0 B where A and B are ( 1) ( 1) positive semidefinite matrices. It is clear that (47) (48) λ k (A) = λ k+1 (W1 H W 1 ), k = 1,..., 1, { λ k (B) = λ k (G G T g k+1, k = 1,..., R 1, ) = 0, k = R,..., 1. Now, (44) follows from (39) and (46) and (45) follows from (40) and (46). To prove (43) we rewrite (38) as (49) σ( H kk ) = {λ max (W 1 W H 1 ) + G, λ (W 1 W H 1 ),..., λ I1 (W 1 W H 1 )}. Since the nonzero eigenvalues of W 1 W H 1 coincide with those of W H 1 W 1 and G = tr(b) it follows that (49) is equivalent to (43). (ii) Let H be defined as in (37), where [ ] L 0 W 1 =, G = US 1, x = [ ] T, 0 W1 W 1 is an (I 1 1) ( 1) matrix [ such] that W 1 H W 1 = A and USU H is the reduced 0 0 singular value decomposition of. One can easily verify conditions (i) (iv) in 0 B Lemma 1 hold. Hence, by Lemma 1, (38) (40) also hold. Substituting W 1, G, and x in (38) (40) we obtain (43) (45). Proof of Theorem 6. Let H = T T () T (). Since σ 1 + σ = T + σ 3, it follows that equality (18) holds. Hence, by Lemma 1, H can be factorized as in (37). Therefore, there exists an I R matrix V whose columns are orthonormal and such that T T () = [vec(w 1) G x]v H, or equivalently, T k = [w 1k x[g k1... g kr ]]V H, k = 1,...,. Let W and G denote the I 1 I tensors whose kth frontal slice is [w 1k ]V H and [0 x[g k1... g kr ]]V H, respectively. It is clear that T = W + G, W is ML rank- (L 1, 1, L 1 ) tensor, and G is ML rank-(1, L, L ) tensors, where L 1 min(i 1, ) and L min(i 1, ).

18 18 IGNAT DOMANOV, ALWIN STEGEMAN, AND LIEVEN DE LATHAUWER Proof of Theorem 7. Let H = T T () T (). Then (50) (51) (5) σ 11 σ 1 σ 1I 1 0, are the eigenvalues of i=1 H ii = T (1) T H (1), σ1 σ σi 0, are the first I eigenvalues of H, σ31 σ3 σ3i 3 0, are the eigenvalues of Φ(H) = T (3) T H (3). Necessity. By Lemmas 1 and 13(i), there exist ( 1) ( 1) positive semidefinite matrices A and B such that (4) (45) hold. Thus, by (16) and (50) (5), the values α i, β i, and γ i are eigenvalues of A, B, and A+B, respectively. Hence, by Horn s conjecture, (14) and (15) hold. Sufficiency. Since (14) and (15) hold, from Horn s conjecture it follows that there exist ( 1) ( 1) positive semidefinite matrices A and B such that α i, β i, and γ i are eigenvalues of A, B, and A+B, respectively. Hence, by Lemma 13(ii), there exists a matrix H of form (37) such that (43) (45) hold. By (16) and (44), rank H 1+R I. Let V be an I R matrix whose columns are orthonormal and let T denote an I 1 I tensor with mode- matrix unfolding T () = V [vec(w 1 ) G x] T. Then, H = T T () T (). The proof now follows from (50) (5). 5. Conclusion. In the paper we studied geometrical properties of the set Σ I1,I, := { (σ 11,..., σ 1I 1, σ 1,..., σ I, σ 31,..., σ 3 ) : σ nk is the kth largest mode-n singular value of an I 1 I norm-1 tensor T }, where for each n = 1,, 3 the values σ nk are sorted in descending order. Let π denote a projection of R I1+I+I3 onto the first, (I 1 +1)th, and (I 1 +I +1)th coordinates. We have shown that there exist two convex polyhedrons of positive volume such that the set π(σ I1,I, ) R 3 contains one polyhedron (Theorem ) and is contained in another (Theorem 1). We have also shown that both polyhedrons coincide for cubic tensors, i.e., for I 1 = I = (Corollary 3), and can be different in the non-cubic case (Example 10 and Theorem 11). In Theorem 7, we considered the case where the largest ML singular values of T satisfy the equality σ 11 + σ 1 = 1 + σ 31 or σ 11 + σ 31 = 1 + σ 1 or σ 1 + σ 31 = 1 + σ 11 and described the preimage π 1 (Σ I1,I, ). The description implies that π 1 (Σ I1,I, ) is a convex polyhedron. This seems to indicate that the whole set Σ I1,I, is also a convex polyhedron. As the description of π 1 (Σ I1,I, ) relies on a problem concerning the eigenvalues of the sum of two Hermitian matrices that has long been standing, the complete description of Σ I1,I, could be an even harder problem. We have also proved a higher-order generalizations of Theorem 1 (Theorem 4) and Corollary 3 (Theorem 5). Acknowledgments. The authors express their gratitude to the mathoverflow.net user with for his help in proving Lemma 9 [1]. Appendix A. Definition of Tr n. In our presentation we follow [, p. 30]. The set Tr n of triplets (I, J, K) of cardinality r can be described by induction on r as follows.

19 ON THE LARGEST MULTILINEAR SINGULAR VALUES OF HIGHER-ORDER TENSORS19 Let us write I = {i 1 < i < < i r } and likewise for J and K. Then for r = 1, (I, J, K) is in T n 1 if k 1 = i 1 + j 1 1. For r > 1, (I, J, K) is in T n r if i + j = k + i I j J k K and, for all 1 p r 1 and all (U, V, W ) T r p, i u + j v = k w + u U v V w W r(r + 1), p(p + 1). Thus, T n r is defined recursively in terms of T r 1,..., T r r 1. REFERENCES [1] [] R. Bhatia, Linear Algebra to Quantum Cohomology: The Story of Alfred Horn s Inequalities, Am. Math. Monthly, 108 (001), pp [3] A. Cichocki, D. Mandic, C. Caiafa, A.-H. Phan, G. Zhou, Q. Zhao, and L. De Lathauwer, Tensor decompositions for signal processing applications. From two-way to multiway component analysis, IEEE Signal Process. Mag., 3 (015), pp [4] L. De Lathauwer, B. De Moor, and J. Vandewalle, A Multilinear Singular Value Decomposition, SIAM J. Matrix Anal. Appl., 1 (000), pp [5] L. De Lathauwer and J. Vandewalle, Dimensionality reduction in higher-order signal processing and rank-(r 1, R,..., R n) reduction in multilinear algebra, Linear Algebra Appl., 391 (004), pp [6] D. Gerard and P. Hoff, A higher-order LQ decomposition for separable covariance models, Linear Algebra Appl., 505 (016), pp [7] W. Hackbusch, D. Kressner, and A. Uschmajew, Perturbation of higher-order singular values, INS Preprint No. 1616, (016). [8] W. Hackbusch and A. Uschmajew, On the interconnection between the higher-order singular values of real tensors, Numer. Math., 135(3) (017), pp [9] A. Horn, Eigenvalues of sums of Hermitian matrices, Pacific J. Math., 1 (196), pp [10] A. A. Klyachko, Stable bundles, representation theory and Hermitian operators, Selecta Mathematica, 4 (1998), pp [11] A. Knutson and T. Tao, The honeycomb model of GL n(c) tensor products I: Proof of the saturation conjecture, J. Amer. Math. Soc., 1 (1999), pp [1] A. Knutson and T. Tao, Honeycombs and sums of Hermitian matrices, Notices Amer. Math. Soc., 48 (001), pp [13] T. G. Kolda and B. W. Bader, Tensor decompositions and applications, SIAM Rev., 51 (009), pp [14] S. Krämer, The geometrical description of feasible singular values in the tensor train format, arxiv: [15] P. M. Kroonenberg, Applied Multiway Data Analysis, Wiley, Hoboken, 008. [16] A. Seigal, Gram determinants of real binary tensors, arxiv: [17] N. D. Sidiropoulos, L. De Lathauwer, X. Fu, K. Huang, E. E. Papalexakis, and C. Faloutsos, Tensor Decomposition for Signal Processing and Machine Learning, IEEE Transactions on Signal Processing, 65 (017), pp [18] L. R. Tucker, Some mathematical notes on three-mode factor analysis, Psychometrika, 31 (1966), pp