Lecture 2. Tensor Iterations, Symmetries, and Rank. Charles F. Van Loan

Transcription

1 Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Iterations, Symmetries, and Rank Charles F. Van Loan Cornell University CIME-EMS Summer School June 22-26, 2015 Cetraro, Italy Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 1 / 42

2 The Setting We looked at the nearest Rank-1 tensor problem in Lecture 1. For matrices, the nearest Rank-1 problem is an SVD problem. Is the nearest Rank-1 problem related to some tensor SVD? In Lectures 3 and 4 we will look at 4 different tensor SVD ideas. Structured SVDs, Schur Decompositions, and QR factorizations will be part of the scene. We set the stage for this in Lecture 2 by developing various power methods. It will be an occasion to refine what we know about tensor rank, tensor symmetry, and tensor unfoldings. Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 2 / 42

3 Rayleigh Quotient Ideas Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 3 / 42

4 Rayleigh Quotient Ideas: The Matrix Case The Variational Approach to Singular Values and Vectors The singular values and singular vectors of a general matrix A IR m n are stationary values and vectors for the multilinear form ψ A (x, y) = x T Ay = m n a ij x i y j i=1 j=1 subject to the constraints x = y = 1 Let us understand this connection and then push the same idea for tensors. Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 4 / 42

5 Rayleigh Quotient Ideas: The Matrix Case Gradient Calculations We seek unit vectors x and y that zero the gradient of ψ A (x, y) = ψ(x, y) λ 2 (x T x 1) µ 2 (y T y 1) Since ψ A (x, y) = m x i n a ij y j = ( n m ) y j a ij x i i=1 j=1 j=1 i=1 it follows that we want " Ay λx # " 0 # ψ A(x, y) = A T x µy = 0 Thus, λ = µ = x T Ay = ψ A(x, y). Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 5 / 42

6 Rayleigh Quotient Ideas: The Matrix Case Gradient Calculations ψ A (x, y) = [ Ay (x T Ay)x ] A T x (x T Ay)y = [ 0 ] 0 Singular Values and Vectors If U T AV = Σ = diag(σ i ) is the SVD of A and U = [ u 1 u m ] and V = [ v 1 v n ], then from AV = UΣ and A T U = V Σ T we have Av i = σ i u i A T u i = σ i v i Since σ i = ui T Av i, it follows that Av i (ui T Av i )u i = 0 A T u i (ui T Av i )v i = 0 Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 6 / 42

7 Rayleigh Quotient Ideas: The Matrix Case Gradient Calculations ψ A (x, y) = [ Ay (x T Ay)x ] A T x (x T Ay)y = [ 0 ] 0 Singular Values and Vector If U T AV = Σ = diag(σ i ) is the SVD of A and U = [ u 1 u m ] and V = [ v 1 v n ], then from AV = UΣ and A T U = V Σ T we have Av i = σ i u i A T u i = σ i v i Since σ i = ui T Av i, it follows that Av i (ui T Av i )u i = 0 A T u i (ui T Av i )v i = 0 Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 7 / 42

8 Rayleigh Quotient Ideas: The Tensor Case The Singular Values and Vectors of a Tensor: A Definition If A IR m n p, x IR m, y IR n, and z IR p, then the singular values and vectors of A are the stationary values and vectors of ψ A (x, y, z) = m n i=1 j=1 k=1 p a ijk x i y j z k subject to the constraints x = y = z = 1 Order-3 tensors are plenty good enough to show the main ideas. Generalizations to order d tensors are generally pretty obvious. Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 8 / 42

9 Rayleigh Quotient Ideas: The Tensor Case Take a look at ψ A (x, y, z) Some handy rearrangements o fψ A : m n p ψ A (x, y, z) = a ijk x i y j z k = i=1 j=1 k=1 = = m x i i=1 n p a ijk y j z k j=1 k=1 ( n m y j j=1 p z k i=1 k=1 m k=1 i=1 j=1 ) p a ijk x i z k n a ijk x i y j Before we go after the gradient, let s frame the double summations in linear algebra terms... Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 9 / 42

10 Rayleigh Quotient Ideas: The Tensor Case Suppose m = 4,n = 3, and p = 2 ψ A (x, y, z) = = m i=1 x i n j=1 k=1 p a ijk y j z k m [ ] x i ai11 a i21 a i31 a i12 a i22 a i32 i=1 = x T a 111 a 121 a 131 a 112 a 122 a 132 a 211 a 221 a 231 a 212 a 222 a 232 a 311 a 321 a 331 a 312 a 322 a 332 a 411 a 421 a 431 a 412 a 422 a 432 z 1 y 1 z 1 y 2 z 1 y 3 z 2 y 1 z 2 y 2 z 2 y 3 z y Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 10 / 42

11 Rayleigh Quotient Ideas: The Tensor Case Suppose m = 4,n = 3, and p = 2 ψ A (x, y, z) = m i=1 = y T x i n j=1 k=1 p a ijk y j z k a 111 a 121 a 131 a 112 a 122 a 132 a 211 a 221 a 231 a 212 a 222 a 232 a 311 a 321 a 331 a 312 a 322 a 332 a 411 a 421 a 431 a 412 a 422 a 432 z x The matrix you see here is the mode-1 unfolding of A IR Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 11 / 42

12 Rayleigh Quotient Ideas: The Tensor Case ψ A (x, y, z) in terms of the Mode-1 Unfolding m n p ψ A (x, y, z) = x i a ijk y j z k = x T A (1) z y A (1) = i=1 j=1 k=1 a 111 a 121 a 131 a 112 a 122 a 132 a 211 a 221 a 231 a 212 a 222 a 232 a 311 a 321 a 331 a 312 a 322 a 332 a 411 a 421 a 431 a 412 a 422 a 432 (1,1) (2,1) (3,1) (1,2) (2,2) (3,2) Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 12 / 42

13 Rayleigh Quotient Ideas: The Tensor Case ψ A (x, y, z) in terms of the Mode-2 Unfolding ( n m ) p ψ A (x, y, z) = y j a ijk x i z k = y T A (2) z x A (2) = j=1 i=1 k=1 a 111 a 211 a 311 a 411 a 112 a 212 a 312 a 412 a 121 a 221 a 321 a 421 a 122 a 222 a 322 a 422 a 131 a 231 a 331 a 431 a 132 a 232 a 332 a 432 (1,1) (2,1) (3,1) (4,1) (1,2) (2,2) (3,2) (4,2) Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 13 / 42

14 Rayleigh Quotient Ideas: The Tensor Case ψ A (x, y, z) in terms of the Mode-3 Unfolding p m n ψ A (x, y, z) = a ijk x i y j = z T A (3) y x k=1 z k i=1 j=1 A (3) = [ a111 a 211 a 311 a 411 a 121 a 221 a 321 a 421 a 131 a 231 a 331 a 431 a 112 a 212 a 312 a 412 a 122 a 222 a 322 a 422 a 132 a 232 a 332 a 432 (1,1) (2,1) (3,1) (4,1) (1,2) (2,2) (3,2) (4,2) (1,3) (2,3) (3,3) (4,3) ] Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 14 / 42

15 Rayleigh Quotient Ideas: The Tensor Case Important Skill: Framing a Tensor Computation in Matrix Terms ψ A (x, y, z) = m n p a ijk x i y j z k i=1 j=1 k=1 = x T A (1) (z y) = y T A (2) (z x) = z T A (3) (y x) With these characterizations we can readily compute the stationary vectors and values of this function subject to the constraint that x = y = z = 1. Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 15 / 42

16 Rayleigh Quotient Ideas: The Tensor Case The Gradient Computations Set the gradient of ψ A(x, y, z) = ψ A(x, y, z) λ 2 (x T x 1) µ 2 (y T y 1) τ 2 (z T z 1) to zero. Conclude that λ = µ = τ = ψ(x, y, z). If unit vectors x, y, and z to satisfy ψ A = A (1) (z y) ψ A(x, y, z)x A (2) (z x) ψ A(x, y, z)y A (3) (y x) ψ A(x, y, z)z = then σ = ψ(x, y, z) is a singular value of A and x, y, and z the associated singular vectors How can we solve this (highly structured) system of nonlinear equations? Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 16 / 42

17 A Higher-Order Power Method Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 17 / 42

18 The Power Method for Matrices The Gradient Equations Ay = σ x where σ = ψ A (x, y) = x T Ay. Iterate... A T x = σ y y a given unit vector Repeat Until Happy x = Ay, x x/ x ỹ = A T x, y ỹ/ ỹ σ = ψ A (x, y) Same as power method applied to A T A.In the limit, σuv T converges to the closest rank-1 to A. Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 18 / 42

19 The Higher-Order Power Method for Tensors The Gradient Equations A (1) (z y) = σ x A (2) (z x) = σ y A (3) (y x) = σ z where σ = ψ A(x, y, z) = x T A (1) (z y) = y T A (2) (z x) = z T A (3) (y x) Iterate... y and z given unit vectors Repeat Until Happy x = A (1) (z y), ỹ = A (2) (z x), z = A (3) (y x), σ = ψ A (x, y, z) x = x/ x y = ỹ/ ỹ z = z/ z Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 19 / 42

20 The Higher-Order Power Method for Tensors A Tempting Repetition Step 1. Compute closest σ 1 u 1 v 1 w 1 to A. Step 2. Compute closest σ 2 u 2 v 2 w 2 to A σ 1 u 1 v 1 w 1. Step 3. Compute closest σ 3 u 3 v 3 w 3 to A σ 1 u 1 v 1 w 1 σ 2 u 2 v 2 w 2. Step r. Compute closest σ r u r v r w r to r 1 A σ k u k v k w k. k=1 This does not render a best approximation to A as it does for matrices. So maybe we better look more closely at sums of rank-1 tensors and rank. Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 20 / 42

21 The Idea of Tensor Rank Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 21 / 42

22 A IR as a minimal sum of rank-1 tensors. Challenge: Find thinnest possible X, Y, Z IR 2 r so a 111 a 211 a 121 a 221 a 112 a 212 a 122 a 222 = r z k y k x k k=1 where X = [ x 1 x r ], Y = [ y 1 y r ], andz = [ z 1 z r ] are column partitionings. The minimizing r is the tensor rank. Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 22 / 42

23 A IR as a minimal sum of rank-1 tensors. A Surprising Fact a 111 a 211 a 121 If a 221 rank = 2 with prob 79% a 112 = randn(8,1), then a 212 rank = 3 with prob 21% a 122 a 222 This is very different from the matrix case where A = randn(n,n) implies rank(a) = n with probability 100%. A strong hint that tensor rank is decidely more complicated than matrix rank. What are the full rank 2-by-2-by-2 tensors? Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 23 / 42

24 The 79/21 Property Connection to a Generalized Eigenvalue Problem If the a ijk are randn, then ([ a111 a det 121 a 211 a 221 ] [ a112 a λ 122 a 212 a 222 ]) = 0 has real distinct roots with probability 79% and complex conjugate roots with probability 21%. The sum-of-rank-ones expansion for A involves the generalized eigenvectors of this problem. Yet another example of turning a tensor problem into a matrix problem. Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 24 / 42

25 Symmetry Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 25 / 42

26 What is a Symmetric Tensor? The Order-3 Definition: C IR n n n c ikj c ijk = c jik c jki c kij c kji It just means that permuting the indices does not change the value. Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 26 / 42

27 Symmetric Tensors C IR There are 10 values to specify... (1,1,1) (1,1,2),(1,2,1),(2,1,1) (1,1,3),(1,3,1),(3,1,1) (2,2,2) (2,2,1),(2,1,2),(2,2,1) (2,2,3),(2,3,1),(2,2,3) (3,3,3) (3,3,1),(3,1,3),(3,3,1) (3,3,2),(3,2,3),(3,3,2) (1,2,3),(1,3,2),(2,1,3),(2,3,1),(3,1,2),(3,2,1) Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 27 / 42

28 Symmetric Tensors The Modal Unfoldings are all the Same c 111 c 121 c 131 c 112 c 122 c 132 c 113 c 123 c 133 C (1) = c 211 c 221 c 231 c 212 c 222 c 232 c 213 c 223 c 233 c 311 c 321 c 331 c 312 c 322 c c 323 c 333 c 111 c 211 c 311 c 112 c 212 c 312 c 113 c 213 c 313 C (2) = c 121 c 221 c 321 c 122 c 222 c 322 c 123 c 223 c 323 c 131 c 231 c 331 c 132 c 232 c c 233 c 333 c 111 c 211 c 311 c 121 c 221 c 321 c 131 c 231 c 331 C (3) = c 112 c 212 c 312 c 122 c 222 c 322 c 132 c 232 c 332 c 113 c 213 c 313 c 123 c 223 c c 233 c 333 Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 28 / 42

29 Symmetric Tensors Rank-1 Symmetric Tensors If x IR n, then C = x x x is a symmetric rank-1 tensor. This is obvious since c ijk = x i x j x k. Note that vec(x x x) = x x x Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 29 / 42

30 Symmetric Tensors Symmetric Rank An order-3 symmetric tensor C has symmetric rank r if there exists x 1,..., x r IR n and σ IR r such that C = r σ k x k x k x k k=1 and no shorter sum of symmetric rank-1 tensors exists. Symmetric rank is denoted by rank S (C). Note, there may be a shorter sum so C = r k=1 σ k x k ỹ k z k Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 30 / 42

31 Symmetric Tensors: Interesting Aside about Rank Fact If C C n n is an order-d symmetric tensor, then with probability 1 f (d, m) + 1 if (d, n) = (3,5),(4,3),(4,4), or (4,5) rank S (C) = f (d, n) otherwise where ( n + d 1 d f (d, n) = ceil n ) Symmetric Tensor Rank is more tractable than General Tensor Rank. Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 31 / 42

32 Rayleigh Quotient Ideas Symmetric Matrix Eigenvalues If C is a symmeytric matrix, then the stationary values of φ C (x) = x T Cx subject to the constraint that x 2 = 1 are the eigenvalues of C. The associated stationary vectors are eigenvectors. Symmetric Tensor Eigenvalues If C is a symmetric tensor, then the stationary values of φ C (x) = n n n c ijk x i x j x k = x T C (1) (x x) i=1 j=1 k=1 subject to the constraint that x 2 = 1 are the eigenvalues of C. The associated stationary vectors are eigenvectors. Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 32 / 42

33 Rayleigh Quotients: Symmetric Tensor Case Symmetric Higher-Order Power Method Initialize unit vector x. Repeat Until Happy x = C (1) (x x) x = x/ x Sample Convergence Result If the order of C is even and M is a square unfolding, then the iteration converges if M is positive definite. Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 33 / 42

34 The SVD - SymEig Connection The Sym of a Matrix sym(a) = [ 0 A A T 0 The SVD of A Relates to the EVD of sym(a) ] IR (n 1+n 2 ) (n 1 +n 2 ) If A = U diag(σ i ) V T is the SVD of A IR n 1 n 2, then for k = 1:rank(A) [ 0 A A T 0 ] [ uk ±v k where u k = U(:, k) and v k = V (:, k). ] = ±σ k [ uk ±v k ] The above SVD-related power method was essentially traditional power method applied to finding the largest eigenvector of sym(a). Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 34 / 42

35 The SVD - SymEig Connection The Sym of a Matrix sym(a) = [ 0 A A T 0 The SVD of A Relates to the EVD of sym(a) ] IR (n 1+n 2 ) (n 1 +n 2 ) If A = U diag(σ i ) V T is the SVD of A IR n 1 n 2, then for k = 1:rank(A) [ 0 A A T 0 ] [ uk ±v k where u k = U(:, k) and v k = V (:, k). ] = ±σ k [ uk ±v k ] Let us look at the analog of this for tensors. Need transposition Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 35 / 42

36 Tensor Transposition: The Order-3 Case Six possibilities... If C IR n 1 n 2 n 3, then there are 6 = 3! possible transpositions identified by the notation C < [i j k] > where [i j k] is a permutation of [1 2 3]: B = C < [1 2 3] > C < [1 3 2] > C < [2 1 3] > C < [2 3 1] > C < [3 1 2] > C < [3 2 1] > for i = 1:n 1, j = 1:n 2, k = 1:n 3. = b ijk b ikj b jik b jki b kij b kji = c ijk Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 36 / 42

37 Symmetric Embedding of a Tensor C = sym(a) An Order-3 Example... C(:, :, 3) C(:, :, 2) C(:, :, 1) A <[123]> A <[132]> A <[213]> A <[231]> A <[312]> A <[321]> Note the careful placement of A s six transposes Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 37 / 42

38 Connecting A and sym(a) Algorithms Interesting connections between power methods with A and power methods with sym(a) Analysis If σ, u v z is a stationary pair for sym(a) then so are u σ, u v, σ, v, z z σ, u v z Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 38 / 42

39 Symmetric Tensors: Interesting Aside about Rank Interesting Possible Connection Easy: d! rank(a) rank S (sym(a)) Equality is hard or perhaps not true. But if it could be established, then we would have new insight into the tensor rank problem. Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 39 / 42

40 Optional Fun Problems Problem E2. Suppose A IR m n2 with m > n 2. Develop an alternating least squares solution framework for minimizing A(x y) b 2 where b IR n2 and x, y IR n. Problem A2. Same notation as E2. What is the gradient of φ(x, y) = 1 2 A(x y) b 2 2 Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 40 / 42

41 Closing Remarks Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 41 / 42

42 Bridging the Gap via Tensor Unfoldings A Common Framework for Tensor Computations Turn tensor A into a matrix A. 2. Through matrix computations, discover things about A. 3. Draw conclusions about tensor A based on what is learned about matrix A. Structured Matrix Computations from Structured Tensors Lecture 2. Tensor Symmetries and Rank 42 / 42