Lecture 4. Tensor-Related Singular Value Decompositions. Charles F. Van Loan

Transcription

1 From Matrix to Tensor: The Transition to Numerical Multilinear Algebra Lecture 4. Tensor-Related Singular Value Decompositions Charles F. Van Loan Cornell University The Gene Golub SIAM Summer School 2010 Selva di Fasano, Brindisi, Italy

2 Where We Are Lecture 1. Introduction to Tensor Computations Lecture 2. Tensor Unfoldings Lecture 3. Transpositions, Kronecker Products, Contractions Lecture 4. Tensor-Related Singular Value Decompositions Lecture 5. The CP Representation and Tensor Rank Lecture 6. The Tucker Representation Lecture 7. Other Decompositions and Nearness Problems Lecture 8. Multilinear Rayleigh Quotients Lecture 9. The Curse of Dimensionality Lecture 10. Special Topics

3 What is this Lecture About? Pushing the SVD Envelope The SVD is a powerful tool for exposing the structure of a matrix and for capturing its essence through optimal, data-sparse representations: A = rank(a) k=1 σ k u k v T k ˆr k=1 σ k u k v T k For a tensor A, let s try for something similar... A = whatever!

4 What is this Lecture About? The Higher Order Singular Value Decomposition (HOSVD) The HOSVD of a tensor A IR n 1 n d involves computing the matrix SVDs of its modal unfoldings A (1),..., A (d). This results in a representation of A as a sum of rank-1 tensors. Before we present the HOSVD, we need to understand the mode-k matrix product which can be thought of as a tensor-times-matrix product.

5 What is this Lecture About? The Kronecker Product Singular Value Decomposition (KPSVD) The KPSVD of a matrix A represents A as an SVD-like sum of Kronecker products. Applied to an unfolding of A, it results in a representation of A as a sum of low rank tensors. Before we present the KPSVD, we need to understand the nearest Kronecker Product problem.

6 The Mode-k Matrix Product Main Idea A mode-k matrix product is a special contraction that involves a matrix and a tensor. A new tensor of the same order is obtained by applying the matrix to each mode-k fiber of the tensor.

7 The Mode-k Matrix Product A mode-1 example when the Tensor A is Second Order... [ ] [ ] [ b11 b 12 b 13 m11 m = 12 a11 a 12 a 13 b 21 b 22 b 23 m 21 m 22 a 21 a 22 a 23 (The fibers of A are its columns.) ] A mode-2 example when the Tensor A is Second Order... b 11 b 12 m 11 m 12 m 13 b 21 b 22 b 31 b 32 = m 21 m 22 m 23 a 11 a 21 m 31 m 32 m 33 a 12 a 22 a b 41 b 42 m 41 m 42 m 13 a (The fibers of A are its rows.) The matrix doesn t have to be square.

8 The Mode-k Matrix Product A Mode-2 Example When A IR m 11 m 12 m 13 m 21 m 22 m 23 m 31 m 32 m 33 m 41 m 42 m 43 m 51 m 52 m 53 b 111 b 211 b 311 b 411 b 112 b 212 b 312 b 412 b 121 b 221 b 321 b 421 b 122 b 222 b 322 b 422 b 131 b 231 b 331 b 431 b 132 b 232 b 332 b 432 b 141 b 241 b 341 b 441 b 142 b 242 b 342 b 442 b 151 b 251 b 351 b 451 b 152 b 252 b 352 b 452 = a 111 a 211 a 311 a 411 a 112 a 212 a 312 a 412 a 121 a 221 a 321 a 421 a 122 a 222 a 322 a 422 a 131 a 231 a 331 a 431 a 132 a 232 a 332 a 432 Note that (1) B IR and (2) B (2) = M A (2).

9 The Mode-k Matrix Product In General... If A IR n 1 n d and M IR m k n k, then B(α 1,..., α k 1, i, α k+1,..., α d ) = n k M(i, j) A(α 1,..., α k 1, j, α k+1,..., α d ) j=1 is the mode-k matrix product of M and A. Multiply All the Mode-k Fibers by M... B (k) = M A (k)

10 The Mode-k Matrix Product Notation The mode-k matrix product of M and A is denoted by B = A k M Thus, if B = A k M, then B is defined by B (k) = M A (k). Order and Dimension If A IR n 1 n d, M IR m k n k, and B = A k M, then B = B(1:n 1,..., 1:n k 1, 1:m k, 1:n k+1,..., 1:n k ). Thus, B has the same order as A.

11 Matlab Tensor Toolbox: Mode-k Matrix Product Using ttm n = [ ]; A = tenrand (n); for k =1:4 M = randn (n(k),n(k )); % Compute the mode - k product of tensor A % and matrix M to obtain tensor B... B = ttm (A,M,k); end Problem 4.1. Write a function B = Myttm(A,M,k) that does the same thing as ttm. Make effective use of tenmat.

12 Problem 4.2. If A IR n 1 n d and v IR n k, then the mode-k vector product is defined by n k X j=1 B(α 1,..., α k 1, α k+1,..., α d ) = v(j) A(α 1,..., α k 1, j, α k+1,..., α d ) The Tensor Toolbox function ttv (for tensor-times-vector) can be used to compute this contraction. Note that B = B(1:n 1,..., 1:n k 1, 1:n k+1,..., 1:n k ). Thus, the order of B is one less than the order of A. Write a Matlab function B = Myttv(M,A,k) that carries out the mode-k vector product. Use the fact that B is a reshaping of v T A (k).

13 The Mode-k Matrix Product Property 1. (Reshaping as a Matrix-Vector Product) If A IR n 1 n d, M IR m k n k, and B = A k M then vec(b) = ( ) I nd I nk+1 M I nk 1 I n1 vec(a) = ( ) I nk+1 n d M I n1 n k 1 vec(a) Problem 4.3. Prove this. Hint. Show that if p = [1:k 1 k +1:d 1] then (A km) [ p ] = A [ p ] 1M.

14 The Mode-k Matrix Product Property 2. (Successive Products in the Same Mode) If A IR n 1 n d and M 1, M 2 IR m k n k, then (A k M 1 ) k M 2 = A k (M 1 M 2 ). Problem 4.4. Prove this using Property 1 and the Kronecker product fact that (A B)(C D) = (AC) (BD).

15 The Mode-k Matrix Product Property 3. (Successive Products in Different Modes) If A IR n 1 n d, M k IR m k n k, M j IR m j n j, and k j, then (A k M k ) j M j = (A j M j ) k M k Problem 4.5. Prove this using Property 1 and the Kronecker product fact that (A B)(C D) = (AC) (BD).

16 The Tucker Product Definition If A IR n 1 n d and M k IR m k n k for k = 1:d, then B = A 1 M 1 2 M 2 d M d is a Tucker product of A and M 1,..., M d.

17 Matlab Tensor Toolbox: Tucker Products function B = TuckerProd (A, M) % A is a n(1) -by n(d) tensor. % M is a length - d cell array with % M{k} an m(k)-by -n(k) matrix. % B is an m(1) -by by -m(d) tensor given by % B = A x1 M {1} x2 M {2}... xd M{d} % where " xk" denotes mode - k matrix product. B = A; for k =1: length (A. size ) B = ttm (B,M{k},k); end

18 The Tucker Product Representation Before... Given A IR n 1 n d and M k IR m k n k for k = 1:d, compute the Tucker product B = A 1 M 1 2 M 2 d M d Now... Given A IR n 1 n d, find matrices U k IR n k r k tensor S IR r 1 r d such that for k = 1:d and A = S 1 U 1 2 U 2 d U d is an illuminating Tucker product representation of A.

19 The Tucker Product Representation Multilinear Sum Formulation Suppose A IR n 1 n d and U k IR n k r k. If where S IR r 1 r d, then A = S 1 U 1 2 U 2 d U d A(i) = r S(j)U 1 (i 1, j 1 ) U d (i d, j d ) j=1

20 The Tucker Product Representation Matrix-Vector Product Formulation Suppose A IR n 1 n d and U k IR n k r k. If where S IR r 1 r d, then A = S 1 U 1 2 U 2 d U d vec(a) = (U d U 1 ) vec(s)

21 The Tucker Product Representation Important Existence Situation If A IR n 1 n d and U k IR n k n k is nonsingular for k = 1:d, then A = S 1 U 1 2 U 2 d U d with S = A 1 U U 1 2 d U 1 d. We will refer to the U k as the inverse factors and S as the core tensor. Proof. S 1 U 1 2 U 2 3 U 3 = A 1 U U U U 1 2 U 2 3 U 3 = A 1 (U 1 1 U 1) 2 (U 1 2 U 2) 3 (U 1 3 U 3) = A

22 The Tucker Product Representation Core Tensor Unfoldings Suppose A IR n 1 n d and U k IR n k n k is nonsingular for k = 1:d. If A = S 1 U 1 2 U 2 d U d then the mode-k unfoldings of S satisfy S (k) = U 1 k A (k) (U d U k+1 U k 1 U 1 )

23 The Higher-Order SVD (HOSVD) Definition Suppose A IR n 1 n d and that its mode-k unfolding has SVD A (k) = U k Σ k V T k for k = 1:d. Its higher-order SVD is given by A = S 1 U 1 2 U 2 d U d where S = A 1 U T 1 2 U T 2 d U T d A Tucker product representation where the inverse factor matrices are the left singular vector matrices for the unfoldings A (1),..., A (d).

24 Matlab Tensor Toolbox: Computing the HOSVD function [S, U] = HOSVD ( A) % A is an n(1) -by by -n(d) tensor. % U is a length - d cell array with the % property that U{ k} is the left singular % vector matrix of A s mode - k unfolding. % S is an n(1) -by by -n(d) tensor given by % A x1 U {1} x2 U {2}... xd U{d} S = A; for k =1: length (A. size ) C = tenmat (A,k); [U{k}, Sigma,V] = svd (C. data ); S = ttm (S,U{k},k); end

25 The Higher-Order SVD (HOSVD) The HOSVD of a Matrix If d = 2 then A is a matrix and the HOSVD is the SVD. Indeed, if A = A (1) = U 1 Σ 1 V1 T A T = A (2) = U 2 Σ 2 V2 T then we can set U = U 1 = V 2 and V = U 2 = V 1. Note that S = (A 1 U1 T ) 2 U2 T = (U1 T A) 2 U 2 = U1 T AV 1 = Σ 1.

26 The Higher-Order SVD (HOSVD) The Modal Unfoldings of the Core Tensor S If A = S 1 U 1 2 U 2 d U d is the HOSVD of A IR n 1 n d, then for k = 1:d S (k) = U T k A (k) (U d U k+1 U k 1 U 1 ) The Core Tensor S Encodes Singular Value Information Since Uk T A (k)v k = Σ k is the SVD of A (k), we have S (k) = Σ k Vk T (U d U k+1 U k 1 U 1). It follows that the rows of S (k) are mutually orthogonal and that the singular values of A (k) are the 2-norms of these rows.

27 The Higher-Order SVD (HOSVD) The HOSVD as a Multilinear Sum If A = S 1 U 1 2 U 2 d U d is the HOSVD of A IR n 1 n d, then n A(i) = S(j)U 1 (i 1, j 1 ) U d (i d, j d ) j=1 The HOSVD as a Matrix-Vector Product If A = S 1 U 1 2 U 2 d U d is the HOSVD of A IR n 1 n d, then vec(a) = (U d U 1 ) vec(s) Note that U d U 1 is orthogonal.

28 Problem 4.6. Suppose A = S 1 U 1 2 U 2 d U d is the HOSVD of A IR n 1 n d and that r min{n 1,..., n d }. Through Matlab experimentation, what can you say about the approximation A S r 1 U 1(:, 1:r) 2 U 2(:, 1:r) d U d (:, 1:r) where S r = S(1:r, 1:r,..., 1:r)? Problem 4.7. Formulate an HOQRP factorization for a tensor A IR n 1 n d that is based on the QR-with-column-pivoting factorizations A (k) P k = Q k R k for k = 1:d. Does the core tensor have any special properties?

29 The Nearest Kronecker Product Problem The Problem Given the block matrix A 11 A 1,c2 A =..... A i2,j 2 IR r 1 c 1 A r2,1 A r2,c 2 find B IR r 2 c 2 and C IR r 1 c 1 so that is minimized. φ A (B, C) = A B C F Looks like the nearest rank-1 matrix problem which has an SVD solution.

30 The Nearest Kronecker Product Problem The Tensor Connection The block matrix A 11 A 1,c2 A =..... A i2,j 2 IR r 1 c 1 A r2,1 A r2,c 2 corresponds to an order-4 tensor A i2,j 2 (i 1, j 1 ) A(i 1, j 1, i 2, j 2 ) and so does M = (M ij ) = B C: M i2,j 2 (i 1, j 1 ) B(i 1, j 1 )C(i 2, j 2 )

31 The Nearest Kronecker Product Problem The Objective Function in Tensor Terms φ A (B, C) = A B C F r 1 c 1 r 2 c 2 = A(i 1, j 1, i 2, j 2 ) B(i 2, j 2 )C(i 1, j 1 ) i 1=1 j 1=1 i 2=1 j 2=1 We are trying to approximate an order-4 tensor with a pair of order-2 tensors. Analogous to approximating a matrix (an order-2 tensor) with a rank-1 matrix (a pair of order-1 tensors.)

32 The Nearest Kronecker Product Problem Reshaping the Objective Function (3-by-2 case) a 11 a 12 a 13 a 14 a 21 a 22 a 23 a 24 a 31 a 32 a 33 a 34 a 41 a 42 a 43 a 44 a 51 a 52 a 53 a 54 a 61 a 62 a 63 a 64 a 11 a 21 a 12 a 22 a 31 a 41 a 32 a 42 a 51 a 61 a 52 a 62 a 13 a 23 a 14 a 24 a 33 a 43 a 34 a 44 a 53 a 63 a 54 a 64 b 11 b 12 b 21 b 22 b 31 b 32 = b 11 b 21 b 31 b 12 b 22 b 32 [ c11 c 12 c 21 c 22 ] F [ c11 c 21 c 12 c 22 ] F

33 The Nearest Kronecker Product Problem Minimizing the Objective Function (3-by-2 case) It is a nearest rank-1 problem, φ A (B, C) = a 11 a 21 a 12 a 22 a 31 a 41 a 32 a 42 a 51 a 61 a 52 a 62 a 13 a 23 a 14 a 24 a 33 a 43 a 34 a 44 a 53 a 63 a 54 a 64 b 11 b 21 b 31 b 12 b 22 b 32 [ ] c11 c 21 c 12 c 22 F = Ã vec(b)vec(c)t F with SVD solution: Ã = UΣV T vec(b) = σ 1 U(:, 1) vec(c) = σ 1 V (:, 1)

34 The Nearest Kronecker Product Problem The Tilde Matrix a 11 a 12 a 13 a 14 A = a 21 a 22 a 23 a 24 a 31 a 32 a 33 a 34 a 41 a 42 a 43 a 44 = a 51 a 52 a 53 a 54 a 61 a 62 a 63 a 64 implies à = a 11 a 21 a 12 a 22 a 31 a 41 a 32 a 42 a 51 a 61 a 52 a 62 a 13 a 23 a 14 a 24 a 33 a 43 a 34 a 44 a 53 a 63 a 54 a 64 = A 11 A 12 A 21 A 22 A 31 A 32 vec(a 11 ) T vec(a 21 ) T vec(a 31 ) T vec(a 12 ) T vec(a 22 ) T vec(a 32 ) T.

35 The Nearest Kronecker Product Problem Solution via Lanczos SVD Since only the big singular value triple (σ 1, u 1, v 1 ) associated with à s SVD is required, we can use Lanczos SVD. Note that if A is sparse, then à is sparse.

36 Problem 4.8. Write a Matlab function [B,C] = NearestKP(A,r1,c1,r2,c2) that solves the nearest Kronecker product problem where A IR m n with m = r 1r 2, n = c 1c 2 and A is regarded as an r 2-by-c 2 block matrix with r 1-by-c 1 blocks. Make effective use of Matlab s sparse SVD solver svds. Problem 4.9. Write a Matlab function [B,C] = NearestTensor(A) where A is an order-4 tensor, B is an order-2 tensor, and C is an order-2 tensor such that s X φ A(B, C) = A(i) B(i 3, i 4)C(i 1, i 2) 2 is minimized. Use tenmat to set up à and use svd. i

37 The Kronecker Product SVD (KPSVD) Theorem If A = A 11 A 1,c A r2,1 A r2,c 2 A i2,j 2 IR r 1 c 1 then there exist U 1,..., U rkp IR r 2 c 2, V 1,..., V rkp IR r 1 c 1, and scalars σ 1 σ rkp > 0 such that A = r KP σ k U k V k. k=1 The sets {vec(u k )} and {vec(v k )} are orthonormal and r KP is the Kronecker rank of A with respect to the chosen blocking.

38 The Kronecker Product SVD (KPSVD) Constructive Proof Compute the SVD of Ã: Ã = UΣV T = r KP σ k u k vk T k=1 and define the U k and V k by for k = 1:r KP. vec(u k ) = u k vec(v k ) = v k U k = reshape(u k, r 2, c 2 ), V k = reshape(v k, r 1, c 1 )

39 Problem Suppose H =» F11 G 11 F 12 G 12 F 21 G 21 F 22 G 22 where F ij IR m m and G ij IR m m. Regarded as a 2m-by-2m block matrix, what is the Kronecker rank of H. Problem Suppose H = F 1. F p ˆ G 1 G q where the F i and G i are all m-by-m. Regarded as a p-by-q block matrix, what is the Kronecker rank of H?

40 The Kronecker Product SVD (KPSVD) Nearest rank-r If r r KP, then A r = r σ k U k V k k=1 is the nearest matrix to A (in the Frobenius norm) that has Kronecker rank r.

41 The Tensor Outer Product Operation Applied to Order-1 Tensors... A = b c b IR n 1, c IR n 2 means A(i 1, i 2 ) = b(i 1 )c(i 2 ) Note that tenmat(a, [1], [2]) = bc T

42 The Tensor Outer Product Operation Applied to Order-2 Tensors... A = B C B IR n 1 r 1, C IR n 2 r 2 means A(i 1, i 2, i 3, i 4 ) = B(i 1, i 2 )C(i 3, i 4 ) Note that tenmat(a, [ 3 1 ], [ 4 2 ]) = B C

43 The Tensor Outer Product Operation More General... A = B C D B IR n 1 r 1, C IR n 2 r 2, D IR n 3 r 3 means A(i 1, i 2, i 3, i 4 ) = B(i 1, i 2 )C(i 3, i 4 )D(i 5, i 6 ) Note that tenmat(a, [ ], [ ]) = B C D

44 The Tensor Outer Product Operation The HOSVD as a Sum of Rank-1 Tensors If A = S 1 U 1 2 U 2 d U d is the HOSVD of A IR n 1 n d, then n A(i) = S(j)U 1 (i 1, j 1 ) U d (i d, j d ) j=1 implies n A = S(j) U 1 (:, j 1 ) U d (:, j d ) j=1

45 The Tensor Outer Product Operation The KPSVD... If tenmat(a,[ 3 1 ], [ 4 2 ]) = i σ i B i C i then A = i σ i B i C i

46 Summary of Lecture 4. Key Words The Mode-k Matrix Product is a contraction between a tensor and a matrix that produces another tensor. The Higher Order SVD of a tensor A assembles the SVDs of A s modal unfoldings. The Nearest Kronecker product problem involves computing the first singular triple of a permuted version of the given matrix. The Kronecker Product SVD characterizes a block matrix as a sum of Kronecker products. By applying it to an unfolding of a tensor A, an outer product expansion for A is obtained. The outer product operation is a way of combining an order-d 1 tensor and an order-d 2 tensor to obtain an order-(d 1 + d 2 ) tensor.