1. Connecting to Matrix Computations. Charles F. Van Loan

Transcription

1 Four Talks on Tensor Computations 1. Connecting to Matrix Computations Charles F. Van Loan Cornell University SCAN Seminar October 27, 2014 Four Talks on Tensor Computations 1. Connecting to Matrix Computations 1 / 56

2 Its About This Four Talks on Tensor Computations 1. Connecting to Matrix Computations 2 / 56

3 Overview Preparation for the Next Big Thing... Scalar-Level Thinking 1960 s Matrix-Level Thinking 1980 s The factorization paradigm: LU, LDL T, QR, UΣV T, etc. Cache utilization, parallel computing, LAPACK, etc. Block Matrix-Level Thinking 2000 s Tensor-Level Thinking New applications, factorizations, data structures, nonlinear analysis, optimization strategies, etc. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 3 / 56

4 The Matrix Factorizations Paradigm A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T A = ULV It s T PAQ T = LUa A = Language UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR A = GG T PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR PAP T = LDL T Q T AQ = D X 1 AX = J U T AU = T AP = QR A = ULV T PAQ T = LU A = UΣV T PA = LU A = QR Four Talks on Tensor Computations 1. Connecting to Matrix Computations 4 / 56

5 Some Ways They Are Used Conversion of a hard matrix problem into an easier one. Ax = b Ly = Pb, Ux = y Low-rank approximation. (10 7 -by-10 5 ) (10 7 -by-10) (10-by-10 5 ) Reasoning about Data Re-Use via Block Factorizations A ij A ij A ik A kj not a ij a ij a ik a kj Discovery and Insight min A 1 (A 2 + xy T )Q F Q,x,y 10 4 Four Talks on Tensor Computations 1. Connecting to Matrix Computations 5 / 56

6 Tensor Factorizations and Decompositions The same story is playing out with tensors: = σ 1 w 1 v 1 u 1 + σ 2 w 2 v 2 u Four Talks on Tensor Computations 1. Connecting to Matrix Computations 6 / 56

7 The Big Data Theme A Changing Definition of Big In Matrix Computations, to say that A IR n 1 n 2 that both n 1 and n 2 are big. is big is to say In Tensor Computations, to say that A IR n 1 n d is big is to say that n 1 n 2 n d is big and this need not require big n k. E.g. n 1 = n 2 = = n 1000 = 2. Algorithms that scale with d will induce a transition... Matrix-Based Scientific Computation Tensor-Based Scientific Computation Four Talks on Tensor Computations 1. Connecting to Matrix Computations 7 / 56

8 What is a Tensor? Definition An order-d tensor A IR n 1 n d is a real d-dimensional array A(1:n 1,..., 1:n d ) where the index range in the k-th mode is from 1 to n k. A tensor A IR n 1 n d is cubical if n 1 = = n d. Low-Order Tensors A scalar is an order-0 tensor. A vector is an order-1 tensor. A matrix is an order-2 tensor. We use calligraphic font to designate tensors that have order 3 or greater e.g., A, B, C, etc. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 8 / 56

9 Where Might They Come From? Discretization A(i, j, k, l) might house the value of f (w, x, y, z) at (w, x, y, z) = (w i, x j, y k, z l ). Multiway Analysis A(i, j, k, l) is a value that captures an interaction between four variables/factors. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 9 / 56

10 You Have Seen Them Before Images A color picture is an m-by-n-by-3 tensor: A(:, :, 1) = red pixel values A(:, :, 2) = green pixel values A(:, :, 3) = blue pixel values Obvious extension of the colon notation. More later. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 10 / 56

11 You Have Seen them Before Block Matrices a 11 a 12 a 13 a 14 a 15 a 16 a 21 a 22 a 23 a 24 a 25 a 26 a 31 a 32 a 33 a 34 a 35 a 36 A = a 41 a 42 a 43 a 44 a 45 a 46 a 51 a 52 a 53 a 54 a 55 a 56 a 61 a 62 a 63 a 64 a 65 a 66 Matrix entry a 45 is the (2,1) entry of the (2,3) block: a 45 A(2, 3, 2, 1) Four Talks on Tensor Computations 1. Connecting to Matrix Computations 11 / 56

12 You Have Seen them Before Kronecker Products The m 1 m 2 m 3 -by-n 1 n 2 n 3 matrix A = A 1 (1:m 1, 1:n 1 ) A 2 (1:m 2, 1:n 2 ) A 3 (1:m 3, 1:n 3 ) is a reshaping of the m 1 -by-m 2 -by-m 3 -by-n 1 -by-n 2 -by-n 3 tensor A(i 1, i 2, i 3, j 1, j 2, j 3 ) = A 1 (i 1, j 1 )A 2 (i 2, j 2 )A 3 (i 3, j 3 ) Four Talks on Tensor Computations 1. Connecting to Matrix Computations 12 / 56

13 The Kronecker Product Definition B C is a block matrix whose ij-th block is b ij C. An Example... [ b11 b 12 ] [ b11 C b 12 C ] b 21 b 22 C = b 21 C b 22 C Kronecker products have a replicated block structure. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 13 / 56

14 The Kronecker Product There are three ways to regard A = B C... If then b 11 b 1n A =..... b m1 b mn c 11 c 1q..... c p1 c pq (i). A is an m-by-n block matrix with p-by-q blocks. (ii). A is an mp-by-nq matrix of scalars. (iii). A is an unfolded 4-th order tensor A IR p q m n : A(i 1, i 2, i 3, i 4 ) = B(i 3, i 4 )C(i 1, i 2 ) Four Talks on Tensor Computations 1. Connecting to Matrix Computations 14 / 56

15 Kronecker Product: All Possible Entry-Entry Products 4 9 = 36 [ b11 b 12 b 21 b 22 ] = c 11 c 12 c 13 c 21 c 22 c 23 c 31 c 32 c 33 b 11 c 11 b 11 c 12 b 11 c 13 b 12 c 11 b 12 c 12 b 12 c 13 b 11 c 21 b 11 c 22 b 11 c 23 b 12 c 21 b 12 c 22 b 12 c 23 b 11 c 31 b 11 c 32 b 11 c 33 b 12 c 31 b 12 c 32 b 12 c 33 b 21 c 11 b 21 c 12 b 21 c 13 b 22 c 11 b 22 c 12 b 22 c 13 b 21 c 21 b 21 c 22 b 21 c 23 b 22 c 21 b 22 c 22 b 22 c 23 b 21 c 31 b 21 c 32 b 21 c 33 b 22 c 31 b 22 c 32 b 22 c 33 Four Talks on Tensor Computations 1. Connecting to Matrix Computations 15 / 56

16 Kronecker Products of Kronecker Products Hierarchical [ b11 b 12 b 21 b 22 ] B C D = c 11 c 12 c 13 c 14 c 21 c 22 c 23 c 24 c 31 c 32 c 33 c 34 c 41 c 42 c 43 c 44 d 11 d 12 d 13 d 21 d 22 d 23 d 31 d 32 d 33 A 2-by-2 block matrix whose entries are 4-by-4 block matrices whose entries are 3-by-3 matrices. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 16 / 56

17 Bridging the Gap to Matrix Computations Tensor computations are typically disguised matrix computations and that is because of The Kronecker Product A = A 1 A 2 A 3 an order 6 tensor Tensor Unfoldings Rubik Cube 3 9 matrix Alternating Least Squares Multilinear optimization via component-wise optimization Four Talks on Tensor Computations 1. Connecting to Matrix Computations 17 / 56

18 The Decomposition Paradigm in Matrix Computations Typical... Convert the given problem into an equivalent easy-to-solve problem by using the right matrix decomposition. PA = LU, Ly = Pb, Ux = y = Ax = b Also Typical... Uncover hidden relationships by computing the right decomposition of the data matrix. A = UΣV T = A r i=1 σ i u i v T i Four Talks on Tensor Computations 1. Connecting to Matrix Computations 18 / 56

19 Two Natural Questions Question 1 Can we solve tensor problems by converting them to equivalent, easy-to-solve problems using a tensor decomposition? Question 2 Can we uncover hidden patterns in tensor data by computing an appropriate tensor decomposition? Let s explore the decomposition issue for 2-by-2-by-2 tensors... Four Talks on Tensor Computations 1. Connecting to Matrix Computations 19 / 56

20 The Singular Value Decomposition The 2-by-2 case... [ a11 a 12 a 21 a 22 ] = [ u11 u 12 u 21 u 22 ] [ σ1 0 0 σ 2 ] [ v11 v 12 v 21 v 22 ] T = σ 1 [ u11 u 21 ] [ v11 v 21 ] T + σ 2 [ u12 u 22 ] [ v12 v 22 ] T A reshaped presentation of the same thing... a 11 v 11 u 11 a 21 a 12 = σ v 11 u 21 1 v 21 u 11 + σ 2 a 22 v 21 u 21 v 12 u 12 v 12 u 22 v 22 u 12 v 22 u 22 The reshaped rank-1 matrices have special structure... Four Talks on Tensor Computations 1. Connecting to Matrix Computations 20 / 56

21 The Kronecker Product (KP) The KP of a Pair of 2-vectors [ x1 x 2 ] [ y1 y 2 ] = x 1 y 1 x 1 y 2 x 2 y 1 x 2 y 2 2-by-2 SVD in KP Terms... a 11 a 21 a 12 a 22 = σ 1 [ v11 v 21 ] [ u11 u 21 ] + σ 2 [ v12 v 22 ] [ u12 u 22 ] Four Talks on Tensor Computations 1. Connecting to Matrix Computations 21 / 56

22 Reshaping Appreciate the Duality.. [ a11 a 12 ] a 21 a 22 = σ 1 u 1 v1 T + σ 2 u 2 v2 T a 11 a 21 a 12 = σ 1 (v 1 u 1 ) + σ 2 (v 2 u 1 ) a 22 U = [ u 1 u 2 ] V = [ v 1 v 2 ] Four Talks on Tensor Computations 1. Connecting to Matrix Computations 22 / 56

23 A IR as a minimal sum of rank-1 tensors. What is a rank-1 tensor? If R IR has unit rank, then there exist f, g, h IR 2 such that r 111 r 211 r 121 r 221 r 112 r 212 r 122 r 222 = [ h1 h 2 ] This means that if R = (r ijk ), then [ g1 g 2 ] [ f1 r ijk = h k g j f i i = 1:2, j = 1:2, k = 1:2 f 2 ] Four Talks on Tensor Computations 1. Connecting to Matrix Computations 23 / 56

24 A IR as a minimal sum of rank-1 tensors. Challenge: Find thinnest possible X, Y, Z IR 2 r so a 111 a 211 a 121 a 221 a 112 a 212 a 122 a 222 = r z k y k x k k=1 where X = [ x 1 x r ], Y = [ y 1 y r ], andz = [ z 1 z r ] are column partitionings. The minimizing r is the tensor rank. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 24 / 56

25 A IR as a minimal sum of rank-1 tensors. A Surprising Fact a 111 a 211 a 121 If a 221 rank = 2 with prob 79% a 112 = randn(8,1), then a 212 rank = 3 with prob 21% a 122 a 222 This is Different from the Matrix Case... If A = randn(n,n), then rank(a) = n with prob 100%. What are the full rank 2-by-2-by-2 tensors? Four Talks on Tensor Computations 1. Connecting to Matrix Computations 25 / 56

26 The 79/21 Property The 2-by-2 Generalized Eigenvalue Problem (GEV) The generalized eigenvalues of F λg are the zeros of ([ ] [ ]) f11 f det 12 g11 g λ 12 = 0 f 21 f 22 g 21 g 22 Coincidence? If the a ijk are randn, then ([ a111 a det 121 a 211 a 221 ] [ a112 a λ 122 a 212 a 222 ]) = 0 has real distinct roots with probability 79% and complex conjugate roots with probability 21%. What is the connection between the 2-by-2 GEV and the 2-by-2-by-2 tensor rank problem? Four Talks on Tensor Computations 1. Connecting to Matrix Computations 26 / 56

27 Nearness Problems The Nearest Rank-1 Matrix Problem If A IR n 1 n 2 has SVD A = UΣV T then B opt = σ 1 u 1 v T 1 minimizes φ(b) = A B F rank(b) = 1. The Nearest Rank-1 Tensor Problem for A IR Find unit 2-norm vectors u, v, w IR 2 and scalar σ so that a σ w v u 2 is minimized where a 111 a 211 a 121 a = a 221 a 112 a 212 a 122 a 222 Four Talks on Tensor Computations 1. Connecting to Matrix Computations 27 / 56

28 A Highly Structured Nonlinear Optimization Problem It Depends on Four Parameters... φ(σ, θ 1, θ 2, θ 3 ) = = [ a σ cos(θ3 ) sin(θ 3 ) a 111 a 211 a 121 a 221 a 112 σ a 212 a 122 a 222 ] [ cos(θ2 ) sin(θ 2 ) c 3 c 2 c 1 c 3 c 2 s 1 c 3 s 2 c 1 c 3 s 2 s 1 s 3 c 2 c 1 s 3 c 2 s 1 s 3 s 2 c 1 s 3 s 2 s 1 2 ] [ cos(θ1 ) sin(θ 1 ) ] 2 c i = cos(θ i ), s i = sin(θ i ) i = 1:3 Four Talks on Tensor Computations 1. Connecting to Matrix Computations 28 / 56

29 A Highly Structured Nonlinear Optimization Problem And It Can Be Reshaped a 111 a 211 a 121 φ = a 221 a a a a σ 6 4 c 3c 2c 1 c 3c 2s 1 c 3s 2c 1 c 3s 2s 1 s 3c 2c 1 s 3c 2s 1 s 3s 2c 1 s 3s 2s = 2 3 a 111 a 211 a 121 a 221 a a a a c 3c c 3c 2 c 3s c 3s 2 s 3c s 3c s 3s s 3s 2» x1 y 1 2 [ x1 y 1 ] [ c1 = σ s 1 ] [ cos(θ1 ) = σ sin(θ 1 ) ] Idea: Improve σ and θ 1 by minimizing with respect to x 1 and y 1, holding θ 2 and θ 3 fixed. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 29 / 56

30 A Highly Structured Nonlinear Optimization Problem And It Can Be Reshaped a 111 a 211 a 121 φ = a 221 a a a a σ 6 4 c 3c 2c 1 c 3c 2s 1 c 3s 2c 1 c 3s 2s 1 s 3c 2c 1 s 3c 2s 1 s 3s 2c 1 s 3s 2s = 2 3 a 111 a 211 a 121 a 221 a a a a c 3c 1 0 c 3s c 3c 1 0 c 3s 1 s 3c s 3s s 3c s 3s 1» x2 y 2 2 [ x2 y 2 ] [ c2 = σ s 2 ] [ cos(θ2 ) = σ sin(θ 2 ) ] Idea: Improve σ and θ 2 by minimizing with respect to x 2 and y 2, holding θ 1 and θ 3 fixed. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 30 / 56

31 A Highly Structured Nonlinear Optimization Problem And It Can Be Reshaped a 111 a 211 a 121 φ = a 221 a a a a σ 6 4 c 3c 2c 1 c 3c 2s 1 c 3s 2c 1 c 3s 2s 1 s 3c 2c 1 s 3c 2s 1 s 3s 2c 1 s 3s 2s = 2 3 a 111 a 211 a 121 a 221 a a a a c 2c 1 0 c 2s 1 0 s 2c 1 0 s 2s c 2s c 2s s 2c s 2s 1» x3 y 3 2 [ x3 y 3 ] [ c3 = σ s 3 ] [ cos(θ3 ) = σ sin(θ 3 ) ] Idea: Improve σ and θ 3 by minimizing with respect to x 3 and y 3, holding θ 1 and θ 2 fixed. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 31 / 56

32 Componentwise Optimization A Common Framework for Tensor-Related Optimization Choose a subset of the unknowns such that if they are (temporarily) fixed, then we are presented with some standard matrix problem in the remaining unknowns. By choosing different subsets, cycle through all the unknowns. Repeat until converged. In tensor computations, the standard matrix problem that we end up solving is usually the linear least squares problem. In that case, the overall solution process is referred to as alternating least squares. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 32 / 56

33 Bridging the Gap via Tensor Unfoldings A Common Framework for Tensor Computations Turn tensor A into a matrix A. 2. Through matrix computations, discover things about A. 3. Draw conclusions about tensor A based on what is learned about matrix A. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 33 / 56

34 Tensor Unfoldings Some Tensor Unfoldings of A IR A (1) = A (2) = A (3) = a 111 a 121 a 131 a 112 a 122 a 132 a 211 a 221 a 231 a 212 a 222 a 232 a 311 a 321 a 331 a 312 a 322 a 332 a 411 a 421 a 431 a 412 a 422 a a 111 a 211 a 311 a 411 a 112 a 212 a 312 a a 121 a 221 a 321 a 421 a 122 a 222 a 322 a a 131 a 231 a 331 a 431 a 132 a 232 a 332 a 432» a111 a 211 a 311 a 411 a 121 a 221 a 321 a 421 a 131 a 231 a 331 a 431 a 112 a 212 a 312 a 412 a 122 a 222 a 322 a 422 a 132 a 232 a 332 a A tensor unfolding is sometimes referred to as tensor flattening or as a tensor matricization. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 34 / 56

35 Tensor Unfoldings: How They Might Show Up Gradients of Multilinear Forms If f :IR n 1 IR n 2 IR n 3 IR is defined by f (u, v, w) = n 1 n 2 n 3 A(i 1, i 2, i 3 )u(i 1 )v(i 2 )w(i 3 ) i 1=1 i 2=1 i 3=1 and A IR n 1 n 2 n 3, then its gradient is given by A (1) w v f (u, v, w) = A (2) w u A (3) v u where A (1), A (2), and A (3) are the modal unfoldings of A. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 35 / 56

36 More General Unfoldings Example: A = A(1:2, 1:3, 1:2, 1:2, 1:3) B = (1,1) (2,1) (1,2) (2,2) (1,3) (2,3) a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a (1,1,1) (2,1,1) (1,2,1) (2,2,1) (1,3,1) (2,3,1) (1,1,2) (2,1,2) (1,2,2) (2,2,2) (1,3,2) (2,3,2) Four Talks on Tensor Computations 1. Connecting to Matrix Computations 36 / 56

37 A Block Matrix is an Unfolded Order-4 Tensor Some Natural Identifications A block matrix with uniformly sized blocks A 11 A 1,c2 A =..... A i2,j 2 IR r 1 c 1 A r2,1 A r2,c 2 has several natural reshapings: A i1,j 1 (i 2, j 2 ) A(i 1, j 1, i 2, j 2 ) A IR r 1 r 2 c 1 c 2 A i2,j 2 (i 1, j 1 ) A(i 1, j 1, i 2, j 2 ) A IR r 1 c 1 r 2 c 2 Four Talks on Tensor Computations 1. Connecting to Matrix Computations 37 / 56

38 Kronecker Products are Unfolded Tensors The m 1 m 2 m 3 -by-n 1 n 2 n 3 matrix A = A 1 (1:m 1, 1:n 1 ) A 2 (1:m 2, 1:n 2 ) A 3 (1:m 3, 1:n 3 ) is a particular unfolding of the m 1 -by-m 2 -by-m 3 -by-n 1 -by-n 2 -by-n 3 tensor A(i 1, i 2, i 3, j 1, j 2, j 3 ) = A 1 (i 1, j 1 )A 2 (i 2, j 2 )A 3 (i 3, j 3 ) Four Talks on Tensor Computations 1. Connecting to Matrix Computations 38 / 56

39 Parts of a Tensor Fibers A fiber of a tensor A is a vector obtained by fixing all but one A s indices. For example, if A = A(1:3, 1:5, 1:4, 1:7), then A(2, :, 4, 6) = A(2, 1:5, 4, 6) = A(2, 1, 4, 6) A(2, 2, 4, 6) A(2, 3, 4, 6) A(2, 4, 4, 6) A(2, 5, 4, 6) is a fiber. We adopt the convention that a fiber is a column-vector. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 39 / 56

40 The vec Operation Turns matrices into vectors by stacking columns... X = vec(x ) = Four Talks on Tensor Computations 1. Connecting to Matrix Computations 40 / 56

41 The vec Operation Turns tensors into vectors by stacking mode-1 fibers... A(:, 1, 1) A(:, 2, 1) A IR A(:, 3, 1) vec(a) = = A(:, 1, 2) A(:, 2, 2) A(:, 3, 2) a 111 a 211 a 121 a 221 a 131 a 231 a 112 a 212 a 122 a 222 a 132 a 232 We need to specify the order of the stacking... Four Talks on Tensor Computations 1. Connecting to Matrix Computations 41 / 56

42 Parts of a Tensor Slices A slice of a tensor A is a matrix obtained by fixing all but two of A s indices. For example, if A = A(1:3, 1:5, 1:4, 1:7), then A(:, 3, :, 6) = A(1, 3, 1, 6) A(1, 3, 2, 6) A(1, 3, 3, 6) A(1, 3, 4, 6) A(2, 3, 1, 6) A(2, 3, 2, 6) A(2, 3, 3, 6) A(2, 3, 4, 6) A(3, 3, 1, 6) A(3, 3, 2, 6) A(3, 3, 3, 6) A(3, 3, 4, 6) is a slice. We adopt the convention that the first unfixed index in the tensor is the row index of the slice and the second unfixed index in the tensor is the column index of the slice. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 42 / 56

43 Tensor Notation: Subscript Vectors Referencing a Tensor Entry If A IR n 1 n d and i = (i 1,..., i d ) with 1 i k n k for k = 1:d, then A(i) A(i 1,..., i k ) We say that i is a subscript vector. Bold font will be used designate subscript vectors. Bounding Subscript Vectors If L and R are subscript vectors having the same dimension, then L R means that L k R k for all k. Special Subscript Vectors The subscript vector of all ones is denoted by 1. (Dimension clear from context.) If N is an integer, then N = N 1. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 43 / 56

44 Subtensors Specification of Subtensors Suppose A IR n 1 n d with n = [ n 1,..., n d ]. If 1 L R n, then A(L:R) denotes the subtensor B = A(L 1 :R 1,..., L d :R d ) Four Talks on Tensor Computations 1. Connecting to Matrix Computations 44 / 56

45 The Index-Mapping Function col Definition If i and n are length-d subscript vectors, then the integer-valued function col(i, n) is defined by col(i, n) = i 1 + (i 2 1)n 1 + (i 3 1)n 1 n (i d 1)n 1 n d 1 The Formal Specification of Vec If A IR n 1 n d and v = vec(a), then a (col(i, n)) = A(i) 1 i n Four Talks on Tensor Computations 1. Connecting to Matrix Computations 45 / 56

46 The Importance of Notation Contractions A(i 1, i 2, j 1, j 2, j 3 ) = B(i 1, i 2, k 1, k 2 )C(k 1, k 2, j 1, j 2, j 3 ) k 1 k 2 is better written as A(i, j) = k B(i, k)c(k, j) Four Talks on Tensor Computations 1. Connecting to Matrix Computations 46 / 56

47 Software: The Matlab Tensor Toolbox Why? It provides an excellent environment for learning about tensor computations and for building a facility with multi-index reasoning. Who? Brett W. Bader and Tamara G. Kolda, Sandia Laboratories Where? tkolda/tensortoolbox/ Four Talks on Tensor Computations 1. Connecting to Matrix Computations 47 / 56

48 Matlab Tensor Toolbox: A Tensor is a Structure >> n = [ ]; >> A_array = randn ( n); >> A = tensor ( A_array ); >> fieldnames ( A) ans = data size The.data field is a multi-dimensional array. The.size field is an integer vector that specifies the modal dimensions. A.data and A array have same value. A.size and n have the same value. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 47 / 56

49 Matlab Tensor Toolbox: Order and Dimension >> n = [ ]; >> A_array = randn ( n); >> A = tensor ( A_array ) >> d = ndims ( A) d = 4 >> n = size ( A) n = Four Talks on Tensor Computations 1. Connecting to Matrix Computations 47 / 56

50 Matlab Tensor Toolbox: Tensor Operations n = [ ]; % Familiar initializations... X = tenzeros (n); Y = tenones (n); A = tenrand (n); B = tenrand (n); % Extracting Fibers and Slices a_fiber = A (2,:,3,6); a_slice = A (3,:,2,:); % These operations are legal... C = 3* A; C = -A; C = A +1; C = A.^2; C = A + B; C = A./B; C = A.*B; C = A.^B; % Frobenius norm... err = norm (A); % Applying a Function to Each Entry... F = tenfun (@sin,a); G = tenfun (@(x) sin (x )./ exp (x),a); Four Talks on Tensor Computations 1. Connecting to Matrix Computations 48 / 56

51 The Matlab Tensor Toolbox Problem 1. Suppose A = A(1:n 1, 1:n 2, 1:n 3). We say A(i) is an interior entry if 1 < i < n. Otherwise, we say A(i) is an edge entry. If A(i 1, i 2, i 3) is an interior entry, then it has six neighbors: A(i 1 ± 1, i 2, i 3),A(i 1, i 2 ± 1, i 3), and A(i 1, i 2, i 3 ± 1). Implement the following function so that it performs as specified function B = Smooth(A) % A is a third order tensor % B is a third order tensor with the property that each % interior entry B(i) is the average of A(i) s six % neighbors. If B(i) is an edge entry, then B(i) = A(i). Strive for an implementation that does not have any loops. Problem 2. Formulate and solve an order-d version of Problem 1. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 48 / 56

52 Modal Unfoldings Using tenmat Example: A Mode-2 Unfolding of A IR tenmat(a,2) sets up A (2) : a 111 a 211 a 311 a 411 a 112 a 212 a 312 a 412 a 121 a 221 a 321 a 421 a 122 a 222 a 322 a 422 a 131 a 231 a 331 a 431 a 132 a 232 a 332 a 432 (1,1) (2,1) (3,1) (4,1) (1,2) (2,2) (3,2) (4,2) Notice how the fibers are ordered. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 49 / 56

53 Modal Unfoldings Using tenmat Precisely how are the fibers ordered? If A IR n 1 n d, N = n 1 n d, and B = tenmat(a,k), then B the matrix A (k) IR n k (N/n k ) with is where A (k) (i k, col(ĩ k, ñ)) = A(i) ĩ k = [i 1,..., i k 1, i k+1,..., i d ] ñ k = [n 1,..., n k 1, n k+1,..., m d ] Recall that the col function maps multi-indices to integers. For example, if n = [2 3 2] then i (1, 1, 1) (2, 1, 1) 1, 2, 1) (2, 2, 1) (1, 3, 1) (2, 3, 1) (1, 1, 2) (2, 1, 2)... col(i, n) Four Talks on Tensor Computations 1. Connecting to Matrix Computations 50 / 56

54 Matlab Tensor Toolbox: A Tenmat Is a Structure >> n = [ ]; >> A = tenrand (n); >> A2 = tenmat (A,2); >> fieldnames ( A2) ans = tsize rindices cindices data A2.tsize = size of the unfolded tensor = [ ] A2.rindices = mode indices that define A2 s rows = [2] A2.cindices = mode indices that define A2 s columns = [1 3 4] A2.data = the matrix that is the unfolded tensor. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 51 / 56

55 Other Modal Unfoldings Using tenmat Mode-k Unfoldings of A IR n 1 n d A particular mode-k unfolding is defined by a permutation v of [ 1:k 1 k + 1:d ]. Tensor entries get mapped to matrix entries as follows A(i) A(i k, col(i(v), n(v))) Name v How to get it... A (k) (Default) [1:k 1 k +1:d] tenmat(a,k) Forward Cyclic [k + 1:d 1:k 1] tenmat(a,k, fc ) Backward Cyclic [k 1: 1:1 d: 1:k +1] tenmat(a,k, bc ) Arbitrary v tenmat(a,k,v) Four Talks on Tensor Computations 1. Connecting to Matrix Computations 51 / 56

56 Using tenmat Problem 3. Write a Matlab function alpha = MaxFnorm(A) that returns the maximum value of B F where B is a slice of the input tensor A IR n 1 n d. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 52 / 56

57 Summary Key Words Reshaping is about turning a matrix in to a differently sized matrix or into a tensor or into a vector. The Kronecker product is an operation between two matrices that produces a highly structured block matrix. A Rank-1 tensor can be reshaped into a multiple Kronecker product of vectors. A tensor decomposition expresses a given tensor in terms of simpler tensors. The alternating least squares approach to multilinear LS is based on solving a sequence of linear LS problems, each obtained by freezing all but a subset of the unknowns. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 53 / 56

58 Summary Key Words A block matrix is a matrix whose entries are matrices. A fiber of a tensor is a column vector defined by fixing all but one index and varying what s left. A slice of a tensor is a matrix defined by fixing all but two indices and varying what s left. A mode-k unfolding of a tensor is obtained by assembling all the mode-k fibers into a matrix. tensor is a Tensor Toolbox command used to construct tensors. tenmat is a Tensor Toolbox command that is used to construct unfoldings of a tensor. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 54 / 56

59 Summary Recurring themes... Given tensor A IR n 1 n d, there are many ways to unfold a tensor into a matrix A IR N 1 N 2 where N 1 N 2 = n 1 n d. For example, A could be a block matrix whose entries are A-slices. A facility with block matrices and tensor indexing is required to understand the layout possibilities. Computations with the unfolded tensor frequently involve the Kronecker product. Four Talks on Tensor Computations 1. Connecting to Matrix Computations 55 / 56

60 References L. De Lathauwer and B. De Moor (1998). From Matrix to Tensor: Multilinear Algebra and Signal Processing, in Mathematics in Signal Processing IV, J. McWhirter and I. Proudler, eds., Clarendon Press, Oxford, G.H. Golub and C. Van Loan (2013). Matrix Computations, 4th Edition, Johns Hopkins University Press, Baltimore, MD. T.G. Kolda and B.W. Bader (2009). Tensor Decompositions and Applications, SIAM Review 51, P.M. Kroonenberg (2008). Applied Multiway Data Analysis, Wiley. A. Smilde, R. Bro, and P. Geladi (2004). Multi-Way Analysis: Applications in the Chemical Sciences, Wiley, West Sussex, England. B.W. Bader and T.G. Kolda (2006). Algorithm 862: MATLAB Tensor Classes for Fast Algorithm Prototyping, ACM Transactions on Mathematical Software, 32, Four Talks on Tensor Computations 1. Connecting to Matrix Computations 56 / 56