Tamara G. Kolda and Brett W. Bader Part I September 22, 2015
What is tensor? A N-th order tensor is an element of the tensor product of N vector spaces, each of which has its own coordinate system. a = a i e i b = b j ẽ j c = c k ê k (Vector spaces) A = a b = a i b j = Aij e i ẽ j (Matrix, second order tensor) X = a b c = a i b j c k e i ẽ j ê k (Third order tensor) = X ijk... N-th order (ways or modes) tensor has N dimensions.
From matrix to high order tensor Table : Matrix to high order tensor Matrix High order tensor (Columns/Rows)A i, A j (Fibers)X ij, X i k, X jk ; (Slices) X i, X j, X k mode-3, 2, 1; Horizontal, Lateral, Frontal slices A B = A ij B ij X Y = X ijk Y ijk i=1 j=1 i=1 j=1 k=1 A F = A ij X = X i=1 j=1 (Rank one matrix) A = ab T A ij = a i b j (Symmetric) A = A T A = I, A ij = δ ij ijk i=1 j=1 k=1 (Rank one tensor) X = a b c X ijk = a i b j c k (Supersymmetric: cubical + symmetry) X R I I I X ijk is constant when permuting i, j, k X = I, X ijk = δ ijk
Matricization (unfolding, flattening) Vectorization of Matrix: arrange all columns of the matrix to be a column vector. ( 1 3 1 2 4 ) 2 3 4 Mode-n matricization of a tensor: arrange all mode-n fibers to be the columns of the resulting matrix. ( Let s skip the formal definition. ) 12 4 7 X ij1 = 2 10 6 1 16 9 13 3 73 X ij2 = 21 0 26 11 27 19 Mode-1 matricization: X (1) 12 4 7 13 3 73 2 10 6 21 0 26 1 16 9 11 27 19
Mode-2 matricization: X (2) 12 2 1 13 21 11 4 10 16 3 0 27 7 6 9 73 26 19 Mode-3 matricization: X (3) 12 2 1 4 10 16 6 9 ( 13 21 11 3 0 27 26 19 ) How do we change the results if we have an additional slice? 1 1 1 X ij3 = 1 1 1 1 1 1
Tensor multiplication: n-mode product The n-mode product of a tensor X R I 1 R I 2 R I N with a matrix U R J In is denoted by X n U. (X n U) i1 i n 1 ji n+1 i N = x i1 i ni n+1 i N u jin i n=1 I n Y = X n U Y (n) = UX (n) related to change of basis, when a tensor defines a multilinear operator. The n-mode product of a tensor X R I 1 R I 2 R I N with a vector v R In is denoted by X nv. I n (X nv) i1 i n 1 i n+1 i N = x i1 i N v in i n=1 precedence matters, contraction of tensor.
Matrix Kronecker,Khatri Rao and Hadamard Products A, B a 11 B a 1J B Kronecker product: A B = a I 1 B a IJ B Khatri Rao product: A B = [a 1 b 1 Hadamard Product: (A B) ij = A ij B ij Examples: MATLAB: kron(a,b); % Kronecker for i = 1:3 % Khatri-Rao C(:,i) = kron(a(:,i),b(:,i)); end A.*B %Hadamard 1 2 3 A = 4 5 6 7 8 9 a 2 b 2 a K b K ] (column matching) B = ( 1 2 3 4 5 6 ) A B?A B?A B?A A?
Tensor Rank and the CP Decomposition CP Decomposition (CANDECOMP/PARAFAC Decomposition), factorizes a tensor into a sum of component rank-one tensors. CP decomposition can be treated as a generalization of SVD to higher order tensors. R A = σ r u r v r, with σ 1 σ 2 σ R > 0 (SVD) r=1 X R I J K R λ r a r b r c r = λ; A, B, C r=1 A, B, and C are combination of the vectors from the rank-one components and are normalized to length one. In matricized form: X (1) AΛ(C B) T, X (2) BΛ(C A) T, X (3) CΛ(B A) T where Λ = diag(λ). It can be extended to n-th order tensor.
Tensor Rank The rank of a tensor X, denoted rank(x ), is the smallest number of rank-one tensors that generate X as their sum. (the smallest number of components in an exact = CP decomposition. In other words, what is R?) An exact CP decomposition with R = rank(x ) components is called the rank decomposition. Definition is an exact analogue from matrix rank (the dimension of the vector space spanned by its columns). Properties are quite different: (1) Field dependent: may not the same over R and C. (2) No straightforward algorithm to compute the rank of specific tensor (NP-hard problem). Matrix decompositions are not unique (why?), however high order tensor decompositions are often unique.
Low-Rank Approximations Matrix: a best rank-k approximation is given by the leading k factors of the SVD. A rank-k approximation that minimizes A B is given by: k A = σ r u r v r, with σ 1 σ 2 σ R > 0 r=1 Not true for high order tensors! a best rank-k approximation may not exist! (degeneracy)
Computing CP Decomposition Suppose R is fixed, for a third order tensor X, we are looking for CP Decomposition of ˆX : min X ˆX with ˆX = λ; A, B, C ˆX The alternating least squares (ALS) method: (fixed all but one matrix) Fix C and B (specify in any way) and rewrite (how?), Â = AΛ min X (1) Â(C B) T F Â Solution: Â = X (1) [(C B) T ], or a better version: Â = X (1) (C B)[(C T C B T B)]. refers to Moore Penrose pseudoinverse (MATLAB: B = pinv(a)). Normalization to get A, storing norms as λ. Fix A and B to compute C., until reach convergence (i.e. error stops decreasing). Implementability? Convergence? Efficiency?
Tensor toolbox MATLAB Sandia National Laboratories provide a MATLAB Tensor Toolbox which includes an implementation of CP Decomposition. cp als function implements ALS algorithm for CP Decomposition with fixed rank. e.g P = cp als(x, 2); other approaches are avaliable cp apr cp nmu, cp opt, etc. special treatment for sparse tensor: a greedy CP. Applications: signal/image processing neuroscience data mining (user keyword time: chatroom tensors) stochastic PDEs.
THANK YOU!