1. Structured representation of high-order tensors revisited. 2. Multi-linear algebra (MLA) with Kronecker-product data.

Transcription

1 Lect. 4. Toward MLA in tensor-product formats B. Khoromskij, Leipzig 2007(L4) 1 Contents of Lecture 4 1. Structured representation of high-order tensors revisited. - Tucker model. - Canonical (PARAFAC) model. - Two-level and mixed models. 2. Multi-linear algebra (MLA) with Kronecker-product data. - Invariace of some matrix properties. - Commutator, matrix exponential, eigen-value problem. - Lyapunov equation. - Complexity issues. 3. Algebaric methods of tensor-product decomposition.

2 Rank-(r 1,..., r d ) Tucker model B. Khoromskij, Leipzig 2007(L4) 2 Tucker Model (T r ). (orthonormalised set V (l) k l R I l ) A (r) = r 1 k 1 =1... r d k d =1 b k1...k d V (1) k 1... V (d) k d R I 1... I d. Core tens. B = {b k } R r 1... r d is not unique (up to rotations) Complexity (p = 1): r d + rdn n d with r = max r l n. Visualization of the Tucker model with d = 3: I 3 r 3 V (3) A I 3 I 1 = B r 1 r 2 I 2 V (2) I 2 I 1 V (1)

3 CANDECOMP/PARAFAC (CP) tensor format B. Khoromskij, Leipzig 2007(L4) 3 CP Model (C r ). Approx. A by a sum of rank-1 tensors A (r) = r k=1 b k V (1) k V (d) k A, b k R with normalised V (l) k R np. Uniqueness is due to J. Kruskal 77. Complexity: r + rdn. The minimal number r is called a tensor rank of A (r). (3) (3) (3) 1 2 r V V V A b 1 b 2 (2) (2) (2) = V + V V 1 2 (1) (1) (1) 1 2 r V V V b r r Figure 1: Visualization of the CP-model for d = 3.

4 Two-level and mixed models B. Khoromskij, Leipzig 2007(L4) 4 Two-level Tucker model T (U,r,q), A (r,q) = B V (1) V (2)... V (d) T (U,r,q) C (n,q), 1. B R r 1... r d is retrieved by the rank-q CP model C (r,q) 2. V (l) = [V (l) 1 V (l) 2...V (l) r l ] {U}, l = 1,..., d, {U} spans fixed (uniform/adaptive) basis; O(r d ) with r = max l d r l O(dqr) (independent of n!). Mixed model M C,T : A = A 1 + A 2, A 1 C r1, A 2 T r2. Applies to ill-conditioned tensors.

5 Examples of two-level models B. Khoromskij, Leipzig 2007(L4) 5 (I) Tensor-product sinc-interpolation: analytic functions with point singularities, r = (r,..., r), r = q = O(log n log ε ) O(dqr). (II) Adaptive two-level approximation: Tucker + CP decomp. of B with q r O(dqn). (III) Sparse grids: regularity of mixed derivatives, r = (n 1,..., n d ), hyperbolic cross selection q = n log d n O(n log d n). Structured tensor-product models (d-th order tensors of size n d ) Model Notation Memory/A x A B Comput. tools Canonical - CP Cr drn drn 2 ALS/Newton HKT - CP C H,r dr n log q n drn log q n Analytic (quadr.) Nested - CP C T(I),L dr log d n+ rd dr log d n SVD/QR/orthog. iter. Tucker T r r d + drn - Orthogonal ALS Two-level Tucker T (U,r,q) drq/drr 0 qn 2 dr 2 q 2 (mem.) Analyt.(interp.) + CP

6 Challenge of multi-factor analysis B. Khoromskij, Leipzig 2007(L4) 6 Paradigm: linear algebra vs. multi-linear algebra. CP/Tucker tensor-product models have plenty of merits: 1. A (r) is repr. with low cost drn (resp. drn + r d ) n d. 2. V (l) k can be repr. in the data-sparse form: H-matrix (HKT), wavelet-based (WKT), uniform basis. 3. The core tensor B = {b k } can be sparsified via CP model. 4. Efficient numerical MLA Highly nonlinear problems. Remark. CP decomposition (unique!) can t be retrieved by rotation and truncation of the Tucker model, C r = T r if r = 1 d = 2, but C r T r if r = r 2 d 3.

7 Little analogy between the cases d 3 and d = 2 B. Khoromskij, Leipzig 2007(L4) 7 I. rank(a) depends on the number field (say, R or C). II. We do not know any finite algorithm to compute r = rank(a), except simple bounds: rank(a) n d 1 ; rank(a) rank(a 1 ) rank(a n d 2) III. For fixed d and n we do not know the exact value of max{rank(a)}. J. Kruskal 75 proved that: for any tensor we have max{rank(a)} = 3 < 4; for tensors there holds max{rank(a)} = 5 < 9. IV. Probabilistic properties of rank(a): in the set of tensors there is about 79% of rank-2 tensors and 21% of rank-3 tensors, while rank-1 tensors appear with probability 0. Clearly, for n n matrices we have P{rank(A) = n} = 1.

8 Little analogy between the cases d 3 and d = 2 B. Khoromskij, Leipzig 2007(L4) 8 V. However, it is possible to prove very important uniqueness property within the equivalence classes. Two CP-type representations are considered as equivalent if either (a) they differ in the order of terms or (b) for some set of paramers a l k R such that dq there is a transform V (l) k a l k V (l) k. a l k l=1 = 1 (k = 1,..., r), A simplified version of the general uniqueness result is the following (all factors have the same full rank r). Prop. 1. (J. Kruskal, 1977) Let for each l = 1,..., d, the vectors V (l) k, (k = 1,..., r) with r = rank(a), are linear independent. If (d 2)r d 1, then the CP decomposition is uniquely determined up to the equivalence (a) - (b) above.

9 Properties of the Kronecker product B. Khoromskij, Leipzig 2007(L4) 9 A tensor A R I 1... I d can be viewed as: A. An element of linear space of vectors with the l 2 -inner product and related Frobenius norm, which is a multi-variate function of the discrete argument A : I 1... I d R. B. A mapping A : R I 1... I q R I q+1... I d (hence, requiring matrix operations in the tensor format). Def The Kronecker product (KP) operation A B of two matrices A = [a ij ] R m n, B R h g is an mh ng matrix that has the block-representation [a ij B]. Ex In general A B B A. What is the condition on A and B that provides A B = B A?

10 Properties of the Kronecker product B. Khoromskij, Leipzig 2007(L4) Let C R s t, then the KP satisfies the associative law, (A B) C = A (B C) = A B C R mhs ngt, and therefore we do not use brackets. 2. Let C R n r, D R g s, then the matrix-matrix product in the Kronecker format takes the form (A B)(C D) = (AC) (BD). The extension to d-th order tensors is (A 1... A d )(B 1... B d ) = (A 1 B 1 )... (A d B d ).

11 Properties of the Kronecker product B. Khoromskij, Leipzig 2007(L4) We have the distributive law (A + B) (C + D) = A C + A D + B C + B D. 4. Rank relation: rank(a B) = rank(a)rank(b). Invariance of some matrix properties: (1) If A and B are diagonal then A B is also diagonal, and conversely (if A B 0). (2) (A B) T = A T B T, (A B) = A B. (3) Let A and B be Hermitian/normal matrices (A = A resp. A 1 = A). Then A B is of the corresponding type. (4) A R n n, B R m m det(a B) = (deta) m (detb) n. Hint: A B = diag n {B} A I m.

12 Matrix operations with the Kronecker product B. Khoromskij, Leipzig 2007(L4) 12 Thm Let A R n n and B R m m be invertible matrices. Then (A B) 1 = A 1 B 1. Proof. Since det(a) 0, det(b) 0 and the above property (4) we have det(a B) 0. Thus (A B) 1 exists and (A 1 B 1 )(A B) = (A 1 A) (B 1 B) = I nm. Lem Let A R n n and B R m m be unitary matrices. Then A B is a unitary matrix. Proof. Since A = A 1, B = B 1 we have (A B) = A B = A 1 B 1 = (A B) 1.

13 Matrix operations with the Kronecker product B. Khoromskij, Leipzig 2007(L4) 13 Define the commutator [A, B] := AB BA. Lem Let A R n n and B R m m. Then [A I n, I m B] = 0 R m2 n 2. Proof. [A I n, I m B] = (A I n )(I m B) (I m B)(A I n ) = A B A B = 0. Lem Let A, B R n n, C, D R m m and [A, B] = 0, [C, D] = 0. Then [A C, B D] = 0. Proof. Apply the identity (A B)(C D) = (AC) (BD).

14 Matrix operations with the Kronecker product B. Khoromskij, Leipzig 2007(L4) 14 Lem Let A R n n and B R m m. Then tr(a B) = tr(a)tr(b). Proof. Since diag(a ii B) = a ii diag(b), we have tr(a B) = n m a ii b jj = n m a ii b jj. i=1 j=1 i=1 j=1 Thm Let A, B, I R n n. Then exp(a I + I B) = (expa) (expb). Proof. Since [A I, I B] = 0, we have exp(a I + I B) = exp(a I) exp(i B).

15 Matrix operations with the Kronecker product B. Khoromskij, Leipzig 2007(L4) 15 Furthermore, since exp(a I) = k=0 (A I) k, exp(i B) = k! m=0 (I B) m m! the arbitrary term in exp(a I) exp(i B) is given by Imposing 1 k! 1 m! (A I)k (I B) m. (A I) k (I B) m = (A k I k )(I m B m ) = (A k I)(I B m ) A k B m, we finally arrive at 1 k! 1 m! (A I)k (I B) m = ( 1 k! Ak ) ( 1 m! Bm ).

16 Matrix operations with the Kronecker product B. Khoromskij, Leipzig 2007(L4) 16 Thm. 4.2 can be extended to the case of many-term sum exp(a 1 I... I+I A 2... I+...+I... I A d ) = (e A 1 )... (e A d ). Other simple properties: sin(i n A) = I n sin(a), sin(a I m + I n B) = sin(a) cos(b) + cos(a) sin(b),

17 Eigenvalue problem B. Khoromskij, Leipzig 2007(L4) 17 Lem Let A R m m and B R n n have the eigen-data λ j, u j, j = 1,..., m, and µ k, v k, k = 1,..., n, respectively. Then A B has the eigenvalues λ j µ k with the corresponding eigenvectors u j v k, 1 j m, 1 k n. Thm Under the conditions of Lem. 4.5 the eigenvalues/eigenfunctions of A I n + I m B are given by λ j + µ k and u j v k, respectively. Proof. Due to Lem. 4.5 we have (A I n + I m B)(u j v k ) = (A I n )(u j v k ) + (I m B)(u j v k ) = (Au j ) (I n v k ) + (I m u j ) (Bv k ) = (λ j u j ) v k + u j (µ k v k ) = (λ j + µ k )(u j v k ).

18 Lyapunov/Silvester equations B. Khoromskij, Leipzig 2007(L4) 18 For a matrix A R m n we use the vector representation A vec(a) R mn, where vec(a) is an nm 1 vector obtained by stacking A s columns (the FORTRAN-style ordering) vec(a) := [a 11,..., a n1, a 12,..., a nm ] T. In this way, vec(a) is a rearranged version of A. The matrix Sylvester equation for X R m n AX + XB T = G R m m with A R m m, B R n n, can be written in vector form (I n A + B I m )vec(x) = vec(g). In the special case A = B we have the Lyapunov equation.

19 Lyapunov/Silvester equations B. Khoromskij, Leipzig 2007(L4) 19 Now the solvability conditions and certain solution methods can be derived (cf. the results for eigenvalue problems). Silvester equation is uniquely solvable if λ j (A) + µ k (B) 0. Moreover, since I n A and B I m commute, we can apply all methods proposed below to represent the inverse ) (I n A + B I m ) (= 1 e (I n A+B I m )t dt. In particular, if A and B correspond to the discrete elliptic operators in R d with separable coefficients, we obtain the low-rank tensor-product decomposition to the Sylvester solution operator (cf. Lect. 7/2005). 0

20 Kronecker Hadamard product B. Khoromskij, Leipzig 2007(L4) 20 Lemma 4.6 indicates the simple (but important) property of the Hadamard product of two tensors A, B R Id, defined by C = A B = {c i1...i d } (i1...i d ) I d defined by the entry-wise multiplication c i1...i d = a i1...i q b i1...i d. Lem Let both A and B be represented by the CP model with the Kronecker rank r A, r B and with V l k substituted by A l k RI and B l k RI, respectively. Then A B is a tensor with the Kronecker rank r = r A r B given by A B = r A r B k=1 m=1 c k c m (A 1 k B 1 m)... (A d k B d m).

21 Kronecker Hadamard product B. Khoromskij, Leipzig 2007(L4) 21 Proof. It is easy to check that (A 1 B 1 ) (A 2 B 2 ) = (A 1 A 2 ) (B 1 B 2 ), and similar for d-term products. Applying the above relations, we obtain ( ra A B = = c k d k=1 l=1 r A k=1 r B and the assertion follows. A l k m=1c k c m ( d l=1 ) ( rb A l k m=1 ) c m d l=1 ( d B l m Bm l l=1 ) )

22 Complexity of the HKT-matrix arithmetics B. Khoromskij, Leipzig 2007(L4) 22 Complexity issues Let V l k M H,s(T I I, P) in the CP represent. and let N = n d. Data compression. The storage for A is O(rdsn log n), r = O(log α N), α > 0. Hence, we enjoy the sub-linear complexity. Matrix-by-vector complexity of Ax, x C N. For general x one has the linear cost O(rdsN log n). If x = x 1... x d, x i C n, we again arrive at sub-linear complexity O(rdsn log n). Matrix-by-matrix complexity of AB and A B. The H-matr. struct. of the Kronecker factors leads to O(r 2 ds 2 n log q n) operations instead of O(N 3 ).

23 How to construct a Kronecker product? B. Khoromskij, Leipzig 2007(L4) d = 2: SVD and ACA methods in the case of two-fold decompositions. 2. d 2: Analytic approximation for the function-related d-th order tensors (consider in Lect. 5). Def Given the multi-variate function g : Ω R d R with d = dp, p, d N, d 2, Ω = {(ζ 1,..., ζ d ) R d : ζ l L, l = 1,..., d} R d, L > 0, where means the l -norm of ζ l R p (p = 1). Introduce the function-generated d-th order tensor A A(g) := [a i1...i d ] R Id with a i1...i d := g(ζi 1 1,..., ζi d d ). (1) Approximation tools: sinc-methods, exponential fitting.

24 How to construct a Kronecker product? B. Khoromskij, Leipzig 2007(L4) d 3: Algebraic recompression methods: 3A. Greedy algorithms with dictionary { } D := V (1) 2 V (2)... d V (d) : V (l) R n, V (l) = 1. (a) Fit the original tensor A by a rank-one tensor A 1 ; (b) Subtract A 1 from the original tensor A; (c) Approx. the residue A A 1 with another rank-one tensor. For best rank-1 appr. one solves the minimisation problem min A V (1) V (d) F, V (l) R np l, by using ALS or the Newton iteration (proven convergence). In general, convergence theory for Greedy algorithm is still open question (see Lect.1).

25 How to construct a Kronecker product? B. Khoromskij, Leipzig 2007(L4) 25 Def A tensor A C r is orthogonally decomposable if (V (l) k, V (l) k ) = δ k,k (k, k = 1,..., r; l = 1,..., d). Thm (Zhang, Golub) If a tensor of order d 3 is orthogonally decomposable, then this decomposition is unique, and the OGA correctly computes it. Proof: See Lect. 1. (3B) The Newton algorithm to solve the Lagrange eq. in the constrained minimisation: Find A C r and λ (k,l) R s.t. f(a) := A A 0 2 F + r k=1 d l=1 ( ) λ (k,l) V (l) k 2 1 min. (2) Efficient implementation of the Newton algorithm (M. Espig, MPI MIS).

26 How to construct a Kronecker product? B. Khoromskij, Leipzig 2007(L4) 26 (3C) Alternating least-squares (ALS). Mode per mode components update, fix all V (l), l m (m = 1,..., d). Convergence theory only for r = 1 (Golub, Zhang; Kolda 01) Under certain simplifications, the constraint ALS minimisation algorithm can be implemented in O(m 2 n + K it dr 2 m) op. (see Lect. 5). The convergence theory behind these algorithms is not complete, moreover the solution might not be unique or even might not exist.

27 Summary I B. Khoromskij, Leipzig 2007(L4) 27 Motivation: Basic linear algebra can be performed using one-dimensional operations, thus avoiding the exponential scaling in d. Bottleneck: Lack of finite algebraic methods for the robust multi-fold Kronecker decomposition of high order tensors (for d 3). Difficulties with recompression in matrix operations. There are efficient and robust ALS/Newton algorithms. Observation: Analytic approximation methods are of principal importance. Classical example: an approximation by Gaussians. Recent proposals: Sinc meth., exponential fitting, sparse grids.