4. Multi-linear algebra (MLA) with Kronecker-product data.

Transcription

1 ect. 3. Tensor-product interpolation. Introduction to MLA. B. Khoromskij, Leipzig 2007(L3) 1 Contents of Lecture 3 1. Best polynomial approximation. 2. Error bound for tensor-product interpolants. - Polynomial interpolation. - Sinc interpolation. 3. Data-sparse formats to represent high-order tensors. - Tucker model. - Canonical (PARAFAC) model. - Two-level and mixed models. 4. Multi-linear algebra (MLA) with Kronecker-product data.

2 Chebyshev polynomials B. Khoromskij, Leipzig 2007(L3) 2 By E ρ = E ρ (B) with the reference interval B := [ 1, 1], we denote the Bernstein s regularity ellipse (with foci at w = ±1 and the sum of semi-axes equal to ρ > 1), E ρ := {w C : w 1 + w + 1 ρ + ρ 1 }. The Chebyshev polynomials, T n (w), are defined recursively T 0 (w) = 1, T 1 (w) = w, T n+1 (w) = 2wT n (w) T n 1 (w), n = 1, 2,.... Representation T n (x) = cos(n arccosx), x [ 1, 1], implies T n (1) = 1, T n ( 1) = ( 1) n. There holds T n (w) = 1 2 (zn + z n ) with w = 1 2 (z + 1 z ).

3 Best polynomial approximation by Chebyshev series B. Khoromskij, Leipzig 2007(L3) 3 Thm Let F be analytic and bounded by M in E ρ (with ρ > 1). Then the expansion F(w) = C C n T n (w), (1) n=1 holds for all w E ρ (Chebyshev series), and with C n = 1 π 1 1 F(w)T n (w) 1 w 2 dw. Moreover, C n M/ρ n and for w B and for m = 1, 2, 3,..., F(w) C 0 2 m n=1 C n T n (w) 2M ρ 1 ρ m, w B. (2)

4 Lagrangian polynomial interpolation B. Khoromskij, Leipzig 2007(L3) 4 Let P N (B) be the set of polynomials of degree N on B. Define by [I N F](x) P N (B) the interpolation polynomial of F w.r.t. the Chebyshev-Gauss-Lobatto (CGL) nodes ξ j = cos πj N B, j = 0, 1,...,N, with ξ 0 = 1, ξ N = 1, where ξ j are zeroes of the polynomials (1 x 2 )T N (x), x B. The Lagrangian interpolant I N of F has the form I N F := N F(ξ j )l j (x) P N (B) (3) j=0 with l j (x) being the set of interpolation polynomials l j := N k=0,j k x ξ k ξ j ξ k P N (B), j = 0,...,N. Clearly, I N (ξ j ) = F(ξ j ), since l j (ξ j ) = 1 and l j (ξ k ) = 0 k j.

5 Lebesque constant for Chebyshev interpolation B. Khoromskij, Leipzig 2007(L3) 5 Given the set {ξ j } N j=0 of interpolation points on [ 1, 1] and the associated Lagrangian interpolation operator I N. The approximation theory for polynomial interpolation includes the so-called Lebesque constant Λ N R >1, I N u,b Λ N u,b u C(B). (4) In the case of Chebyshev interpolation it can be shown that Λ N grows at most logarithmically in N, Λ N 2 π log N + 1. The interpolation points which produce the smallest value Λ N of all Λ N are not known, but Bernstein 54 proves that Λ N = 2 π log N + O(1).

6 Error bound for polynomial interpolation B. Khoromskij, Leipzig 2007(L3) 6 Thm Let u C [ 1, 1] have an analytic extension to E ρ bounded by M > 0 in E ρ (with ρ > 1). Then we have u I N u,i (1 + Λ N ) 2M ρ 1 ρ N, N N 1. (5) Proof. Due to (2) one obtains for the best polynomial approximations to u on [ 1, 1], min u v,b 2M v P N ρ 1 ρ N. The interpolation operator I N is a projection, that is, for all v P N we have I N v = v. Now apply the triangle inequality, u I N u,b = u v I N (u v),b (1 + Λ N ) u v,b.

7 Tensor-product polynomial interpolation B. Khoromskij, Leipzig 2007(L3) 7 Consider a multi-variate funct. f = f(x 1,...,x d ) : B d R, d 2, defined on a box B d = B 1 B 2... B d with B k = B = [ 1, 1]. Define N-th order tensor product interpolation operator I N f = I 1 N I 2 N... I d Nf P N [B d ], where I k N f denotes the interpolation polynomial w.r.t. x k, at nodes {ξ k } B k, k = 1,...,d. We choose the CGL nodes, hence the interpolation points ξ α B d, α = (i 1,...,i d ) N d 0, are obtained by the Cartesian product of 1D-nodes, ( ξ α := cos πi ) 1 N,...,cosπi d. N

8 Tensor-product polynomial interpolation B. Khoromskij, Leipzig 2007(L3) 8 Again, I N is the projection map, I N : C(B d ) P N := {p 1... p d : p i P N, i = 1,...d} implying stability of I N in the multidimensional case, cf. (4), I N f,b d Λ d N f,b d f C(B d ). (6) To derive an analogue of Thm. 3.2, introduce the product domain E (j) ρ := B 1... B j 1 E ρ (I j ) B j+1... B d, and denote by X j the (d 1)-dimensional subset of variables {x 1,...,x j 1, x j+1,...,x d } with x j B j, j = 1,..., d.

9 Tensor-product polynomial interpolation B. Khoromskij, Leipzig 2007(L3) 9 Assump Given f C (B d ), assume there is ρ > 1 s.t. for all j = 1,...,d, and each fixed ξ X j, there exists an analytic extension ˆf j (x j, ξ) of f(x j, ξ) to E ρ (B j ) C w.r.t. x j bounded in E ρ (B j ) by certain M j > 0, independent on ξ. Thm For f C (B d ), let Assump. 3.1 be satisfied. Then the interpolation error can be estimated by f I N f,b d Λ d N 2M ρ (f) ρ 1 ρ N, (7) where Λ N is the Lebesque const. for the 1D interpolant I k N, and M ρ (f) := max { max ˆf j (x, ξ) }. 1 j d x E ρ (j)

10 Tensor-product polynomial interpolation B. Khoromskij, Leipzig 2007(L3) 10 Proof. Multiple use of (4), (5) and the triangle inequality lead to f I N f f I 1 Nf + I 1 N(f I 2 N... I d Nf) f I 1 Nf + I 1 N(f I 2 Nf) + + INI 1 N(f 2 INf) IN 1... I d 1 N (f Id Nf) [(1 + Λ N ) max x E (1) ρ Λ d 1 N (1 + Λ N)(Λ d N 1) Λ N 1 ˆf 1 (x, ξ) + Λ N (1 + Λ N ) max ˆf 2 (x, ξ) x E ρ (2) (1 + Λ N) max ˆf d (x, ξ) ] x E ρ (d) 2M ρ ρ 1 ρ N. 2 ρ 1 ρ N Hence (7) follows since for x > 1 we have (1+x)(xn 1) x 1 x n.

11 Sinc-approximation of multi-variate functions B. Khoromskij, Leipzig 2007(L3) 11 Consider the separable approximation in the case Ω = R. Extension to the case Ω = R + or Ω = (a, b) is possible. The tensor-product Sinc interpolant C M w.r.t. the first d 1 variables, reads C M f := C 1 M... C d 1 M f, f : Rd R, where CM l f = Cl M (f, h), 1 l d, is the univariate Sinc interp. C M (f, h) = M k= M f(kh)s k,h (x), in x l I l = R, with R d = I 1... I d. Ex Examples of approximated function (x, y R d ) f(x) = x α, f(x) = exp(κ x ) x, f(x, y) = sinc( x y ).

12 Sinc-approximation of multi-variate functions B. Khoromskij, Leipzig 2007(L3) 12 Error bound for tensor-product Sinc interpolant. The estimation of the error f C M f requires the Lebesgue constant Λ M 1 defined by C M (f, h) Λ M f for all f C(R). (8) Stenger 93 proves the inequality Λ M = max x R M k= M S k,h (x) 2 (3 + log(m)). (9) π For each fixed l {1,...,d 1}, choose ζ l I l and define the remaining parameter set by Y l := I 1... I l 1 I l+1... I d R d 1.

13 Sinc-approximation of multi-variate functions B. Khoromskij, Leipzig 2007(L3) 13 Introduces the univariate (parameter dependent) function F l (, y) : I l R, y Y l, which is the restriction of f onto I l. Thm (Hackbusch, Khoromskij) For each l = 1,..., d 1 we assume that for any fixed y Y l, F l (, y) satisfies (a) F l (, y) H 1 (D δ ) with N(F l, D δ ) N l < uniformly in y; (b) F l (, y) has hyper-exponential decay with a = 1, C, b > 0. Then, for all y Y l, the optimal choice h := log M M yields f C M (f, h) C 2πδ Λd 2 M with Λ M defined by (9). max N l e πδm log M (10) l=1,...,d 1

14 Proof of the Sinc-interpolation error B. Khoromskij, Leipzig 2007(L3) 14 The multiple use of (8) and the triangle inequality lead to f C M f f C 1 Mf + C 1 M(f C 2 M...C d Mf) Note that f C 1 Nf + C 1 M(f C 2 Mf) + + C 1 MC 2 M(f C 3 Mf) C 1 M...C d 2 M hence (10) follows. [N 1 + Λ M N Λ d 2 M N d 1] 1 + Λ M Λ d 2 M 2πδ 1 2πδ e πδm log M max N l e πδm log M. l=1,...,d 1 Λ d 1 M 1 Λ M 1 Λd 2 M, Λ M, (f Cd 1 f) M

15 Data-sparse representation of high-order tensors B. Khoromskij, Leipzig 2007(L3) 15 Def A d-th order tensor on I d = I 1... I d, A := [a i1...i d ] R Id, d = pd, p, d, n N with multi-index i l = (i l,1,..., i l,p ) I l = I 1... I p (l = 1,..., d), and i l,m {1,..., n}, for m = 1,..., p (p = 1, 2, 3). The L 2 inner product of tensors induces the Frobenius norm A, B := a i1...i d b i1...i d, A F := A, A. (i 1...i d ) I d A R Id has I d = n dp entries. How to remove d from the exponential?

16 Data-sparse representation of high-order tensors B. Khoromskij, Leipzig 2007(L3) 16 Key ingredient: representation by a sum of rank-1 tensors A = V (1) 2 d V (d), a i1...i d = v (1) i 1 v (d) i d with low dimensional (canonical) comp. V (l) = {v (l) i l } R np. Complexity: dn p. Standard MLA has linear scaling in d. Ex Let A = a 1 a 2, B = b 1 b 2, a i, b i R n (q = 2, p = 1). Then (A, B) = (a 1, b 1 )(a 2, b 2 ), A F = (a 1, a 1 )(a 2, a 2 ) = a 1 a 2, where the latter corresponds to the Frobenius norm of a matrix.

17 Rank-(r 1,..., r d ) Tucker model B. Khoromskij, Leipzig 2007(L3) 17 Tucker Model (T r ). (orthonormalised set V (l) k l R I l ) A (r) = r 1 k 1 =1... r d k d =1 b k1...k d 1 V (1) k d V (d) k d R I 1... I d. (11) Core tens. B = {b k } R r 1... r d is not unique (up to rotations) Complexity (p = 1): r d + rdn n d with r = max r l n. Visualization of the Tucker model with d = 3: I 3 r 3 V (3) A I 3 I 1 = B r 1 r 2 I 2 V (2) I 2 I 1 V (1)

18 CANDECOMP/PARAFAC (CP) tensor format B. Khoromskij, Leipzig 2007(L3) 18 CP Model (C r ). Approx. A by a sum of rank-1 tensors A (r) = r k=1 b k 1 V (1) k 2 d V (d) k A, b k R with normalised V (l) k R np. Uniqueness is due to J. Kruskal 77. Complexity: r + rdn. The minimal number r is called a tensor rank of A (r). (3) (3) (3) 1 2 r V V V A b 1 b 2 (2) (2) (2) = V + V V 1 2 (1) (1) (1) 1 2 r V V V b r r Figure 1: Visualization of the CP-model for d = 3.

19 Two-level and mixed models B. Khoromskij, Leipzig 2007(L3) 19 Two-level Tucker model T (U,r,q), A (r,q) = B 1 V (1) 2 V (2)... d V (d) T (U,r,q) C (n,q), 1. B R r 1... r d is retrieved by the rank-q CP model C (r,q) 2. V (l) = [V (l) 1 V (l) 2...V (l) r l ] {U}, l = 1,..., d, {U} spans fixed (uniform/adaptive) basis; O(r d ) with r = max l d r l O(dqr) (independent of n!). Mixed model M C,T : A = A 1 + A 2, A 1 C r1, A 2 T r2. Applies to ill-conditioned tensors.

20 Challenge of multi-factor analysis B. Khoromskij, Leipzig 2007(L3) 20 There is little analogy between the cases d = 2 and d 3, Paradigm: linear algebra vs. multi-linear algebra (MLA). CP/Tucker tensor-product models have plenty of merits: 1. A (r) is repr. with low cost drn (resp. drn + r d ) n d. 2. V (l) k can be repr. in the data-sparse form: H-matrix (HKT), wavelet-based (WKT), uniform basis. 3. The core tensor B = {b k } can be sparsified as well. 4. Efficient numerical MLA (practical experience). Remark. CP decomposition (unique!) can t be retrieved by rotation and truncation of the Tucker model, C r = T r if r = 1, but C r T r if r = r 2.

21 Examples of T (U,r,q) -models B. Khoromskij, Leipzig 2007(L3) 21 (I) Tensor-product sinc-interpolation: analytic functions with point singularities, r = (r,..., r), r = q = O(log n log ε ) O(dqr). (II) Sparse grids: regularity of mixed derivatives, r = (n 1,..., n d ), hyperbolic cross q = n log d n O(n log d n). (III) Adaptive two-level appr.: Tucker + CP decomp. of B with q r O(dqn). Structured Kronecker product models (d-th order tensors of size n d ) Model Notation Memory/A x A B Comp. tools Canonical - CP Cr drn drn 2 ALS/Newton HKT - CP C H,r dr n log q n drn log q n Analytic (quadr.) Nested - CP C T(I),L dr log d n+ rd dr log d n SVD/QR/orthog. iter. Tucker T r r d + drn - Orthogonal ALS Two-level Tucker T (U,r,q) drq/drr 0 qn 2 dr 2 q 2 (mem.) Analyt.(interp.)+ CP

22 Properties of the Kronecker product B. Khoromskij, Leipzig 2007(L3) 22 The Kronecker product (KP) operation A B of two matrices A = [a ij ] R m n, B R h g is an mh ng matrix that has the block-representation [a ij B] (corr. to p = 2). 1. Let C R s t, then the KP satisfies the associative law, (A B) C = A (B C), and therefore we do not use brackets. The matrix A B C := (A B) C has (mhs) rows and (ngt) columns. 2. Let C R n r and D R g s, then the standard matrix-matrix product in the Kronecker format takes the form (A B)(C D) = (AC) (BD). The corresponding extension to q-th order tensors is (A 1... A q )(B 1... B q ) = (A 1 B 1 )... (A q B q ).

23 Properties of the Kronecker product B. Khoromskij, Leipzig 2007(L3) We have the distributive law (A + B) (C + D) = A C + A D + B C + B D. 4. Rank relation: rank(a B) = rank(a)rank(b). Ex In general A B B A. What is the condition on A and B that provides A B = B A? Invariance of some matrix properties: (1) If A and B are diagonal then A B is also diagonal, and conversely (if A B 0). (2) Let A and B be Hermitian/normal matrices (A = A resp. A 1 = A). Then A B is of the corresponding type. (3) A R n n, B R m m det(a B) = (deta) n (detb) m.

24 Kronecker product: matrix operations B. Khoromskij, Leipzig 2007(L3) 24 Thm Let A R n n and B R m m be invertible matrices. Then (A B) 1 = A 1 B 1. Proof. Since det(a) 0, det(b) 0 and the above property (3) we have det(a B) 0. Thus (A B) 1 exists and (A 1 B 1 )(A B) = (A 1 A) (B 1 B) = I nm. Lem Let A R n n and B R m m be unitary matrices. Then A B is a unitary matrix. Proof. Since A = A 1, B = B 1 we have (A B) = A B = A 1 B 1 = (A B) 1.

25 Kronecker product: matrix operations B. Khoromskij, Leipzig 2007(L3) 25 Define the commutator [A, B] := AB BA. Lem Let A R n n and B R m m. Then [A I n, I m B] = 0 R m2 n 2. Proof. [A I n, I m B] = (A I n )(I m B) (I m B)(A I n ) = A B A B = 0. Rem Let A, B R n n, C, D R m m and [A, B] = 0, [C, D] = 0. Then [A C, B D] = 0. Proof. Apply the identity (A B)(C D) = (AC) (BD).