Tensor-Product Representation of Operators and Functions (7 introductory lectures) Boris N. Khoromskij

Size: px

Start display at page:

Download "Tensor-Product Representation of Operators and Functions (7 introductory lectures) Boris N. Khoromskij"

Denis Walsh
5 years ago
Views:

1 1 Everything should be made as simple as possible, but not simpler. A. Einstein ( ) Tensor-Product Representation of Operators and Functions (7 introductory lectures) Boris N. Khoromskij University of Leipzig/MPI MIS, winter sem. 2006/2007

2 Outline of the Lecture Course B. Khoromskij, Leipzig 2006(L1) 2 1. (A) Ubiquitous data-sparse matrix/tensor arithmetics. (B) Separable approximation of multi-variate functions in R d. 2. Sinc interpolation and quadratures; Celebrated sampling theorem; Fourier kingdom. 3. Tucker/canonical decomposition of high-order tensors: Formatted multi-linear algebra, approximation theory, numerical methods (there is still much to be understood!). 4. Kronecker-product representation to multi-dimensional integral operators Au = R R d g(, y)u(y)dy. 5. Structured representation to matrix-valued functions A 1, A α. 6. Applicability to the Hartree-Fock/Kohn-Sham and Ornstein-Zernicke equations.

3 ect. 1. (A) Ubiquitous data-sparse matrix/tensor arithmetics B. Khoromskij, Leipzig Basic physical models are described by nonlocal data transforms. Examples: 1. Multi-dimensional integral operators in R d (convolution, Fourier and Laplace transforms) 2. Elliptic/parabolic solution operators (Green s functions) 3. Density matrix calculation for many-particle systems (Hartree-Fock and Kohn-Sham equations in R 3 ) 4. Convolution and functional transforms from the Ornstein-Zernike equation in R 3 (theory of disordered matter) 5. Collision integrals from the deterministic Boltzmann equation in R 3 (dilute gas). 6. Multi-dimensional data in chemometric, psychometric, higher-order statistics, financial math.,...

4 Nonlocal operators in wide range applications B. Khoromskij, Leipzig 2006(L1) 4 Functions and integral oper. (e.g., convolution) in R d : A R nd n d R n2... n 2 A 1, A α, α > 0, exp( ta), sign(a). Objectives in many-particle models via Hartree-Fock eq.:» 1 2 V c(x) + ρ(x, y) = NP e /2 i=1 Z R 3 dy ρ(y, y) φ(x) 1 Z dy x y 2 R 3 φ i (x)φ i (y) electron density matrix, e µ x - density function for hydrogen atom, 1 x ρ(x, y) x y Hartree-Fock-Slater equation» 12 Z + V (x) + ρ(y) x y dy αv ρ(x) ψ = λψ, V ρ (x) = R 3 φ(y) = λφ(y), - Newton potential. j ff 3 1/3 π ρ(x) Ornstein-Zernike integral-algebraic eq. in R 3 (molecular density) Z h(r) = c(r)+ρ c( r r )h( r )dr, h(r) = exp[ βu(r)+h(r) c(r)] 1 R 3 Boltzmann integral-differential eq. in R 3 (dilute gas).

5 Breaking down the complexity B. Khoromskij, Leipzig 2006(L1) 5 Approximate multi-variate functions/multi-dimensional operators avoiding the curse of dimensionality Goal: Solving basic eq. with O(nd log q n) cost instead of O(n d ). Structured tensor decomposition in R d : Orthogonal rank-(r 1,..., r d ) Tucker model, T r Canonical (CP) approximation, C r Two-level rank-(r 1,..., r d ; q) and mixed models Approximation tools: Sinc interpolation/quadratures; exponential fitting. Greedy algorithms. Direct minimisation of the cost functional. Truncated Newton-Schulz iteration. Numerics: 1 x y, e x y γ, e x y x y, cos x y x y, c k e x y k.

6 Huge problems: special methods vs. super-computers B. Khoromskij, Leipzig 2006(L1) 6 The algebraic operations on high-dimensional, densely populated tensors require huge computational resources; cf. linear complexity O(N) with N = n d. Standard asymptotically optimal methods suffer from the curse of dimensionality (R. Bellman). Complexity of matrix operations in full arithmetics: N Stor N A v = O(N 2 ); N A 1 N A B N L U = O(N 3 ); N EV D N SV D = O(N 3 ). A paradigm of up-to-date numerical simulations: the faster the computer is the better asymptotical complexity of fast algorithms is required (speed increases proportional to memory).

7 Large problems in low dimensions B. Khoromskij, Leipzig 2006(L1) 7 In low dimensions (d 3) the goal is O(N)-methods. Basic principles: making use of hierarchical structures, low-rank pattern and recursive algorithms. Based on recursions via hierarchical structures: Classical Fourier ( ) methods, FFT in O(N log N) op. FFT-based circulant convolution, Toeplitz, Hankel matrices. Multiresolution representation via wavelets, FWT in O(N) op. Multigrid methods: O(N) - elliptic problem solvers. Domain decomposition: O(N/p) - parallel algorithms. Fast multipole, panel clustering, H-matrix in O(c d N log β N) op. Well suited for integral (nonlocal) operators in FEM/BEM.

8 Old and new ideas or what we are going to discuss B. Khoromskij, Leipzig 2006(L1) 8 In multi-dimensional perspective O(N)-complex. is not enough since exponential scaling in d: N = n d In the Schrödinger eq. d = 3N e ( mol. in 1 cm 3 of water). The challenge is to develop O(dn)-algorithms! Main ideas: tensor-product data formats + structured representation of low-dimensional components. Based on tensor-product data organization: Kronecker tensor-product (KT) representation in R N, N = n d (multiway decomposition): O(dn q log β n), q q(d) - fixed. Effective multi-linear algebra. Combination of KT formats with H-matrix, wavelet or FFT-based structures: O(dn log β n) op.

9 Alternative directions: Different compression strategies B. Khoromskij, Leipzig 2006(L1) 9 High order methods: hp-fem/bem, spectral methods, bcfem (Khoromskij, Melenk), Richardson extrapolation. Adaptive mesh refinement: a priori/a posteriori strateg. Dimension reduction: boundary/interface equations, Schur complement/domain decomposition methods. Combination of tensor-product basis with anisotropic adaptivity: hyperbolic cross approximation by FEM/wavelet (sparse grids). Model reduction: multi-scale, homogenization, neural networks. Monte-Carlo meth. (e.g., random walk dynamics, stochastic PDEs).

10 (B) Separabe approximation of functions B. Khoromskij, Leipzig 2006(L1) 10 Rank-1 approximation of a multi-variate function f = f(x 1,..., x d ) in the set of separable functions M 1 = {u : u(x) = φ (1) (x 1 )... φ (d) (x d ), φ (l) H}. (1) f is from a certain class H (say, H = L 2 (R d )). H is a real, separable Hilbert space of functions defined on R (say, H = L 2 (R)). Advantages: It poses tramendous reduction of comput. cost, removing d from the exponential, n d d n. dth order tensors can be interpreted as functions of d discrete arguments (multi-dimensional arrays), f : R n 1... n d R.

11 Tucker model B. Khoromskij, Leipzig 2006(L1) 11 Def. 1. (Tucker model). Rank-r Tucker approximation via a linear combination of separable products, M r = {u : u(x) = r k=1 b k φ (1) k 1 (x 1 )... φ (d) k d (x d ), b k R, φ (l) k H}, with k = (k 1,..., k d ), 1 k l r l, and r = (r 1,..., r d ), r l N. The set of coeff. B = {b k } R r 1... r d is called the core tensor. Storage cost: r d + rdn. Maximal canonical rank: r d 1. Assume φ (l) k l to be orthonormal, i.e., (φ (l) k l, φ (l) m l ) = δ kl,m l, k l, m l = 1,..., r l, l = 1,..., d. V l H r l (l = 1,..., d) is the set of r l -tuples Φ (l) = (φ (l) 1,..., φ(l) r l ) with orthonormal components.

12 Remarks on the Schrödinger eq. B. Khoromskij, Leipzig 2006(L1) 12 In the context of the Schrödinger eq., a separable function u(x) = φ (1) (x 1 )... φ (d) (x d ) is called Hartree product, while φ (l) k l (x l ) are known as single-particle functions. The time-dependent solution of the Schrödinger eq. in molecular dynamics is approximated (for a fixed time) by a linear combination of Hartree products from the set M r. Due to the Pauli principle, approximations of M-electron systems are built from anti-symmetrised products of single-particle functions (Slater determinants), M S := {u : u(x) = det(φ (i) (x j )) M i,j=1, φ (i) L 2 }.

13 Canonical decomposition B. Khoromskij, Leipzig 2006(L1) 13 Def. 2. (Canonical model). Approximation in the set M r = {u : u(x) = r k=1 b k φ (1) k (x 1)... φ (d) k (x d), φ (l) k H} M r, with b k R and with normalised components φ (l) k = 1 is the special case of the Ticker approximation in M r, r = (r,..., r), under the constraint: all off-diagonal elements of B = {b k } are zero. Since M r is not a linear space, we obtain a difficult nonlinear approximation problem on estimation f H : σ(f, S) := inf f s, (2) s S where either S = M r or S = M r, or some subspaces S M r.

14 Computing the Tucker decomposition B. Khoromskij, Leipzig 2006(L1) 14 Physical interpretation of the Tucker model is not easy since nonuniqueness in b k and φ (l) k l : the rotation transform φ (l) k l φ (l) k l := r l m l =1 S (l) k l,m l φ (l) m l, b k b k := r 1 r d (S (1) ) T k 1,m 1 (S (d) ) T k d,m d b k m 1 =1 m d =1 defines the same { u for any choice of orthogonal r l r l matrices S (l) = S (l) k l,m l }, l = 1,..., d. Not a problem from the computational point of view: The minimisation problem (2) is equivalent to the dual maximisation problem on V l (l = 1,..., d), not including b k.

15 Computing the Tucker decomposition B. Khoromskij, Leipzig 2006(L1) 15 Lem. 1. Assume that there exists a minimiser of the problem (2). Then, for given Φ (l) = (φ (l) 1,..., φ(l) r l ) V l (l = 1,..., d), the core tensor b k, minimising (2) is represented by ( ) b k = f, φ (1) k 1 ( )... φ (d) k d ( ), k = (k 1,..., k d ). (3) For given f H, the minimisation problem (2) with S = M r, is equivalent to the maximisation problem σ(f; {V l } d l=1 ) := Proof. Let f (r) = k problem (2), then sup Φ (l) V l, l=1,...,d X k f(x 1,..., x d )φ (1) k 1 (x 1 )... φ (d) k (x d d ) b k φ (1) k 1 (x 1 )... φ (d) k d (x d ) be the solution of f (r) = B F := b 2 k, since orthonormal components do not effect the L 2 -norm. k 2.

16 Computing the Tucker decomposition B. Khoromskij, Leipzig 2006(L1) 16 With fixed components Φ (l) (l = 1,..., d), relation (2) is actually a linear least-squares problem w.r.t. b k, (f, f) 2(f, k b k φ (1) k 1 (x 1 )... φ (d) k d (x d )) + (B, B) min. Solving the corresponding Lagrange equation (f, k δb k φ (1) k 1 (x 1 )... φ (d) k d (x d ))+(B, δb) = 0 for all δb R r 1... r d, implies (3) and then f f (r) 2 = f 2 B 2 F. Then substitution of (3) proves the assertion.

17 Computing canonical decomposition B. Khoromskij, Leipzig 2006(L1) 17 For S = M r, canonical decomposition can be considered in the framework of best r-term approximation with regard to a redundant dictionary. Def. 3. A system D of functions from H is called a dictionary if each g D has norm one and its linear span is dense in H. Denote by Σ r (D) the collection of s H which can be written in the form s = c g g, Λ D, #Λ r N with c g R. g Λ For f H, the best r-term approximation error is defined by σ r (f, D) := inf f s. s Σ r (D)

18 Pure Greedy Algorithm B. Khoromskij, Leipzig 2006(L1) 18 The Pure Greedy Algorithm (PGA) inductively computes an estimate to the best r-term approximation. Let g = g(f) D be an element maximising (f, g). Define G(f) G(f, D) := (f, g)g, R(f) R(f, D) := f G(f). The PGA reads as: Given f H, introduce R 0 (f) := f and G 0 (f) := 0. Then, for all 1 m r, we inductively define G m (f) := G m 1 (f) + G(R m 1 (f)), R m (f) := f G m (f) = R(R m 1 (f)).

19 Pure Greedy Algorithm B. Khoromskij, Leipzig 2006(L1) 19 PGA applied to functions characterised via the approximation property (low order approximation) σ r (f, D) r q, r = 1, 2,..., with some q (0, 1/2], leads to the error bound (Temlyakov) f G r (f, D) C(q, D)r q, r = 1, 2,..., which is too pessimistic in our applications. Our goal: An efficient r-term approximation to analytic funct. with point singularities, allowing exponential convergence σ r (f, D) C exp( r q ), r = 1, 2,..., with q = 1 or q = 1/2. We will discuss quadrature- and interpolation-based methods as well as the direct approximation by exponential sums.

20 Special cases of PGA B. Khoromskij, Leipzig 2006(L1) 20 The output of PGA, G r (f, D), is proven to realise the best r-term approximation in the particular case when D is an orthogonal basis of H. Results of such kind can be generalised to the case of λ-quasiorthogonal dictionaries (Temlyakov). For the approximation problem on M r we set D := {g H M 1, g = 1}, and hence Σ r (D) = M r. The assumption that the components {φ (l) k } (l = 1,..., d) belong to an orthogonal basis of H implies the orthogonality requirement for D.

21 The case d = 2 B. Khoromskij, Leipzig 2006(L1) 21 The approximation of functions f(x, y) by bilinear forms r u k (x)v k (y) in L 2 ([0, 1]), k=1 was considerd by E. Schmidt in The result is an analogue to SVD for rectangular matrices. Let {s k (J f )} be a nonincreasing sequence of SVs of the IO i.e., s 1 s , J f (g) := 1 0 f(x, y)g(y)dy, s k (J f ) := λ k (A) 1/2, A = J f J f, J f adjoint to J f with orthonormal sequences {ϕ k (x)}, {ψ k (y)}, Aψ k (y) = λ k ψ k (y); A ϕ k (x) = λ k ϕ k (x).

22 The case d = 2 B. Khoromskij, Leipzig 2006(L1) 22 The Schmidt expansion is given by f(x, y) = s k (J f )ϕ k (x)ψ k (y). k=1 The best bilinear approximation property was proven, rx f(x, y) s k ϕ k (x)ψ k (y) k=1 L 2 = inf u k,v k L 2, k=1,...,r rx f(x, y) u k (x)v k (y) Schmidt s expansion ensures that the best bilinear appr. can be realised by the PGA. The kernel function of A is given by k=1 L 2. f A (x, y) := 1 0 f(x, z)f(z, y)dz, hence, for Nyström s approx. the problem is reduced to SVD.

23 Orthogonal Decomposition with d 3 B. Khoromskij, Leipzig 2006(L1) 23 Let d 3. The PGA produces best nonlinear approximation in a situation with orthogonal components. We call the decomposition in M r, r f = a k v k, v k M 1, v k = 1, k=1 orthogonal if (v m, v k ) = 0 for all m k. Greedy orthogonal decomposition (GOD): Set R 0 (f) = f, G 0 (f) = 0 and define the pth residual tensor as p R p (f) := f b k u k, u k M 1, p = 1, 2,... k=1 On each recurrent step, we find the best 1-term approximation to R p (f) under orthogonality constraints,

24 Orthogonal Decomposition with d 3 B. Khoromskij, Leipzig 2006(L1) 24 min G p(b, u) := R p 1 (f) bu 2 with u U p 1, u M 1, u =1 where U p = {u 1,..., u p } with U 0 =. Since G p (b, u) = R p 1 2 2b(R p 1, u) + b 2 u 2, we solve the Lagrange eq. for b, G p b = 2(R p 1, u) + 2b u 2 = 0 b = (R p 1, u) = (f, u), to obtain u p as a solution of max u M 1, u =1 (f, u) with u U p 1 (challenging problem!). Finally, let b p = (f, u p ).

25 Greedy completely orthogonal decomposition B. Khoromskij, Leipzig 2006(L1) 25 The decomposition in M r, f = r k=1 a k v k, v k = φ (1) k (x 1)... φ (d) k (x d) M 1, is called completely orthogonal if (φ (l) k, φ(l) m ) = δ k,m l = 1,..., d, Φ (l) V l. Greedy completely orthogonal decomposition is defined as GOD with additional orthogonality constraint Φ (l) V l. Lem. 2. Let f H allow a rank-r completely orthogonal decomposition. Then this decomposition is unique, and the GCOD algorithm correctly computes it. Proof. Uniqueness id due to orthogonality assumption.

26 Greedy completely orthogonal decomposition B. Khoromskij, Leipzig 2006(L1) 26 The GCOD reduces to solving (for u p ) max (f, u) u M 1, u =1 with Φ(l) p V l (simple problem), and letting b p = (f, u p ). For example, with p = 1 and for we have r k=1 a k u 1 = d c k,l max l=1 d ( l=1 r k=1 with c k,l φ (l) k (x l)), r c 2 k,l = 1, l = 1,..., d. k=1 Assuming a 1 a 2... a r > 0, we obtain c 1,l = 1, c 2,l =... = c r,l = 0 u 1 = v 1 (Hint: use symmetry in l). This ensures b 1 = (f, u 1 ) = a 1. Hence we prove inductively p u p = a k v k. k=1

27 Literature to Lecture 1 B. Khoromskij, Leipzig 2006(L1) B.N. Khoromskij: An introduction to Structured Tensor-product representation of Discrete Nonlocal Operators. Lecture notes 27, MPI MIS, Leipzig W. Hackbusch and B.N. Khoromskij: Tensor-product Approximation to Operators and Functions in High Dimensions. Preprint 139, MPI MIS, Leipzig V.N. Temlyakov: Greedy Algorithms and M-Term Approximation with Regard to Redundant Dictionaries. J. of Approx. Theory 98 (1999), URL:

1. Structured representation of high-order tensors revisited. 2. Multi-linear algebra (MLA) with Kronecker-product data.

Lect. 4. Toward MLA in tensor-product formats B. Khoromskij, Leipzig 2007(L4) 1 Contents of Lecture 4 1. Structured representation of high-order tensors revisited. - Tucker model. - Canonical (PARAFAC)