Krylov subspace methods for linear systems with tensor product structure

Transcription

1 Krylov subspace methods for linear systems with tensor product structure Christine Tobler Seminar for Applied Mathematics, ETH Zürich 19. August 2009

2 Outline 1 Introduction 2 Basic Algorithm 3 Convergence bounds 4 Solving the compressed equation 5 Numerical experiments

3 Outline 1 Introduction 2 Basic Algorithm 3 Convergence bounds 4 Solving the compressed equation 5 Numerical experiments

4 Linear system with tensor product structure We consider the linear system Ax = b with d A = I n1 I ns 1 A s I ns+1 I nd, s=1 b = b 1 b d, A s R ns ns positive definite, b s R ns. Example for 3 dimensions: (A 1 I I + I A 2 I + I I A 3 )x = b 1 b 2 b 3

5 Linear system with tensor product structure We consider the linear system Ax = b with d A = I n1 I ns 1 A s I ns+1 I nd, s=1 b = b 1 b d, A s R ns ns positive definite, b s R ns. Example for 3 dimensions: (A 1 I I + I A 2 I + I I A 3 )x = b 1 b 2 b 3

6 Tensor decompositions CP decomposition: Tucker decomposition: k vec(x ) = a r b r c r r=1 a r R m, b r R n, c r R p vec(x ) = m ñ p i=1 s=1 l=1 G ijl a i b j c l =(A B C) vec(g) X c1 a1 b1 ck ak bk G R m ñ p, a i R m, b j R n, c l R p B C X A G

7 About this system The eigenvalues of the matrix A are given by all possible sums λ (1) i 1 + λ (2) i λ (d) i d where λ (s) i s denotes an eigenvalue of A s. For A s positive definite, the system has a unique solution. Note that x and b are vector representations of tensors in R n 1 n d, and b represents a rank-one tensor. A tensor arising from the discretization of a sufficiently smooth function f can be approximated by a short sum of rank-one tensors. By superposition, we can insert such a tensor as right-hand side into our system.

8 About this system The eigenvalues of the matrix A are given by all possible sums λ (1) i 1 + λ (2) i λ (d) i d where λ (s) i s denotes an eigenvalue of A s. For A s positive definite, the system has a unique solution. Note that x and b are vector representations of tensors in R n 1 n d, and b represents a rank-one tensor. A tensor arising from the discretization of a sufficiently smooth function f can be approximated by a short sum of rank-one tensors. By superposition, we can insert such a tensor as right-hand side into our system.

9 About this system The eigenvalues of the matrix A are given by all possible sums λ (1) i 1 + λ (2) i λ (d) i d where λ (s) i s denotes an eigenvalue of A s. For A s positive definite, the system has a unique solution. Note that x and b are vector representations of tensors in R n 1 n d, and b represents a rank-one tensor. A tensor arising from the discretization of a sufficiently smooth function f can be approximated by a short sum of rank-one tensors. By superposition, we can insert such a tensor as right-hand side into our system.

10 Tensorized Krylov subspaces A Krylov subspace is defined as K k (A, b) = span { b, Ab,..., A k 1 b }. We define a tensorized Krylov subspace as K K (A, b) := span( K k1 (A 1, b 1 ) K kd (A d, b d ) ). Note that K k0 (A, b) K K (A, b) for K = (k 0,..., k 0 ). Tensorized Krylov subspaces are richer than standard Krylov subspaces.

11 Tensorized Krylov subspaces A Krylov subspace is defined as K k (A, b) = span { b, Ab,..., A k 1 b }. We define a tensorized Krylov subspace as K K (A, b) := span( K k1 (A 1, b 1 ) K kd (A d, b d ) ). Note that K k0 (A, b) K K (A, b) for K = (k 0,..., k 0 ). Tensorized Krylov subspaces are richer than standard Krylov subspaces.

12 Tensorized Krylov subspaces A Krylov subspace is defined as K k (A, b) = span { b, Ab,..., A k 1 b }. We define a tensorized Krylov subspace as K K (A, b) := span( K k1 (A 1, b 1 ) K kd (A d, b d ) ). Note that K k0 (A, b) K K (A, b) for K = (k 0,..., k 0 ). Tensorized Krylov subspaces are richer than standard Krylov subspaces.

13 Outline 1 Introduction 2 Basic Algorithm 3 Convergence bounds 4 Solving the compressed equation 5 Numerical experiments

14 Reminder: CG method (Ax = b) The best approximation of x in K k (A, b) is defined by: x k x A = min x x A. x K k (A,b) Find U k with (orthonormal) columns that span the Krylov subspace K k (A, b). Set x k = U k y, then y is the solution of the compressed system H k y = b, H k = U k AU k, b = U k b. Convergence bound (κ = κ 2 (A)) ( κ 1 ) k. x k x A C(A, b, κ) κ + 1

15 Reminder: CG method (Ax = b) The best approximation of x in K k (A, b) is defined by: x k x A = min x x A. x K k (A,b) Find U k with (orthonormal) columns that span the Krylov subspace K k (A, b). Set x k = U k y, then y is the solution of the compressed system H k y = b, H k = U k AU k, b = U k b. Convergence bound (κ = κ 2 (A)) ( κ 1 ) k. x k x A C(A, b, κ) κ + 1

16 Reminder: CG method (Ax = b) The best approximation of x in K k (A, b) is defined by: x k x A = min x x A. x K k (A,b) Find U k with (orthonormal) columns that span the Krylov subspace K k (A, b). Set x k = U k y, then y is the solution of the compressed system H k y = b, H k = U k AU k, b = U k b. Convergence bound (κ = κ 2 (A)) ( κ 1 ) k. x k x A C(A, b, κ) κ + 1

17 Tensorized Krylov: A vec(x ) = b Applying the Arnoldi method to A s, b s results in U s, H s with U s A s U s = H s, U s column-orthogonal, H s upper Hessenberg matrix. The columns of U s span K ks (A s, b s ), and similarly the columns of U := U 1 U d span K K (A, b). Solve the compressed system Hy = b with x K = Uy, b = U b and H = U AU.

18 Tensorized Krylov: A vec(x ) = b Applying the Arnoldi method to A s, b s results in U s, H s with U s A s U s = H s, U s column-orthogonal, H s upper Hessenberg matrix. The columns of U s span K ks (A s, b s ), and similarly the columns of U := U 1 U d span K K (A, b). Solve the compressed system Hy = b with x K = Uy, b = U b and H = U AU.

19 Tensorized Krylov: A vec(x ) = b Applying the Arnoldi method to A s, b s results in U s, H s with U s A s U s = H s, U s column-orthogonal, H s upper Hessenberg matrix. The columns of U s span K ks (A s, b s ), and similarly the columns of U := U 1 U d span K K (A, b). Solve the compressed system Hy = b with x K = Uy, b = U b and H = U AU.

20 Compressed system The structure of Hy = b corresponds to that of Ax = b: H = d I k1 I ks 1 H s I ks+1 I kd s=1 b = b 1 b d = b 2 (e 1 e 1 ) For dense core tensor y, x K is a tensor in Tucker decomposition: x K = (U 1 U d )y

21 Outline 1 Introduction 2 Basic Algorithm 3 Convergence bounds 4 Solving the compressed equation 5 Numerical experiments

22 Convergence, s.p.d. case (1) Similarly to CG, we have: x K x A = min x K K (A,b) x x A Every vector in a Krylov subspace K ks (A s, b s ) can be seen as p(a s )b s, with p a polynomial of order at most k s 1. A similar thought reduces the bound calculation to the min-max problem 1 E Ω (K) := min p(λ 1,..., λ d ) Ω, p Π λ K λ d where Π K is a space of multivariate polynomials, and Ω contains the eigenvalues (λ 1,..., λ d ), with λ i [λ min (A i ), λ max (A i )].

23 Convergence, s.p.d. case (2) Inserting an upper bound for E Ω (K), we find x K x A d s=1 with κ s = 1 + λmax(as) λ min(a s) λ min (A). For the case A s, k s constant: ( κs 1 ks, C(A, b, κ s ) κs + 1) ( κ 1 ) k, x K x A C(A, b, d) κ + 1 with κ = d 1 d + κ 2(A) d. Note that the convergence rate improves with increasing dimension.

24 Convergence, non-symmetric positive definite case General convergence bound: x K x 2 d s=1 0 with ˆα s := j s α j and α j = λ min (A j + A j )/2. e ˆαst U s e ths e 1 e tas b s 2 dt The bound on U s e ths e 1 e tas b s 2 will depend on additional knowledge on A s. For example, when the field of values of each matrix A s is contained in a known ellipse, an explicit convergence bound can be found.

25 Outline 1 Introduction 2 Basic Algorithm 3 Convergence bounds 4 Solving the compressed equation 5 Numerical experiments

26 Grasedyck s method, system Hy = b For H positive definite: H 1 = 0 exp( th)dt The exponential of H has a Kronecker product structure, too: exp( th) = exp( t Approximation y m : d d Ĥ s ) = exp( tĥs) = s=0 s=1 s=1 d exp( th s ), y m = m j=1 ω j d s=1 ( exp α j H s ) bs, with certain coefficients α j, ω j.

27 Coefficients of the exponential sum (1) The coefficients α j, ω j should minimize sup 1 m z Λ(H) z ω j e α j z Case 1: H symmetric and condition number known The eigenvalues of H are real: Λ(H) [λ min, λ max ]. There are coefficients α j, ω j s.t. j=1 y y m 2 C(H, b) ( exp mπ 2 log(8κ 2 (H)) These coefficients may be found using a variant of the Remez algorithm. Tabellated values have been made available by Hackbusch. ).

28 Coefficients of the exponential sum (2) Case 2: Nonsymmetric H and/or unknown condition number There is an explicit formula for α j, ω j, where the eigenvalues of H only need to have positive real part. y y 2m+1 2 C(H, b) exp(µ/π) exp( m), where µ = max{ Im(Λ(H)) }. The convergence is significantly slower than for Case 1.

29 Theoretical convergence bounds for the coefficients Case 1 for cond. 1e5 Case 1 for cond. 1e15 Case 2 y y k k

30 Outline 1 Introduction 2 Basic Algorithm 3 Convergence bounds 4 Solving the compressed equation 5 Numerical experiments

31 Symmetric example: Poisson Equation Finite difference discretization of the Poisson equation in d dimensions: u = f u = 0 in Ω = [0, 1] d on Γ := Ω, where the right-hand side f is a separable function. 2 1 A s = h , b s : random numbers. 1 2 The approximation error is measured by relative residual, Ax K b 2 b 2.

32 Convergence for system size 200 d relative residual Numerical convergence d = 5 d = 10 d = 50 d = 100 relative residual Theoretical convergence rate d = 5 d = 10 d = 50 d = k k

33 Convergence for system size 1000 d relative residual Numerical convergence d = 5 d = 10 d = 50 d = 100 relative residual Theoretical convergence rate d = 5 d = 10 d = 50 d = k k

34 Extended Krylov subspaces Using extended Krylov subspaces K ks (A s, b s ) := span(k ks (A s, b s ) K ks+1(a 1 s, b s )), the algorithm works analogously. Convergence bound (A s, k s constant): ( κ 1 ) k. x K x 2 C(A, b, d) κ + 1 The convergence rate depends on κ: κ d 1 d + 1 d d 1 (d 1) d κ 2 2 (A) d 1 d d = 2 : κ = κ2 (A), d 0 : κ d 1 4 d + κ 2(A). d

35 Extended Krylov, system size 200 d relative residual Numerical convergence d = 5 d = 10 d = 50 d = 100 relative residual Theoretical convergence rate d = 5 d = 10 d = 50 d = k k

36 Non-symmetric example Finite difference discretization in d dimensions of the convection-diffusion equation u (c,..., c) u = f u = 0 in Ω = [0, 1] d on Γ := Ω, where f is again a separable function. A s = 1 h c 4h For c = 10, all eigenvalues of A are real. For c = 100, this is not the case, and the convergence is significantly worsened.

37 Non-symmetric case, system size 200 d relative residual Convergence for c = 10 d = 5 d = 10 d = 50 d = 100 relative residual Convergence for c = d = k k

38 Conclusions An efficient algorithm to calculate a low-rank approximation of the solution tensor The computational complexity is linear in the number of dimensions Only matrix-vector operations with the full system matrices are required A theoretical convergence bound was found.

39 Conclusions An efficient algorithm to calculate a low-rank approximation of the solution tensor The computational complexity is linear in the number of dimensions Only matrix-vector operations with the full system matrices are required A theoretical convergence bound was found. Thank you for your attention!

40 Literature D. Kressner, C. Tobler, Krylov subspace methods for linear systems with tensor product structure, Research report No , SAM, ETH Zurich, L. Grasedyck, Existence and computation of low Kronecker-rank approximations for large linear systems of tensor product structure, Computing, 72(3-4): , (2004). D. Braess, W. Hackbusch, Approximation of 1/x by exponential sums in [1, ), IMA J. Numer. Anal., 25(4): , (2005).