Introduction and Stationary Iterative Methods

Transcription

1 Introduction and C. T. Kelley NC State University tim Research Supported by NSF, DOE, ARO, USACE DTU ITMAN, 2011

2 Outline Notation and Preliminaries General References What you Should Know Convergence and the Banach Lemma Matrix Splittings and Classical Methods References for

3 General References What you Should Know General References C. T. Kelley, Iterative Methods for Linear and Nonlinear Equations, no. 16 in Frontiers in Applied Mathematics, SIAM, Philadelphia, J. W. Demmel, Applied Numerical Linear Algebra, SIAM, Philadelphia, G. H. Golub and C. G. VanLoan, Matrix Computations, Johns Hopkins studies in the mathematical sciences, Johns Hopkins University Press, Baltimore, 3 ed., E. Isaacson and H. B. Keller, Analysis of numerical methods, Wiley, New York, G. W. Stewart, Introduction to matrix computations, Academic Press, New York, 1973.

4 General References What you Should Know Background I assume you have had courses in Numerical methods (Gaussian elimination, SVD, QR,... ) Linear Algebra (Vector spaces, norms, inner products,... ) Calculus and differential equations Some functional analysis would help. If at any time I use something you are not familiar with, stop me and I will review.

5 General References What you Should Know Other things you should know I have never done this before. I may have too much or too little material. Some of the codes I will give you are new. So they may have a few bugs. I ve set too many exercises. You ll have to be selective or stay up late.

6 General References What you Should Know What s in Monday s Directory A copy of today s lectures in pdf. A matlab code gauss.m, which you ll need for one of the exercises. A copy of a paper that may help.

7 Vectors Notation and Preliminaries General References What you Should Know All vectors are column vectors of dimension N. ξ 1 η 1 ξ 2 η 2 x =. ξ N, y =. η N Components of vectors use Greek letters. Scalar product N x T y = ξ i η i, where x T is the row vector i=1 x T = (ξ 1,..., ξ N )., RN

8 Matrices Notation and Preliminaries General References What you Should Know Matrices are in upper case, columns in lower case. a a 1M a a 2M A =..... = (a 1, a 2,..., a M ) a N1... a NM is an M N matrix.

9 Transpose Notation and Preliminaries General References What you Should Know A T = a a 1N a a 2N..... a M1... a MN is an N M matrix. This is consistent with x T being a row vector. We mostly do real arithmetic. In complex arithmetic we use A #, the complex conjugate transpose, in place of A T.

10 Linear Equations Notation and Preliminaries General References What you Should Know Ax = b explicitly a 11 ξ a 1j ξ j + + a 1N ξ N = b 1. a i1 ξ i + + a ij ξ j + + a in ξ N = b i. a N1 ξ a Nj ξ j + + a NN ξ N = b N Unless we explicitly say otherwise, A is nonsingular.

11 Vector Notation and Preliminaries General References What you Should Know Our vector norms will be the l p norms 1/p N x p = ξ j p (1 p < ) and x = max ξ j 1 j N j=1 The l 2 norm is connected with the scalar product x T x = x 2 2.

12 Matrix Norms Notation and Preliminaries General References What you Should Know Let be a norm on R N. The Induced Matrix Norm of an N N matrix A is defined by A = max x =1 Ax. We use induced norms. They have the important property: Ax A x, which implies that if Ax = b then A 1 1 x b A x.

13 General References What you Should Know Types of linear equations Dense: A has very few non-zero entries. Sparse: A has many zeros. Structured: A has structure which algorithms can use For example: sparsity, symmetry (A = A T ), connection to differential or integral equations. Unstructured: One must use general methods.

14 General References What you Should Know Structure Sparsity Symmetry A T = A A is symmetric positive definite (spd) if A = A T and x T Ax > 0 for all x 0. In this case x A = (x T Ax) 1/2 is a vector norm. Normality A T A = AA T Diagonalizability A = V ΛV 1 where Λ is diagonal. A is diagonalizable if and only if A has N linearly independent eigenvectors and then V = (v 1,... v n ). If A is symmetric then V is orthogonal, VV T = V T V = I. If A is normal then V is unitary, VV # = V # V = I.

15 General References What you Should Know Direct and Iterative Methods Direct Methods solve Ax = b in finite time in exact arithmetic. Examples: Gaussian Elimination and other matrix factorizations FFT for some problems (Toeplitz, Hankel, PDEs) Iterative Methods produce a sequence {x n } which (you hope) converges to x = A 1 x. Examples: Stationary iterative methods (L1 and L3) Krylov methods (L2, 3, 4) Multigrid methods (L3)

16 Condition Numbers Notation and Preliminaries General References What you Should Know The Condition Number of A relative to the norm is κ(a) = A A 1, where κ(a) is understood to be infinite if A is singular. κ p (A) means relative to the l p norm. If A is poorly conditioned (say κ > 10 8 ) we may not be able to obtain an accurate solution with any choice of algorithm. Poor conditioning is a property of A. Algorithms cannot help.

17 Termination of Iterations General References What you Should Know Most iterative methods terminate when the Residual r = b Ax is sufficiently small. One termination criterion is r k r 0 < τ where r 0 = b Az 0 and z 0 is a reference value. So, what does a small relative residual tell us about the error e = x x?

18 Residuals, Errors, Conditioning General References What you Should Know Theorem: Let b, x, x 0 R N. Let A be nonsingular and let x = A 1 b. 1 r κ(a) r 0 e r κ(a) e 0 r 0. We prove this. Note that r = b Ax = Ae so and e = A 1 Ae A 1 Ae = A 1 r r = Ae A e.

19 General References What you Should Know So, and as asserted. e e 0 A 1 r r A 1 = κ(a) r 0 r 0 e e 0 A 1 r r A 1 = κ(a) 1 r 0 r 0

20 Application of the Theorem General References What you Should Know Most of the methods we use will set the reference vector to zero. Hence r 0 = b and e 0 = x = A 1 b and the theorem connects the Relative Residual to the Relative Error b Ax b x x x If A is poorly conditioned, then termination on small relative residuals may be unreliable.

21 Convergence and the Banach Lemma Matrix Splittings and Classical Methods A Stationary Iterative Method converts Ax = b to x = Mx + c and the iteration is x n+1 = Mx n + c M is called the iteration matrix. This iteration is also called Richardson Iteration. The method is called stationary because the formula does not change as a function of x n.

22 Banach Lemma Notation and Preliminaries Convergence and the Banach Lemma Matrix Splittings and Classical Methods Let M be N N. Assume that M < 1 for some induced matrix norm. Then (I M) is nonsingular (I M) 1 = l=0 Ml (I M) 1 (1 M ) 1

23 Proof of Banach Lemma: I Convergence and the Banach Lemma Matrix Splittings and Classical Methods We will show that the series M l = (I M) 1. l=0 The partial sums S k = form a Cauchy sequence in R N N. To see this note that for all m > k m S k S m M l. And... k l=0 M l l=k+1

24 Proof of Banach Lemma: II Convergence and the Banach Lemma Matrix Splittings and Classical Methods M l M l because is a matrix norm that is induced by a vector norm. Hence m ( ) 1 M S k S m M l = M k+1 m k 0 1 M l=k+1 as m, k. So the series converges. Let S = l=0 M l

25 Proof of Banach Lemma: II Convergence and the Banach Lemma Matrix Splittings and Classical Methods Clearly Finally MS = M l+1 = M l = S I and so l=0 l=1 (I M)S = I and S = (I M) 1. (I M) 1 M l = (1 M ) 1. l=0

26 Convergence and the Banach Lemma Matrix Splittings and Classical Methods Convergence for If M < 1 for any induced matrix norm then the stationary iteration x n+1 = Mx n + c converges for all c and x 0 to x = (I M) 1 c Proof: Clearly x n+1 = n M l c + M n x 0 (I M) 1 c = x. l=0

27 Convergence Speed Notation and Preliminaries Convergence and the Banach Lemma Matrix Splittings and Classical Methods Let M = α < 1 and x = (I M) 1 c. Then Proof: x x n α n x x 0. x x n = l=n Ml c M n x 0 = M( l=n 1 Ml c M n 1 x 0 ) = M(x x n 1 ) α x x n 1 α n x x 0.

28 Spectral Radius Notation and Preliminaries Convergence and the Banach Lemma Matrix Splittings and Classical Methods The spectrum of M σ(m), is the set of eigenvalues of M. The spectral radius of M is ρ(m) = max λ λ σ(m) Theorem ρ(m) < 1 if and only if M < 1 for some induced matrix norm. A stationary iterative method will x n+1 = Mx n + c converges for all initial iterates and right sides if and only if ρ(m) < 1. The spectral radius does not depend on any norm.

29 Predicting Convergence Convergence and the Banach Lemma Matrix Splittings and Classical Methods Suppose you know that M α < 1. Then e n+1 = x x n+1 = (Mx + c) (Mx n + c) = Me n Hence e n α n e 0 and e n τ e 0 if α n < τ or n > log(τ)/ log(α).

30 Preconditioned Richardson Iteration Convergence and the Banach Lemma Matrix Splittings and Classical Methods If I A < 1 then one can apply Richardson iteration directly to Ax = b x n+1 = (I A)x n + b Sometimes one can find a approximate inverse B for which I BA < 1 and precondition with B to obtain BAx = Bb and the iteration is x n+1 = (I BA)x n + Bb

31 Approximate Inverse Preconditioning: I Convergence and the Banach Lemma Matrix Splittings and Classical Methods Theorem: If A and B are N N matrices and B is an approximate inverse of A, then A and B are both nonsingular and and A 1 A 1 B B 1 I BA, B 1 B I BA 1 I BA, A B 1 A 1 I BA, A I BA 1 I BA.

32 Approximate Inverse Preconditioning: II Convergence and the Banach Lemma Matrix Splittings and Classical Methods Proof: Let M = I BA. The Banach Lemma implies that I M = I (I BA) = BA is nonsingular. Hence both A and B are nonsingular. Moreover A 1 B 1 = (I M) M = 1 1 I BA.

33 Convergence and the Banach Lemma Matrix Splittings and Classical Methods Approximate Inverse Preconditioning: III Use A 1 = (I M) 1 B to get the first part A 1 B (I M) 1 The second pair of inequalities follows from and the first. B 1 I BA. A 1 B = (I BA)A 1, A B 1 = B 1 (I BA)

34 Matrix Splittings Notation and Preliminaries Convergence and the Banach Lemma Matrix Splittings and Classical Methods One way to convert Ax = b to Mx = c is to split A where A 1 is nonsingular A = A 1 + A 2 A 1 y = q is easy to solve for all q and then solve x = A 1 1 (b A 2x) Mx + c. Here M = A 1 1 A 2 and c = A 1 1 b. Remember A 1 z means solve A 1 y = z, not compute A 1 1.

35 Jacobi Iteration: I Notation and Preliminaries Convergence and the Banach Lemma Matrix Splittings and Classical Methods Write Ax = b explicitly a 11 ξ a 1N ξ N = β 1. a N1 ξ a NN ξ N = β N and solve the ith equation for ξ i, pretending the other components are know. You get ξ i = 1 β i a ij ξ j a ii j i which is a linear fixed point problem equivalent to Ax = b.

36 Convergence and the Banach Lemma Matrix Splittings and Classical Methods Jacobi Iteration: II The iteration is ξ New i = 1 a ii β i j i a ij ξ Old j So what are M and c? Split A = A 1 + A 2, where A 1 = D, A 2 = L + U, D is the diagonal of A, and L and U are the (strict) lower and upper triangular parts. then x New = D 1 (b (L + U))x Old.

37 Jacobi Iteration: III Notation and Preliminaries Convergence and the Banach Lemma Matrix Splittings and Classical Methods So the iteration is x n+1 = D 1 (L + U)x n + D 1 b and the iteration matrix is M JAC = D 1 (L + U). Is there any reason for ρ(m JAC ) < 1?

38 Convergence and the Banach Lemma Matrix Splittings and Classical Methods Convergence for Strictly Diagonally Dominant A Theorem: Let A be an N N matrix and assume that A is strictly diagonally dominant. That is for all 1 i N 0 < j i a ij < a ii. Then A is nonsingular and the Jacobi iteration converges to x = A 1 b for all b.

39 Convergence and the Banach Lemma Matrix Splittings and Classical Methods Proof: Convergence for Strictly Diagonally Dominant A Our assumptions imply that a ii 0, so the iteration is defined. We can prove everything else showing that M JAC < 1. Remember that M JAC < 1 is the maximum absolute row sum. By assumptions, the ith row sum of M = M JAC satisfies That s it. N m ij = j=1 j i a ij < 1. a ii

40 Observations Notation and Preliminaries Convergence and the Banach Lemma Matrix Splittings and Classical Methods Convergence of Jacobi implies A is nonsingular. Showing M JAC < 1 for any norm would do. The l norm fit the assumptions the best. We have said nothing about the speed of convergence. Jacobi iteration does not depend on the ordering of the variables. Each ξi New can be processed independently of all the others. So Jacobi is easy to parallelize.

41 Gauss-Seidel Iteration Convergence and the Banach Lemma Matrix Splittings and Classical Methods Gauss-Seidel changes Jacobi by updating each entry as soon as the computation is done. So ξ New i = 1 β i a ii j<i a ij ξ New j j>i a ij ξ Old j You might think this is better, because the most up-to-date information is in the formula.

42 Gauss-Seidel Iteration Convergence and the Banach Lemma Matrix Splittings and Classical Methods One advantage of Gauss-Seidel is that you need only store one copy of x. This loop does the job with only one vector. for i=1:n do sum=0; for j i do sum=sum+a(i,j)*x(j) end for x(i) = (b(i) + sum)/a(i,i) end for

43 Gauss-Seidel Iteration Matrix Convergence and the Banach Lemma Matrix Splittings and Classical Methods From the formula, running for i = 1,... N. ξi New = 1 β i a ij ξj New a ij ξ Old j a ii j<i j>i you can see that so (D + U)x n+1 = b Lx n M GS = (D + U) 1 L and c = (D + U) 1 b.

44 Backwards Gauss-Seidel Convergence and the Banach Lemma Matrix Splittings and Classical Methods Gauss-Seidel depends on the ordering. Backwards Gauss-Seidel is ξ New i = 1 β i a ii j>i a ij ξ New j j<i a ij ξ Old j running from i = N, So M BGS = (D + L) 1 U.

45 Symmetric Gauss-Seidel Convergence and the Banach Lemma Matrix Splittings and Classical Methods A symmetric Gauss-Seidel iteration is a forward Gauss-Seidel iteration followed by a backward Gauss-Seidel iteration. This leads to the iteration matrix M SGS = M BGS M GS = (D + U) 1 L(D + L) 1 U. If A is symmetric then U = L T. In that event M SGS = (D + U) 1 L(D + L) 1 U = (D + L T ) 1 L(D + L) 1 L T.

46 SOR iteration Notation and Preliminaries Convergence and the Banach Lemma Matrix Splittings and Classical Methods Add a relaxation parameter ω to Gauss-Seidel. M SOR = (D + ωl) 1 ((1 ω)d ωu). Much better performance with good choice of ω.

47 Convergence and the Banach Lemma Matrix Splittings and Classical Methods Observations Gauss-Seidel and SOR depend on order of variables. So they are harder to parallelize. While they may perform better than simple Jacobi, it s not a lot better. These methods are not competitive with Krylov methods. They require the least amount of storage, and are still used for that reason.

48 Splitting Methods to Preconditioners Convergence and the Banach Lemma Matrix Splittings and Classical Methods Splitting methods can be seen as preconditioned Richardson iteration. You want to find the preconditioner B so that the iteration matrix from the splitting So I M = BA. M = A 1 1 A 2 = I BA.

49 Convergence and the Banach Lemma Matrix Splittings and Classical Methods Jacobi preconditioning For the Jacobi splitting A 1 = D, A 2 = L + U, we get D 1 (L + U) = I BA so BA = I + D 1 (L + U) = D 1 A Jacobi preconditioning is multiplication by D 1. This can be a surprisingly good preconditioner for Krylov methods.

50 References for References for P. Henrici, Discrete Variable Methods in Ordinary Differential Equations, Wiley, New York, R. J. LeVeque, Finite Difference Methods for Ordinary and Partial Differential Equations, SIAM, I. Stakgold, Green s Functions and Boundary Value Problems, Wiley-Interscience, New York, 1979.

51 in 1D References for One space dimension u (x) = f (x) for x (0, 1) Homogeneous Dirichlet boundary conditions u(0) = u(1) = 0 Eigenvalue Problem u (x) = λu(x) for x (0, 1), u(0) = u(1) = 0

52 Solution of References for We can diagonalize the operator using the solutions of the eigenvalue problem {u n } is an orthonormal basis for u n (x) = sin(nπx)/ 2, λ = n 2 π 2 L 2 0 = cl L 2{u C([0, 1]) u(0) = u(1) = 0} and the boundary value problem s solution is u(x) = 1 u n (x) n 2 π 2 n=1 1 1 u n (z)f (z) dz.

53 Properties of the Operator References for The operator d 2 dx 2 : C 2 0 ([0, 1]) C([0, 1]) is injective symmetric with respect to the L 2 scalar product has an L 2 0 orthonormal basis of eigenfunctions has positive eigenvalues

54 References for Solution Operator The solution of on [0, 1] with homogeneous Dirichlet boundary conditions is u(x) = 1 0 g(x, z)f (z) dz where x(1 z) 0 < x < z g(x, z) = z(1 x) z < x < 1

55 Central Difference Approximation References for Add u(x + h) = u(x) + u (x)h + u (x)h 2 /2 + u (h)h 3 /6 + O(h 4 ) to u(x h) = u(x) u (x)h + u (x)h 2 /2 u (h)h 3 /6 + O(h 4 ) and get u (x) = ( u(x + h) + 2u(x) u(x h))/h 2 + O(h 2 )

56 Finite Difference Equations References for Equally spaced grid x i = ih, 0 i N + 1, h = 1/(N + 1). Approximate u(x i ) by ξ i. Let u = (ξ 1,..., ξ N ) T. Boundary conditions imply that ξ 0 = ξ N+1 = 0. Finite difference equations at interior grid points are for 1 i N. ξ i 1 + 2ξ i ξ i+1 h 2 = b i f (x i )

57 Matrix Representation References for Au = b where A is tridiagonal and symmetric , , A = ,... 0 h ,, 0, ,...,, 0 1 2

58 Jacobi and Gauss-Seidel References for Jacobi: for i=1:n do ξi New (1/2)(h 2 b i + ξ Old i 1 + ξold i+1 ) end for Gauss-Seidel: for i=1:n do ξ i (1/2)(h 2 b i + ξ i 1 + ξ i+1 ) end for

59 Jacobi Iteration in MATLAB References for for ijac=1:n xnew(1) =.5*(hˆ2 * b(1) + xold(2)); for i=2:n 1 xnew(i) =.5*(hˆ2 * b(i) + xold(i 1) + xold(i+1)); end xnew(n) =.5*(hˆ2 * b(n) + xold(n 1)); xold=xnew; end How would you turn this into Gauss-Seidel with a text editor?

60 Jacobi Example Notation and Preliminaries References for Let s solve u = 0, u(0) = u(1) = 0. with h = 1/101 and N = 100. The solution is u = 0. We will use as an intial iterate u 0 = x(1 x) cos(49πx) We will take 100 Jacobi iterations.

61 Initial Error as Function of x References for 0.35 Initial Error as Function of $x$ error x

62 Final Error as Function of x References for 0.25 Final Error as Function of $x$ error x

63 Final Error as Function of x References for Convergence History 0.25 l error Iterations

64 Eigenvalues and Eigenvectors References for Theorem: A is symmetric positive definite. The eigenvalues are λ n = h 2 2 (1 cos(πnh)) = π 2 n 2 + O(h 2 ). The eigenvectors u n = (ξ n 1,..., ξn N )T are given by ξ n i = 2/h sin(niπh)

65 References for Comments and Proof Eigenvalues agree with continuous problem to second order. κ(a) = λ N /λ 1 = O(N 2 ) = O(h 2 ). ξi n = u n (x i ) 2/h Proof: Note that ξ0 n = ξn N+1 = 0 ξ n i 1 + 2ξn i ξ n i+1 = 2/h( sin(n(i 1)πh) + 2 sin(niπh) sin(n(i 1)πh))

66 End of Proof Notation and Preliminaries References for Set x = niπh and y = nπh. Use the trig identities sin(x ± y) = sin(x) cos(y) ± sin(y) cos(x) to get ξi 1 n + 2ξn i ξi+1 n = sin(x y) + 2 sin(x) sin(x + y) = 2 sin(x)(1 cos(y)) = λ n ξ n i as asserted.

67 Jacobi does Poorly for Poisson References for If you apply Jacobi to Poisson s equation, iteration matrix is M = D 1 (L + U) = I D 1 (D + L + U) = I D 1 A as we have seen. For Poisson, D = (2/h 2 )I so M = I D 1 A = I (h 2 /2)A. The eigenvalues of M are µ n = 1 (h 2 /2)λ n. So ρ(m) = 1 O(h 2 ) which is very bad. The performance gets worse as the mesh is refined!

68 Observations Notation and Preliminaries References for Jacobi (and GS, SOR,... ) are not scalable. The number of iterations needed to reduce the error by a given amount depends on the grid. Fixing this for PDE problems requires a different approach. You can solve the 1D problem in O(N) time with a tridiagonal solver, but... direct methods become harder to use for 2D and 3D problems on complex geometries with unstructured grids.

69 References for in Two Dimensions Equation: u xx u yy = f (x, y) for 0 < x, y < 1 Boundary conditions: u(0, y) = u(x, 0) = u(1, y) = u(x, 1) = 0 Similar properties to 1-D Physical Grid: (x i, x j ), x i = i h. Begin with two-dimensional matrix of unknowns u ij u(x i, x j ). Must order the unknowns (ie the grid points) to prepare for a packaged linear solver.

70 References for which leads to... u xx 1 (u(x h, y) 2u(x, y) + u(x + h, y)) h2 u yy 1 h 2 (u(x, y h) 2u(x, y) + u(x, y h))

71 Discrete 2D Poisson, Version 1 References for 1 h 2 ( U i 1,j U i,j 1 + 4U ij U i+1,j U i,j+1 ) = f ij f (x i, x j ) Jacobi, Gauss-Seidel,... are still easy. Here s GS for i=1:n do for j=1:n do U ij 1 ( 4 h 2 ) f ij + U i 1,j + U i,j 1 + U i+1,j + U i,j+1 end for end for So how did I order the unknowns?

72 Ordering the Unknowns References for N 2 N + 1 N 2 N N N + 1 2N N N + 1 N N N

73 References for Creating a Matrix-Vector Representation Define u = (ξ 1,..., ξ N 2) T R N2 by ξ N(i 1)+j = U i,j and β N(i 1)+j = U i,j The Matrix representation is Au = b where...

74 Matrix Laplacian in 2D: I References for A = 1 h 2 T I , 0 I T I, I T I, ,, 0, I T I 0...,...,, 0 I T where I is the N N identity matrix and T is the N N tridiagonal matrix

75 Discrete Laplacian in 2D: I References for T = , , , ,, 0, ,...,, 0 1 4

76 References for Mapping the 2D vector to/from a 1D vector Use the MATLAB reshape command. Example: N = u 2d = u 1d = reshape(u 2d, N N, 1) = (1, 4, 7, 2, 5, 8, 3, 6, 9) T and u 2d = reshape(u 1d, N, N). This means you can do things on the physical grid and still give solvers linear vectors when they need them.

77 Richardson Iteration Use the trapezoid rule to discretize the integral equation u(x) sin(x y)u(y) dy = cos(x) If you write your discrete equation as u Mu = b, prove that ρ(m) < 1/2. Write a MATLAB code to demonstrate that the convergence is independent of the mesh width.

78 Poisson Equation Compute the eigenvalues and eigenvectors for the 2D discrete Poission equation. Encode the 2D Laplacian as a MATLAB sparse matrix and use the eigs command to find a few eigenvectors and eigenvalues to verify your work in the previous exercise. Solve the 1D and 2D Poission equations with Jacobi and Gauss-Seidel with zero boundary data and f 1 as the right side. Try more interesting right sides f (x) and f (x, y).