Some definitions. Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization. A-inner product. Important facts

Transcription

1 Some definitions Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 A matrix A is SPD (Symmetric Postive Definite) 1. A = A T 2. x Ax = x, Ax = x T Ax > 0 for x 0 A is negative definite if A is SPD A is skew-symmetric if A T = A A is indefinite if symmetric and neither SPD nor negative definite. Two symmetric matrices satisfy A > B if (A B) is SPD March 2015 Important facts 1 / 66 A-inner product 2 / 66 All eigenvalues of a symmetric matrix are real A symmetric matrix has an orthogonal basis of eigenvectors A symmetric matrix A is positive definite if and only if all λ(a) > 0. If A is SPD then x, y A = x t Ay is a weighted inner product on R n, the A-inner product, and x A = x, x A = x t Ax is a weighted norm, the A-norm. 3 / 66 4 / 66

2 Proof The A-norm is a weighted 2-norm bilinear Positive Symmetric u+v, y A = (u+v) t Ay = u t Ay +v t Ay = u, y A + v, y A and αu, y A = (αu) t Ay = α ( u t Ay ) = α u, y A. x, x A = x t Ax > 0, for x 0, since A is SPD. x, y A = x t Ay = (x t Ay) t = y t A t x tt = y t Ax = y, x A. The norm induced by a 2 2 SPD matrix A is just a weighted 2-norm. Proof: Let (λ 1, φ 1 ) and (λ 2, φ 2 ) be the two eigenvalues and eigenvectors of A Any vector v = α 1 φ 1 + α 2 φ 2 v 2 A = (α 1φ 1 + α 2 φ 2 ) T A(α 1 φ 1 + α 2 φ 2 ) v 2 A = (α 1φ 1 + α 2 φ 2 ) T (α 1 λ 1 φ 1 + α 2 λ 2 φ 2 ) v 2 A = (α2 1 λ 1 + α 2 2 λ 2) Hence, x, x A induces a norm x A = x, x A 5 / 66 6 / 66 Homework Finding a solution = finding a minimum For A SPD, the unique solution of Ax = b Text, Exercises 258, 259 is also the unique minimizer of the functional J(x) = 1 2 x, Ax x, b = 1 2 (x T Ax) x T b Proof Take derivatives of both sides. J x i = 0 = 1 2 ((Ax) i + (Ax) i ) b i 7 / 66 9 / 66

3 Notation Theorem 260 x = A 1 b is the argument that minimizes J(y): Let A be SPD. The solution of Ax = b is the unique minimizer of J(x). For any y x = arg min y R N J(y) And J(x + y) = J(x) y t Ay > J(x) x y 2 A = 2 (J(y) J(x)). 10 / / 66 Proof of first claim Proof of second claim J(x + y) = 1 2 (x + y)t A(x + y) (x + y) t b = 1 2 x t Ax + y t Ax y t Ay x t b y t b = ( 1 2 x t Ax x t b) + (y t Ax y t b) y t Ay = J(x) + y t (Ax b) y t Ay = J(x) y t Ay > J(x). A is SPD, so y 0 = y t Ay > 0. LHS x y 2 A = (x y) t A(x y) = x t Ax y t Ax x t Ay + y t Ay = ( since Ax = b) = x t b y t b y t b + y t Ay = y t Ay 2y t b + x t b RHS 2 (J(y) J(x)) = y t Ay 2y t b x t Ax + 2x t b = (since Ax = b) = y t Ay 2y t b x t [Ax b] + x t b = y t Ay 2y t b + x t b 12 / / 66

4 The 2 2 case 2 2 system Ax = b with [ a c A = c d Eigenvalues: (a λ)(d λ) c 2 = 0 ]. The 2 2 case cont d Write x = [x 1, x 2 ] T J(x 1, x 2 ) = 1 [ a c 2 (x 1, x 2 ) c d ] [ ] x1 [x x 1, x 2 ] 2 [ b1 = 1 2 (ax cx 1 x 2 + dx 2 2 ) (b 1 x 1 + b 2 x 2 ). b 2 ] The surface z = J(x 1, x 2 ) is a paraboloid opening up if and only if a > 0, d > 0, and c 2 ad < 0. Hessian matrix is SPD (Theorem 260) λ = (a + d) ± (a + d) 2 4(ad c 2 ) 2 Positive definite (λ > 0) = (a + d) > 0 and c 2 ab < 0 z=j(x,y) Paraboloid opening upward (again) Consider b 1 = b 2 = 0: x 1, x 2 are components of the error J(x 1, x 2 ) = 1 2 (ax cx 1x 2 + dx 2 2 ) Set x 1 = r a cos θ and x 2 = r d sin θ J = r 2 cos 2 θ + r 2 sin 2 θ + 2r 2 c sin θ cos θ ad ( = r c ) sin 2θ ad J > 0 if and only if ± c ad < 1 or c 2 < ad. 14 / 66 General Descent Method, Algorithm 263 Given 1. Ax = b 2. a quadratic functional J( ) that is minimized at x = A 1 b 3. a maximum number of iterations itmax 4. an initial guess x 0 : Compute r 0 = b Ax 0 for n=1:itmax ( ) Choose a direction vector d n Find α = α n by solving the 1d minimization problem: ( ) α n = arg min α J(x n + αd n ) x n+1 = x n + α n d n r n+1 = b Ax n+1 if converged, exit, end end 15 / / / 66

5 Steepest descent J = r when J(x) = 1 2 x t Ax x t b J = 1 2 x T Ax x T b Functional: J(x) = 1 2 x t Ax x t b Descent direction: d n = J(x n ) J x i = 1 2 (Ax) i (x T A) i b i But 1 2 (x T A) i = 1 2 ((Ax)T ) i = 1 2 (Ax) i So J x i J = r = (Ax) i b i = r i 19 / / 66 Formula for α Homework Proof of following is a homework problem: For arbitrary d n, α n = (d n ) T r n (d n ) T Ad n Text, 264. d Hint: Set dα (J(x + αd)) = 0 and solve. 21 / / 66

6 Notation Descent demonstration Use the, notation α for steepest descent becomes α n = d n, r n d n, Ad n = d n, r n d n, d n A. descentdemo.m 23 / / 66 Different descent methods: descent direction d n Different descent methods: choice of functional J If A is SPD the most common choice is Steepest descent direction: d n = J(x n ) Random directions: d n = a randomly chosen vector Gauss-Seidel like descent: d n cycles through the standard basis of unit vectors e 1, e 2,, e N and repeats if necessary. Conjugate directions: d n cycles through an A-orthogonal set of vectors. J(x) = 1 2 x t Ax x t b Minimum residual methods: for general A, J(x) = 1 b Ax, b Ax. 2 Various combinations such as residuals plus updates: J(x) = 1 2 b Ax, b Ax x x n, x x n 25 / / 66

7 Homework Stationary Iterative Methods Text, 265, 268 A = M N Iterative method Mx n+1 = b + Nx n Equivalently M(x n+1 x n ) = b Ax n 27 / / 66 Examples Householder lemma (269) FOR: M = ρi, N = ρi A Jacobi: M = diag(a) Gauss-Seidel: M = D + L (lower triangular part of A). Let A be SPD Let x n be given by M(x n+1 x n ) = b Ax n Let e n = x x n then e n, Ae n e n+1, Ae n+1 = (x n+1 x n ), P(x n+1 x n ) where P = M + M T A. 30 / / 66

8 Convergence of FOR, Jacobi, GS and SOR Proof Corollary 270 For A SPD, P SPD = convergence is monotonic in the A norm: e n A > e n+1 A >... 0 as n 0. Householder lemma in this case gives e n, Ae n e n+1, Ae n+1 = (x n+1 x n ), P(x n+1 x n ), so e n, Ae n > e n+1, Ae n+1, or e n A 0 and monotone decreasing, so has a limit Cauchy criterion: ( e n, Ae n e n+1, Ae n+1 ) 0. Householder gives x n+1 x n P 0. So that M ( x n+1 x n) = b Ax n 0 e n+1 A < e n A 32 / / 66 Monotone convergence Proofs 1. For Jacobi, M = diag(a), so 1. FOR converges monotonically in A if P = M + M t A = 2ρI A > 0 if ρ > 1 2 λ max(a). 2. Jacobi converges monotonically in the A-norm if diag(a) > 1 2 A. 3. Gauss-Seidel converges monotonically in the A norm for SPD A in all cases. 4. SOR converges monotonically in the A-norm if 0 < ω < 2. P = M + M t A = 2diag(A) A > 0 if diag(a) > 1 2 A. 2. For GS, A = D + L + L T (since A SPD) P = M + M T A = (D + L) + (D + L) T A = 3. For SOR, D + L + D + L T (D + L + L T ) = D > 0 P = M + M t A = M + M t (M N) = M t + N = ω 1 D + L t + 1 ω ω D U = 2 ω ω D > 0, for 0 < ω < 2 4. Convergence of FOR in the A-norm is left as an exercise. 34 / / 66

9 Homework Picking ρ for FOR Text, Exercises 275, 278 In Exercise 278, you are to: 1. Give details of the proof that P > 0 implies convergence of FOR. 2. Prove the Householder lemma. Hint: expand the right side using P = M + M T A and the identities x n+1 x n = e n e n+1 and y, Wz = W T y, z. FOR: ρ optimal = (λ max + λ min )/2 This gives fastest overall convergence for fixed ρ Hard to calculate. What about ρ n = Best possible for this step 36 / / 66 Variable ρ FOR again Residuals r n = b A x n for FOR satisfy Given x 0 and a maximum number of iterations, itmax: for n=0:itmax Compute r n = b Ax n Compute ρ n via a few auxiliary calculations x n+1 = x n + (ρ n ) 1 r n if converged, exit, end end r n+1 = r n ρ 1 Ar n. Proof: x n+1 = x n ρ 1 r n. Multiply by A and add b to both sides b Ax n+1 = b Ax n ρ 1 Ar n, 39 / / 66

10 Homework Option 1 Residual Minimization Pick ρ n to minimize r n+1 2 : r n+1 2 = r n ρ 1 Ar n 2 = r n ρ 1 Ar n, r n ρ 1 Ar n. Since r n is fixed, this is a simple function of ρ Text, Exercise 281 Easy! Preparation for following work. or J(ρ) = r n ρ 1 Ar n, r n ρ 1 Ar n J(ρ) = r n, r n 2 r n, Ar n ρ 1 Ar n, Ar n ρ 2. Taking J (ρ) = 0 and solving for ρ = ρ optimal gives ρ optimal = Ar n, Ar n r n, Ar n. The cost of using this optimal value at each step: two extra dot products per step. 41 / / 66 Option 2 J minimization Algorithm for Option 2 Minimize J(x n+1 ): φ(ρ) = J(x n + ρ 1 r n ) = 1 2 x n + ρ 1 r n, A(x n + ρ 1 r n ) x n + ρ 1 r n, b Expanding, setting φ (ρ) = 0 and solving, as before, gives Option 2 requires SPD A Faster than Option 1 ρ optimal = r n, Ar n r n, r n. Given x 1 the matrix A a maximum number of iterations, itmax: r 1 = b Ax 1 for n=1:itmax ρ n = Ar n, r n / r n, r n x n+1 = x n + (1/ρ n ) r n if satisfied, exit, end r n+1 = b Ax n+1 end 43 / / 66

11 Homework The normal equations Ae = r so r 2 2 = Ae 2 2 = e 2 A T A Text, Exercise 283. Implement and test Options 1 and 2. A T A is SPD, so minimizing e 2 is minimizing A T A (A T A)x = A T b. Both bandwidth and condition number of A T A are much larger than A, but A T A is SPD. 45 / / 66 Condition number of normal equations (284) Steepest descent Let A be N N and invertible. Then A T A is SPD. If A is SPD then λ(a T A) = λ(a) 2 and cond 2 (A T A) = [cond 2 (A)] 2. Proof Symmetry: (A T A) T = A T A TT = A T A Positivity: x, AA T x = A T x, A T x = A T x 2 > 0 Condition no.: For A SPD Reduce J as much as possible in gradient direction Option 2 cond 2 (A t A) = cond 2 (A 2 ) = λ max(a 2 ) λ min (A 2 ) = ( ) 2 λmax (A) = (cond 2 (A)) 2. λ min (A) Squaring the condition number greatly increases number of iterations 47 / / 66

12 Steepest descent algorithm Example: It is still slow! Given SPD A x 0, r 0 = b Ax 0 maximum number of iterations itmax for n=0:itmax r n = b Ax n α n = r n, r n / r n, Ar n x n+1 = x n + α n r n if converged, exit, end end α n was called 1/ρ n earlier. N = 2, x = (x 1, x 2 ) T. [ ] [ ] A =, b = J( x ) = 1 [ [x 1, x 2 ] 0 50 ] [ x1 x 2 ] [ 2 [x 1, x 2 ] 0 = 1 2 (2x x2 2 ) 2x 1 = x1 2 2x x }{{} ellipse = (x 1 1) x )2 ( 1 5 ] Steepest descent illustration 50 / 66 SD is no faster than FOR (287) 51 / 66 limit=(1, 0) x 0 =(11,1) x 4 =(6.37,0.54) Let A be SPD and κ = λ max (A)/λ min (A).. The steepest descent method converges to the solution of Ax = b for any x 0. The error x x n satisfies ( ) n κ 1 x x n A x x 0 A κ + 1 and J(x n ) J(x) But no eigenvalue estimates are required! ( ) n κ 1 ( J(x 0 ) J(x) ). κ / / 66

13 Proof Theorem 288: inequality is sharp! x x n, A(x x n ) = x, b + 2J(x n ) Minimizing J is the same as minimizing x x n A Steepest descent reduces J maximally FOR reduces x x n FOR A with known bound Doing one iteration of each, starting from same x n 1, ( ) κ 1 x x n A x xfor n A x x n 1 A κ + 1 Ax = b If x 0 = x (λ 2 φ 1 ± λ 1 φ 2 ) where φ 1 and φ 2 are the eigenvectors of λ 1 = λ min (A) and λ 2 = λ max (A) respectively Then the convergence rate is precisely λ 2 λ 1 = κ 1 λ 2 + λ 1 κ + 1 Proof 54 / 66 Homework 55 / 66 x 0 = x (λ 2 φ 1 ± λ 1 φ 2 ) e 0 = λ 2 φ 1 ± λ 1 φ 2 r 0 = Ae 0 = λ 1 λ 2 φ 1 λ 1 λ 2 φ 2 Ar 0 = λ 2 1λ 2 φ 1 λ 1 λ 2 2φ 2 α 0 = r 0, r 0 / r 0, Ar 0 = λ2 1 λ2 2 + λ2 1 λ2 2 λ 3 1 λ2 2 + λ2 1 λ3 2 2 = λ 1 + λ 2 x 1 = x 0 + αr 0 ( 2 = x λ 2 λ 1 λ 2 λ 1 + λ 2 = x λ 2 ( λ2 λ 1 λ 2 + λ 1 = x λ 2 λ 1 λ 2 + λ 1 (λ 2 φ 1 λ 1 φ 2 ) ) ( φ 1 ) φ 1 λ 1 ( ±λ1 λ 2 λ 1 + λ 2 λ λ 1 λ 2 λ 1 + λ 2 ) φ 2 ) φ 2 Text, Exercise / / 66

14 Exercise 290: Convection-Diffusion Equation Comments about CDE ɛ u + c u = f inside Ω = [0, 1] [0, 1] [ ] 1 c = 0 u = g on Ω The convection term is discretized as c is like a wind, blowing the solution to the right. Interesting behavior for ɛ < c Pe= cell Péclet number Discretization becomes unstable as Pe becomes large. c u = u x u(i + 1, j) u(i 1, j) 2h Modify jacobi2d.m 58 / 66 Real problem 59 / 66 Add in convection term. Ends up with a factor of h in the numerator Debug with f = 1, g = x y. Exact solution is u = g Also exact discrete solution! Diffusion is debugged already Debug with N = 5 Can see the solution on the screen h = 0.2 so can recognize solution values on sight Start from exact: must get answer in 1 iteration. Start from exact, run many iterations: must stay exact Start from zero, get exact? If not, program in exact solution, print norm of error at each iteration. Double-check original code! N = 50, f = x + y, g = 0 Jacobi, steepest descent, residual minimization ɛ = 1.0, 0.1, 0.01, / / 66

15 Solution, ɛ = 1.0 Solution, ɛ = 0.1 Solution, ɛ = / 66 Solution, ɛ = / / / 66

16 Iterative methods Numbers of iterations ɛ Jacobi Steepest descent Residual minimization diverges / 66