Introduction to Simulation - Lecture 10. Modified Newton Methods. Jacob White

Size: px

Start display at page:

Download "Introduction to Simulation - Lecture 10. Modified Newton Methods. Jacob White"

Jesse Randall
6 years ago
Views:

1 Introduction to Simulation - Lecture 0 Modified Newton Methods Jacob White Thans to Deepa Ramaswamy, Jaime Peraire, Michal Rewiensi, and Karen Veroy

2 Outline Damped Newton Schemes SMA-HPC 003 MIT Globally Convergent if Jacobian is Nonsingular Difficulty with Singular Jacobians Introduce Continuation Schemes Problem with Source/Load stepping More General Continuation Scheme Improving Continuation Efficiency Better first guess for each continuation step Arc Length Continuation

3 Multidimensional Newton Method Newton Algorithm Newton Algorithm for Solving x ) = 0 0 x = = Initial Guess, 0 Repeat { ) ) Compute x, J x Solve = + ) = J x x x x for x + + } Until x + x x + ), small enough SMA-HPC 003 MIT

4 Multidimensional Newton Method If Main Theorem ) a J x β ) Inverse is bounded Multidimensional Convergence Theorem Theorem Statement b) J x J y x y Derivative is Lipschitz Cont Then Newton s method converges given a sufficiently close initial guess SMA-HPC 003 MIT

5 Multidimensional Newton Method Multidimensional Convergence Theorem Implications If a function s first derivative never goes to zero, and its second derivative is never too large Then Newton s method can be used to find the zero of the function provided you all ready now the answer. Need a way to develop Newton methods which converge regardless of initial guess! SMA-HPC 003 MIT

6 Non-converging Case fx) -D Picture x 0 x X Limiting the changes in X might improve convergence SMA-HPC 003 MIT

7 Newton Method with Limiting Newton Algorithm Newton Algorithm for Solving x ) = 0 0 x = = Initial Guess, 0 Repeat { ) ) Compute x, J x Solve J x x = x for x + + SMA-HPC 003 MIT x = x + limited x + + = + } Until x + x + ), small enough

8 Newton Method with Limiting Damped Newton Scheme General Damping Scheme Solve J x x = x for x + + x = x + α x + + Key Idea: Line Search Pic α to minimize SMA-HPC 003 MIT + + α x ) x + ) + ) + + α + α + α ) x x x x x x Method Performs a one-dimensional search in Newton Direction T

9 Newton Method with Limiting If Damped Newton Convergence Theorem ) a J x β ) Inverse is bounded b) J x J y x y Derivative is Lipschitz Cont Then ] There exists a set of α ' s 0, such that + ) + ) = + < ) x x α x γ x with γ< Every Step reduces -- Global Convergence! SMA-HPC 003 MIT

10 Newton Method with Limiting 0 x Initial Guess, = = 0 Repeat { Solve ) ) Compute x, J x J x x = x for x + + ] + x + α x ) ind α 0, such that is minimizd e x = x + α x + + = + } Until x + x + ) Damped Newton Nested Iteration, small enough SMA-HPC 003 MIT

11 Newton Method with Limiting v v 0 v V - d V d Vt I I e ) = 0 d Damped Newton Example I r s Vr = 0 0 Nodal Equations with Numerical Values v ) 0) v f v = + 0 e ) = 0 0

12 Newton Method with Limiting Damped Newton Example cont. v ) 0) v f v = + 0 e ) = 0 0

13 Newton Method with Limiting 0 x Initial Guess, = = 0 Repeat { Solve ) ) Compute x, J x J x x = x for x Damped Newton Nested Iteration + + ] + x + α x ) ind α 0, such that is minimizd e x = x + α x + + = + } Until x + x + ), small enough How can one find the damping coefficients? SMA-HPC 003 MIT

14 Newton Method with Limiting Damped Newton Theorem Proof By definition of the Newton Iteration + = - α x x J x x Newton Direction Multidimensional Mean Value Lemma x) y) J y) x y) x y Combining x x x + ) + J x α J x x α J x SMA-HPC 003 MIT

15 Newton Method with Limiting Damped Newton Theorem Proof-Cont rom the previous slide x x x + + J x α J x x α J x Combining terms and moving scalars out of norms x SMA-HPC 003 MIT + ) α x α J x x Using the Jacobian Bound and splitting the norm + β x α x α x ) + Yields a quadratic in the damping coefficient

16 Newton Method with Limiting Simplifying quadratic from previous slide Two Cases: β + Damped Newton Theorem Proof-Cont-II α + α ) x ) x ) x β ) x ) < Pic α = Standard Newton) β α + α ) x ) < β ) x ) > Pic α = β x SMA-HPC 003 MIT β α + α ) x ) < β x

17 Newton Method with Limiting Combining the results from the previous slide + γ x ) x The above result does imply 0 x ) x + or the case where β x ) Damped Newton Theorem Proof-Cont-III not good enough, need γ independent from not yet a convergence theorem > 0 β x ) β x ) γ 0 Note the proof technique irst Show that the iterates do not increase Second Use the non-increasing fact to prove convergence SMA-HPC 003 MIT

18 Newton Method with Limiting 0 x Initial Guess, = = 0 Repeat { Solve ) ) Compute x, J x J x x = x for x + + ] + x + α x ) ind α 0, such that is minimizd e x = x + α x + + = + } Until x + x + ) Damped Newton Nested Iteration, small enough Many approaches to finding α SMA-HPC 003 MIT

19 Newton Method with Limiting fx) Damped Newton Singular Jacobian Problem x x x D 0 x X SMA-HPC 003 MIT Damped Newton Methods push iterates to local minimums inds the points where Jacobian is Singular

20 Continuation Schemes Basic Concepts Source or Load-Stepping Newton converges given a close initial guess Generate a sequence of problems Mae sure previous problem generates guess for next problem Heat-conducting bar example. Start with heat off, T= 0 is a very close initial guess. Increase the heat slightly, T=0 is a good initial guess 3. Increase heat again SMA-HPC 003 MIT

21 Continuation Schemes ) Basic Concepts General Setting Solve x λ, λ = 0 where: a) x 0,0 = 0 is easy to solve ) ) = x) b) x, c) x λ ) is sufficiently smooth x λ ) Ends the continuation Starts the continuation Hard to insure! Dissallowed 0 λ SMA-HPC 003 MIT

22 Basic Concepts Continuation Schemes Template Algorithm Solve x 0,0, ) ) x λ ) prev = x 0) δλ=0.0, λ = δλ While λ < { } x 0 λ) = x λ ) prev ) Try to Solve x λ, λ = 0 with Newton If Newton Converged x λ ) prev = x λ), λ = λ+ δλ, δλ = δλ Else δλ = δλ, λ = λprev + δλ SMA-HPC 003 MIT

23 V s Continuation Schemes + - R v Diode Basic Concepts Source/Load Stepping Examples f v λ), λ) = idiode v) + v λvs ) = 0 R λ ) f v, idiode v = + Not λ dependent! v v R f f L x, λ = f x, y = 0 x f x, y + λ f = 0 y l Source/Load Stepping Does Not Alter Jacobian SMA-HPC 003 MIT

24 Continuation Schemes ) Jacobian Altering Scheme Description x λ, λ = λ x λ + λ x λ 0,0 ) ) x 0,0 ) λ = 0 x = x 0 = 0 x Observations = I Problem is easy to solve and Jacobian definitely nonsingular. ) ) x λ = x,= x 0,0 ) x ) x = x Bac to the original problem and original Jacobian SMA-HPC 003 MIT

25 Continuation Schemes Basic Algorithm Solve x 0,0, ) ) x λ ) prev = x 0) δλ=0.0, λ = δλ While λ < { } x 0 λ) = x λ ) prev +? ) Jacobian Altering Scheme Try to Solve x λ, λ = 0 with Newton If Newton Converged x λ ) prev = x λ), λ = λ+ δλ, δλ = δλ Else δλ = δλ, λ = λprev + δλ SMA-HPC 003 MIT

26 Continuation Schemes Jacobian Altering Scheme Initial Guess for each step. x λ) x λ + δλ λ λ + δλ λ + δλ) = x λ) 0 0 x Initial Guess Error λ SMA-HPC 003 MIT

27 Continuation Schemes x λ + δλ λ + δλ, ) x λ) λ) x Have rom last step s Newton SMA-HPC 003 MIT Jacobian Altering Scheme Update Improvement x λ λ + x 0), ) λ), λ) x λ δλ) x λ) ) x x λ, ) λ λ δλ + + λ), x x λ + δλ) x λ )) = λ, 0 Better Guess for next step s Newton λ δλ

28 Continuation Schemes Jacobian Altering Scheme Update Improvement Cont. If ) x, = x + x λ λ λ λ λ λ Then x, λ ) λ x λ ) = x Easily Computed SMA-HPC 003 MIT

29 Continuation Schemes Jacobian Altering Scheme Update Improvement Cont. II. λ) λ) x λ) x,, λ 0 x λ+ δλ) = x λ ) δλ x λ Graphically x λ) 0 x λ + δλ λ λ + δλ 0 λ SMA-HPC 003 MIT

30 Continuation Schemes x λ) Jacobian Altering Scheme Still can have problems Must switch bac to increasing lambda Arc-length steps Must switch from increasing to decreasing lambda 0 lambda steps λ SMA-HPC 003 MIT

31 Continuation Schemes x λ) Jacobian Altering Scheme Arc-length Steps? Arc-length steps arc-length x + δλ SMA-HPC 003 MIT 0 x, λ ) = 0 λ Must Solve or Lambda λ λ λ ) prev + x x prev arc = 0

32 Continuation Schemes Jacobian Altering Scheme Arc-length steps by Newton ) x, λ x, λ ) + x x x λ + = ) T λ λ x x λ ) prev λ λprev x, λ ) λ λ prev λprev + x x arc SMA-HPC 003 MIT

33 Continuation Schemes x λ) Jacobian Altering Scheme Arc-length Turning point What happens here? 0 Upper left-hand Bloc is singular λ x x, ) λ x, λ ) λ ) T x x λ ) prev λ λprev SMA-HPC 003 MIT

34 Summary Damped Newton Schemes SMA-HPC 003 MIT Globally Convergent if Jacobian is Nonsingular Difficulty with Singular Jacobians Introduce Continuation Schemes Problem with Source/Load stepping More General Continuation Scheme Improving Efficiency Better first guess for each continuation step Arc-length Continuation

Introduction to Simulation - Lecture 9. Multidimensional Newton Methods. Jacob White

Introduction to Simulation - Lecture 9 Multidimensional Newton Methods Jacob White Thanks to Deepak Ramaswamy, Jaime Peraire, Michal Rewienski, and Karen Veroy Outline Quick Review of -D Newton Convergence