Nonlinear Systems CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear Systems 1 / 24
Part III: Nonlinear Problems Not all numerical problems can be solved with \ in Matlab. CS 205A: Mathematical Methods Nonlinear Systems 2 / 24
Question Have we already seen a nonlinear problem? CS 205A: Mathematical Methods Nonlinear Systems 3 / 24
Question Have we already seen a nonlinear problem? minimize A x 2 such that x 2 = 1 nonlinear! CS 205A: Mathematical Methods Nonlinear Systems 3 / 24
Root-Finding Problem Given: f : R n R m Find: x with f( x ) = 0 CS 205A: Mathematical Methods Nonlinear Systems 4 / 24
Issue: Regularizing Assumptions f(x) = { 1 x 0 1 x > 0 CS 205A: Mathematical Methods Nonlinear Systems 5 / 24
Issue: Regularizing Assumptions f(x) = { 1 x 0 1 x > 0 f(x) = { 1 x Q 1 otherwise CS 205A: Mathematical Methods Nonlinear Systems 5 / 24
Typical Regularizing Assumptions Continuous f( x) f( y) as x y CS 205A: Mathematical Methods Nonlinear Systems 6 / 24
Typical Regularizing Assumptions Continuous f( x) f( y) as x y Lipschitz f( x) f( y) C x y CS 205A: Mathematical Methods Nonlinear Systems 6 / 24
Typical Regularizing Assumptions Continuous f( x) f( y) as x y Lipschitz f( x) f( y) C x y Differentiable Df( x) exists for all x CS 205A: Mathematical Methods Nonlinear Systems 6 / 24
Typical Regularizing Assumptions Continuous f( x) f( y) as x y Lipschitz f( x) f( y) C x y Differentiable Df( x) exists for all x C k k derivatives exist and are continuous CS 205A: Mathematical Methods Nonlinear Systems 6 / 24
Today f : R R CS 205A: Mathematical Methods Nonlinear Systems 7 / 24
Property of Continuous Functions Intermediate Value Theorem Suppose f : [a, b] R is continuous. Suppose f(x) < u < f(y). Then, there exists z between x and y such that f(z) = u. CS 205A: Mathematical Methods Nonlinear Systems 8 / 24
Property of Continuous Functions Intermediate Value Theorem Suppose f : [a, b] R is continuous. Suppose f(x) < u < f(y). Then, there exists z between x and y such that f(z) = u. Used in Homework 1! CS 205A: Mathematical Methods Nonlinear Systems 8 / 24
Reasonable Starting Point f(l) f(r) < 0 CS 205A: Mathematical Methods Nonlinear Systems 9 / 24
Bisection Algorithm 1. Compute c = l+r /2. 2. If f(c) = 0, return x = c. 3. If f(l) f(c) < 0, take r c. Otherwise take l c. 4. Return to step?? until r l < ε; then return c. CS 205A: Mathematical Methods Nonlinear Systems 10 / 24
Two Important Questions 1. Does it converge? CS 205A: Mathematical Methods Nonlinear Systems 11 / 24
Two Important Questions 1. Does it converge? Yes! Unconditionally. CS 205A: Mathematical Methods Nonlinear Systems 11 / 24
Two Important Questions 1. Does it converge? Yes! Unconditionally. 2. How quickly? CS 205A: Mathematical Methods Nonlinear Systems 11 / 24
Convergence Analysis Examine E k with x k x < E k. CS 205A: Mathematical Methods Nonlinear Systems 12 / 24
Bisection: Linear Convergence E k+1 1 2 E k for E k r k l k CS 205A: Mathematical Methods Nonlinear Systems 13 / 24
Fixed Points g(x ) = x CS 205A: Mathematical Methods Nonlinear Systems 14 / 24
Fixed Points g(x ) = x Question: Same as root-finding? CS 205A: Mathematical Methods Nonlinear Systems 14 / 24
Simple Strategy x k+1 = g(x k ) CS 205A: Mathematical Methods Nonlinear Systems 15 / 24
Convergence Criterion E k x k x = g(x k 1 ) g(x ) CS 205A: Mathematical Methods Nonlinear Systems 16 / 24
Convergence Criterion E k x k x = g(x k 1 ) g(x ) C x k 1 x if g is Lipschitz = CE k 1 CS 205A: Mathematical Methods Nonlinear Systems 16 / 24
Alternative Criterion Lipschitz near x with good starting point. CS 205A: Mathematical Methods Nonlinear Systems 17 / 24
Alternative Criterion Lipschitz near x with good starting point. e.g. C 1 with g (x ) < 1 CS 205A: Mathematical Methods Nonlinear Systems 17 / 24
Convergence Rate of Fixed Point When it converges... Always linear (why?) CS 205A: Mathematical Methods Nonlinear Systems 18 / 24
Convergence Rate of Fixed Point When it converges... Always linear (why?) Often quadratic! CS 205A: Mathematical Methods Nonlinear Systems 18 / 24
Newton s Method x k+1 = x k f(x k) f (x k ) CS 205A: Mathematical Methods Nonlinear Systems 19 / 24
Newton s Method x k+1 = x k f(x k) f (x k ) Fixed point iteration on g(x) x f(x) f (x) CS 205A: Mathematical Methods Nonlinear Systems 19 / 24
Convergence of Newton Simple Root A root x with f (x ) 0. CS 205A: Mathematical Methods Nonlinear Systems 20 / 24
Convergence of Newton Simple Root A root x with f (x ) 0. Quadratic convergence in this case! CS 205A: Mathematical Methods Nonlinear Systems 20 / 24
Convergence of Newton Simple Root A root x with f (x ) 0. Quadratic convergence in this case! Higher-order approximations? CS 205A: Mathematical Methods Nonlinear Systems 20 / 24
Issue Differentiation is hard! CS 205A: Mathematical Methods Nonlinear Systems 21 / 24
Secant Method x k+1 = x k f(x k)(x k x k 1 ) f(x k ) f(x k 1 ) CS 205A: Mathematical Methods Nonlinear Systems 22 / 24
Secant Method x k+1 = x k f(x k)(x k x k 1 ) f(x k ) f(x k 1 ) Trivia: Converges at rate 1+ 5 2 1.6180339887... CS 205A: Mathematical Methods Nonlinear Systems 22 / 24
Secant Method x k+1 = x k f(x k)(x k x k 1 ) f(x k ) f(x k 1 ) Trivia: Converges at rate 1+ 5 2 1.6180339887... ( Golden Ratio ) CS 205A: Mathematical Methods Nonlinear Systems 22 / 24
Hybrid Methods Want: Convergence rate of secant/newton with convergence guarantees of bisection CS 205A: Mathematical Methods Nonlinear Systems 23 / 24
Hybrid Methods Want: Convergence rate of secant/newton with convergence guarantees of bisection e.g. Dekker s Method: Take secant step if it is in the bracket, bisection step otherwise CS 205A: Mathematical Methods Nonlinear Systems 23 / 24
Single-Variable Conclusion Unlikely to solve exactly, so we settle for iterative methods Must check that method converges at all Convergence rates: Linear: Ek+1 CE k for some 0 C < 1 Superlinear: E k+1 CEk r for some r > 1 Quadratic: r = 2 Cubic: r = 3 Time per iteration also important Next CS 205A: Mathematical Methods Nonlinear Systems 24 / 24