Nonlinear Systems CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Nonlinear Systems 1 / 27
Part III: Nonlinear Problems Not all numerical problems can be solved with \ in Matlab. CS 205A: Mathematical Methods Nonlinear Systems 2 / 27
Question Have we already seen a nonlinear problem? CS 205A: Mathematical Methods Nonlinear Systems 3 / 27
Question Have we already seen a nonlinear problem? minimize A x 2 such that x 2 = 1 nonlinear! CS 205A: Mathematical Methods Nonlinear Systems 3 / 27
Root-Finding Problem Given: f : R n R m Find: x with f( x ) = 0 CS 205A: Mathematical Methods Nonlinear Systems 4 / 27
Root-Finding Applications Collision detection (graphics, astronomy) Graphics rendering (ray intersection) Robotics (kinematics) Optimization (line search) CS 205A: Mathematical Methods Nonlinear Systems 5 / 27
Issue: Regularizing Assumptions f(x) = { 1 when x 0 1 when x > 0 CS 205A: Mathematical Methods Nonlinear Systems 6 / 27
Issue: Regularizing Assumptions f(x) = g(x) = { 1 when x 0 1 when x > 0 { 1 when x Q 1 when x Q CS 205A: Mathematical Methods Nonlinear Systems 6 / 27
Typical Regularizing Assumptions Continuous f( x) f( y) as x y CS 205A: Mathematical Methods Nonlinear Systems 7 / 27
Typical Regularizing Assumptions Continuous f( x) f( y) as x y Lipschitz f( x) f( y) 2 c x y 2 for all x, y (same c) CS 205A: Mathematical Methods Nonlinear Systems 7 / 27
Typical Regularizing Assumptions Continuous f( x) f( y) as x y Lipschitz f( x) f( y) 2 c x y 2 for all x, y (same c) Differentiable Df( x) exists for all x CS 205A: Mathematical Methods Nonlinear Systems 7 / 27
Typical Regularizing Assumptions Continuous f( x) f( y) as x y Lipschitz f( x) f( y) 2 c x y 2 for all x, y (same c) Differentiable Df( x) exists for all x C k k derivatives exist and are continuous CS 205A: Mathematical Methods Nonlinear Systems 7 / 27
Today f : R R CS 205A: Mathematical Methods Nonlinear Systems 8 / 27
Property of Continuous Functions Intermediate Value Theorem Suppose that f : [a, b] R is continuous and that f(a) < u < f(b) or f(b) < u < f(a). Then, there exists z (a, b) such that f(z) = u CS 205A: Mathematical Methods Nonlinear Systems 9 / 27
Reasonable Input Continuous function f(x) l, r R with f(l) f(r) < 0 (why?) CS 205A: Mathematical Methods Nonlinear Systems 10 / 27
Bisection Algorithm 1. Compute c = l+r /2. 2. If f(c) = 0, return x = c. 3. If f(l) f(c) < 0, take r c. Otherwise take l c. 4. Return to step 1 until r l < ε; then return c. CS 205A: Mathematical Methods Nonlinear Systems 11 / 27
Bisection: Illustration f(x) f(x) > 0 l c x r x f(x) < 0 CS 205A: Mathematical Methods Nonlinear Systems 12 / 27
Two Important Questions 1. Does it converge? CS 205A: Mathematical Methods Nonlinear Systems 13 / 27
Two Important Questions 1. Does it converge? Yes! Unconditionally. CS 205A: Mathematical Methods Nonlinear Systems 13 / 27
Two Important Questions 1. Does it converge? Yes! Unconditionally. 2. How quickly? CS 205A: Mathematical Methods Nonlinear Systems 13 / 27
Convergence Analysis Examine E k with x k x < E k. CS 205A: Mathematical Methods Nonlinear Systems 14 / 27
Bisection: Linear Convergence E k+1 1 2 E k for E k r k l k CS 205A: Mathematical Methods Nonlinear Systems 15 / 27
Fixed Points g(x ) = x CS 205A: Mathematical Methods Nonlinear Systems 16 / 27
Fixed Points g(x ) = x Question: Same as root-finding? CS 205A: Mathematical Methods Nonlinear Systems 16 / 27
Simple Strategy x k+1 = g(x k ) CS 205A: Mathematical Methods Nonlinear Systems 17 / 27
Convergence Criterion E k x k x = g(x k 1 ) g(x ) CS 205A: Mathematical Methods Nonlinear Systems 18 / 27
Convergence Criterion E k x k x = g(x k 1 ) g(x ) c x k 1 x if g is Lipschitz = ce k 1 CS 205A: Mathematical Methods Nonlinear Systems 18 / 27
Convergence Criterion E k x k x = g(x k 1 ) g(x ) c x k 1 x = ce k 1 if g is Lipschitz = E k c k E 0 0 as k (c < 1) CS 205A: Mathematical Methods Nonlinear Systems 18 / 27
Alternative Criterion Lipschitz near x with good starting point. CS 205A: Mathematical Methods Nonlinear Systems 19 / 27
Alternative Criterion Lipschitz near x with good starting point. e.g. C 1 with g (x ) < 1 CS 205A: Mathematical Methods Nonlinear Systems 19 / 27
Convergence Rate of Fixed Point When it converges... Always linear (why?) CS 205A: Mathematical Methods Nonlinear Systems 20 / 27
Convergence Rate of Fixed Point When it converges... Always linear (why?) Often quadratic! ( board) CS 205A: Mathematical Methods Nonlinear Systems 20 / 27
Approach for Differentiable f(x) f(x) x 0 x 1 x 2 x CS 205A: Mathematical Methods Nonlinear Systems 21 / 27
Newton s Method x k+1 = x k f(x k) f (x k ) CS 205A: Mathematical Methods Nonlinear Systems 22 / 27
Newton s Method x k+1 = x k f(x k) f (x k ) Fixed point iteration on g(x) x f(x) f (x) CS 205A: Mathematical Methods Nonlinear Systems 22 / 27
Convergence of Newton Simple Root A root x with f (x ) 0. CS 205A: Mathematical Methods Nonlinear Systems 23 / 27
Convergence of Newton Simple Root A root x with f (x ) 0. Quadratic convergence in this case! ( board) CS 205A: Mathematical Methods Nonlinear Systems 23 / 27
Issue Differentiation is hard! CS 205A: Mathematical Methods Nonlinear Systems 24 / 27
Secant Method x k+1 = x k f(x k)(x k x k 1 ) f(x k ) f(x k 1 ) CS 205A: Mathematical Methods Nonlinear Systems 25 / 27
Secant Method x k+1 = x k f(x k)(x k x k 1 ) f(x k ) f(x k 1 ) Trivia: Converges at rate 1+ 5 2 1.6180339887... CS 205A: Mathematical Methods Nonlinear Systems 25 / 27
Secant Method x k+1 = x k f(x k)(x k x k 1 ) f(x k ) f(x k 1 ) Trivia: Converges at rate 1+ 5 2 1.6180339887... ( Golden Ratio ) CS 205A: Mathematical Methods Nonlinear Systems 25 / 27
Hybrid Methods Want: Convergence rate of secant/newton with convergence guarantees of bisection CS 205A: Mathematical Methods Nonlinear Systems 26 / 27
Hybrid Methods Want: Convergence rate of secant/newton with convergence guarantees of bisection e.g. Dekker s Method: Take secant step if it is in the bracket, bisection step otherwise CS 205A: Mathematical Methods Nonlinear Systems 26 / 27
Single-Variable Conclusion Unlikely to solve exactly, so we settle for iterative methods Must check that method converges at all Convergence rates: Linear: Ek+1 CE k for some 0 C < 1 Superlinear: E k+1 CEk r for some r > 1 Quadratic: r = 2 Cubic: r = 3 Time per iteration also important Next CS 205A: Mathematical Methods Nonlinear Systems 27 / 27