Optimization II: Unconstrained Multivariable CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 1 / 20
Unconstrained Multivariable Problems minimize f : R n R CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 2 / 20
Recall f( x) Direction of steepest ascent CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 3 / 20
Recall f( x) Direction of steepest descent CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 4 / 20
Observation If f( x) 0, for sufficiently small α > 0, f( x α f( x)) f( x) CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 5 / 20
Gradient Descent Algorithm Iterate until convergence: 1. g k (t) f( x k t f( x k )) 2. Find t 0 minimizing (or decreasing) g k 3. x k+1 x k t f( x k ) CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 6 / 20
Stopping Condition f( x k ) 0 Don t forget: Check optimality! CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 7 / 20
Line Search g k (t) f( x k t f( x k )) One-dimensional optimization Don t have to minimize completely: Wolfe conditions CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 8 / 20
Newton s Method (again!) f( x) f( x k ) + f( x k ) ( x x k ) + 1 2 ( x x k) H f ( x k )( x x k ) CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 9 / 20
Newton s Method (again!) f( x) f( x k ) + f( x k ) ( x x k ) + 1 2 ( x x k) H f ( x k )( x x k ) = x k+1 = x k [H f ( x k )] 1 f( x k ) CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 9 / 20
Newton s Method (again!) f( x) f( x k ) + f( x k ) ( x x k ) + 1 2 ( x x k) H f ( x k )( x x k ) = x k+1 = x k [H f ( x k )] 1 f( x k ) Consideration: What if H f is not positive (semi-)definite? CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 9 / 20
Motivation f might be hard to compute but H f is harder H f might be dense: n 2 CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 10 / 20
Quasi-Newton Methods Approximate derivatives to avoid expensive calculations e.g. secant, Broyden,... CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 11 / 20
Common Optimization Assumption f known H f unknown or hard to compute CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 12 / 20
Quasi-Newton Optimization x k+1 = x k α k B 1 k f( x k) B k H f ( x k ) CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 13 / 20
Warning <advanced material> See Nocedal & Wright CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 14 / 20
Broyden-Style Update B k+1 ( x k+1 x k ) = f( x k+1 ) f( x k ) CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 15 / 20
Additional Considerations B k should be symmetric B k should be positive (semi-)definite CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 16 / 20
Davidon-Fletcher-Powell (DFP) min B k+1 B k B k+1 s.t. B k+1 = B k+1 B k+1 ( x k+1 x k ) = f( x k+1 ) f( x k ) CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 17 / 20
Observation B k+1 B k small does not mean B 1 is small k+1 B 1 k CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 18 / 20
Observation B k+1 B k small does not mean B 1 is small k+1 B 1 k Idea: Try to approximate directly B 1 k CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 18 / 20
BFGS Update min H k+1 H k HB k+1 s.t. H k+1 = H k+1 x k+1 x k = H k+1 ( f( x k+1 ) f( x k )) State of the art! CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 19 / 20
Lots of Missing Details Choice of Limited-memory alternative Next CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 20 / 20