Numerical optimization. Numerical optimization. Longest Shortest where Maximal Minimal. Fastest. Largest. Optimization problems

Transcription

1 1 Numerical optimization Alexander & Michael Bronstein, Michael Bronstein, 2010 tosca.cs.technion.ac.il/book Numerical optimization Advanced topics in vision Processing and Analysis of Geometric Shapes EE Technion, Spring Alexander & Michael Bronstein, Michael Bronstein, 2010 tosca.cs.technion.ac.il/book Advanced topics in vision Processing and Analysis of Geometric Shapes EE Technion, Spring Optimization problems Generic unconstrained minimization problem Slowest Longest Shortest where Maximal Minimal Vector space Fastest is the search space is a cost (or objective) function A solution Largest Smallest The value is the minimizer of is the minimum Common denominator: optimization problems 5 Local vs. global minimum 6 Local vs. global in real life Find minimum by analyzing the local behavior of the cost function Local minimum False summit 8,030 m Main summit 8,047 m Global minimum Broad Peak (K3), 12th highest mountain on Earth 1

2 7 8 Convex functions One-dimensional optimality conditions A function defined on a convex set is called convex if Point is the local minimizer of a -function if for any and For convex function local minimum = global minimum. Approximate a function around as a parabola using Taylor expansion guarantees the minimum at guarantees the parabola is convex Convex Non-convex 9 10 Gradient Gradient In multidimensional case, linearization of the function according to Taylor gives a multidimensional analogy of the derivative. In Euclidean space ( ), can be represented in standard basis in the following way: i-th place The function, denoted as, is called the gradient of In one-dimensional case, it reduces to standard definition of derivative which gives Example 1: gradient of a matrix function Example 2: gradient of a matrix function Given (space of real matrices) with standard inner product Compute the gradient of the function where is an matrix Compute the gradient of the function where is an matrix For square matrices 2

3 13 14 Hessian Optimality conditions, bis Linearization of the gradient Point is the local minimizer of a -function if. gives a multidimensional analogy of the secondorder derivative. for all, i.e., the Hessian is a positive definite matrix (denoted ) The function is called the Hessian of, denoted as Ludwig Otto Hesse ( ) Approximate a function around as a parabola using Taylor expansion In the standard basis, Hessian is a symmetric matrix of mixed second-order derivatives guarantees the minimum at guarantees the parabola is convex Optimization algorithms Generic optimization algorithm Descent direction Step size Determine descent direction Choose step size such that Update iterate Increment iteration counter Solution Descent direction Step size Stopping criterion Stopping criteria Line search Near local minimum, (or equivalently ) Stop when gradient norm becomes small Optimal step size can be found by solving a one-dimensional optimization problem Stop when step size becomes small One-dimensional optimization algorithms for finding the optimal step size are generically called exact line search Stop when relative objective change becomes small 3

4 19 20 Armijo [ar-mi-xo] rule Descent direction The function sufficiently decreases if How to descend in the fastest way? Armijo rule (Larry Armijo, 1966): start with and decrease it by Go in the direction in which the height lines are the densest multiplying by some until the function sufficiently decreases Devil s Tower Topographic map Steepest descent Steepest descent Directional derivative: how much changes in the direction (negative for a descent direction) Find a unit-length direction minimizing directional derivative L 2 norm L 1 norm Normalized steepest descent Coordinate descent (coordinate axis in which descent is maximal) Steepest descent algorithm Condition number Compute steepest descent direction Condition number is the ratio of maximal and minimal eigenvalues of the Hessian, 1 1 Choose step size using line search Update iterate Increment iteration counter Problem with large condition number is called ill-conditioned Steepest descent rate is slow for ill-conditioned problems 4

5 25 26 Q-norm Preconditioning Using Q-norm for steepest descent can be regarded as a change of Change of coordinates coordinates, called preconditioning Preconditioner should be chosen to improve the condition number of Q-norm L 2 norm the Hessian in the proximity of the solution In system of coordinates, the Hessian at the solution is Function Gradient (a dream) Descent direction Newton method as optimal preconditioner Another derivation of the Newton method Best theoretically possible preconditioner direction, giving descent Approximate the function as a quadratic function using second-order Taylor expansion Ideal condition number (quadratic function in ) Problem: the solution is unknown in advance Close to solution the function looks like a quadratic function; the Newton Newton direction: use Hessian as a preconditioner at each iteration method converges fast Newton method Frozen Hessian Observation: close to the optimum, the Hessian does not change Compute Newton direction significantly Reduce the number of Hessian inversions by keeping the Hessian from Choose step size using line search previous iterations and update it once in a few iterations Such a method is called Newton with frozen Hessian Update iterate Increment iteration counter 5

6 31 32 Cholesky factorization Truncated Newton Decompose the Hessian Solve the Newton system approximately where is a lower triangular matrix Solve the Newton system A few iterations of conjugate gradients or other algorithm for the solution of linear systems can be used in two steps Forward substitution Backward substitution Andre Louis Cholesky ( ) Such a method is called truncated or inexact Newton Complexity:, better than straightforward matrix inversion Non-convex optimization Iterative majorization Using convex optimization methods with non-convex functions does not guarantee global! There is no theoretical guaranteed global optimization, just heuristics Construct a majorizing function satisfying. Majorizing inequality: for all is convex or easier to optimize w.r.t. Local minimum Global minimum Good initialization Multiresolution Iterative majorization Constrained optimization Find such that MINEFIELD CLOSED ZONE Update iterate Increment iteration counter Solution 6

7 37 38 Constrained optimization problems Generic constrained minimization problem An example Inequality constraint Equality constraint where are inequality constraints are equality constraints A subset of the search space in which the constraints hold is called feasible set A point belonging to the feasible set is called a feasible solution A minimizer of the problem may be infeasible! Feasible set Inequality constraint is active at point if, inactive otherwise A point is regular if the gradients of equality constraints and of active inequality constraints are linearly independent Lagrange multipliers KKT conditions Main idea to solve constrained problems: arrange the objective and constraints into a single function If is a regular point and a local minimum, there exist Lagrange multipliers and such that and minimize it as an unconstrained problem is called Lagrangian for all and for all such that for active constraints and zero for inactive constraints Known as Karush-Kuhn-Tucker conditions and are called Lagrange multipliers Necessary but not sufficient! KKT conditions Geometric interpretation Sufficient conditions: Consider a simpler problem: If the objective is convex, the inequality constraints are convex and the equality constraints are affine, the KKT conditions are sufficient Equality constraint In this case, minimizer) is the solution of the constrained problem (global constrained The gradient of objective and constraint must line up at the solution 7

8 43 44 Penalty methods Penalty methods Define a penalty aggregate where and are parametric penalty functions For larger values of the parameter, the penalty on the constraint violation is stronger Inequality penalty Equality penalty Penalty methods 45 Find and initial value of by solving an unconstrained optimization problem initialized with Set Set Update Solution 8