Methodological Foundations of Biomedical Informatics (BMSC-GA 4449) Optimization

Transcription

1 Methodological Foundations of Biomedical Informatics (BMSCGA 4449) Optimization

2 Op#miza#on A set of techniques for finding the values of variables at which the objec#ve func#on amains its cri#cal (minimal or maximal) values, within given constraints on the domain. Classes of techniques: Closed form Numerical Local vs. global

3 A simple example f(x) = 1/3 x 3 x, on [ 2.5, 2.5] d/dx f(x) = x 2 1 Has two roots x 1 = 1, x 2 = 1 d 2 /dx 2 f(x) = 2x At x 1 f(x) is posi#ve => x 1 is a (local) minimum At x 2 f(x) is nega#ve => x 2 is a (local) maximum

4 A simple example f(x) x We have only discovered local minima We failed to check the boundaries for possibility of special points In fact, x 3 = 2.5 is the global minimum and x 4 = 2.5 is the global maximum.

5 Constrained Op#miza#on The previous example was a case of free optima optimization, because it did not place any restrictions on the domain of the objective function. If there are restrictions on the domain, the optimization is called constrained, and such are the optima we are looking in these cases.

6 Constrained Op#miza#on Graphically, the difference between the free optima and the constrained optima can be shown as: Constrained maximum Free maximum constraint

7 Constrained maximum Free maximum constraint The free optima occurs at the peak of the surface. If we specify a specibic relationship between variables x 1 and x 2 (a constraint) then the search for an optimum is restricted to a slice of the surface. The constrained maximum occurs at the peak of the slice.

8 Approached to constrained op#miza#on Subs#tu#on method Lagrange mul#pliers method

9 Subs#tu#on Method Suppose we would like to op#mize the func#on f(x 1, x 2 ) = 5 x 1 x 2. If there no constraints on x1 and x2, the func#on can take infinite values, so it does not have a maximum. Now suppose we have an addi#ve restric#on on the independent variables: 100 = 2 x 1 + x 2. We can use the constraint to express one variable in terms of the other: x 2 = x 1

10 Subs#tu#on Method Now subs#tute the solu#on into the objec#ve func#on f(x 1, x 2 ) = g(x 1 ) = 5 x 1 (100 2 x 1 )= 10 x x 1 We are lec with a free maxima problem, which we can solve in the usual way. Compute the deriva#ve d/dx 1 f = 20 x Solve for zero of the deriva#ve 20 x = 0! 500 = 20 x 1! x 1 = 25 Check whether this cri#cal point is a maximum or a minimum d 2 /dx 1 2 f = 20 < 0! maximum Subs#tute back to find the solu#on for x 2 : x 2 = x 1 = 100 2! 25 = 50

11 Subs#tu#on method Although subs#tu#on method is straighdorward it is hard to generalize to arbitrarily complex constraints. It may be hard or impossible to use the constraints to express one variable in terms of the other.

12 Lagrange mul#pliers Equality constraints: op#mize f(x) subject to g i (x)=0, 1 i k g can be arbitrary func?ons of the independent variables Method of Lagrange mul#pliers: convert to a higher dimensional problem Minimize the augmented func#on Over q(x, λ) = f (x)+ λ i g i (x) (x 1 x n ;λ 1 λ k )

13 Lagrange Mul#pliers Given the augmented function, the Birst order condition for optimization is as follows: q(x, λ) = f (x, λ)+ x 1 x 1! q(x, λ) = f (x, λ)+ x n x 1 λ 1 q(x, λ) = g 1 (x) = 0! λ k q(x, λ) = g k (x) = 0 λ i λ i # g i (x) = 0 x 1 g i (x) = 0 x n & Simultaneous system of equations

14 Lagrange Mul#pliers Second order condion are determined from the Hessian matrix evaluated at cri#cal points: " H(x, λ) = # 2 x 2 q 2 λ x q 2 x λ q 2 λ 2 q " = & # 2 x 1 2 q! 2 x 1 x n q 2 x 1 λ 1 q! 2 x 1 λ k q! "!!! 2 x n x 1 q! 2 x n 2 q 2 x n λ 1 q! 2 x n λ k q 2 λ 1 x 1 q! 2 λ 1 x n q 2 λ 1 2 q! 2 λ 1 λ k q!!! "! 2 λ k x 1 q! 2 λ k x n q 2 λ k x 1 q! 2 λ k 2 q &

15 Lagrange Mul#pliers If at the cri#cal point is H > 0, the point is a local maximum If at the cri#cal point is H < 0, the point is a local maximum

16 Lagrange Mul#pliers: Example Using previous example q(x 1, x 2, λ) = 5x 1 x 2 + λ(100 2x 1 x 2 ) Where the constraint 100 = 2 x 1 + x 2 is expressed as g(x 1, x 2 )=100 2 x 1 x 2 = 0 This gives rise to the following system of equa#ons in 3 unknowns q x 1 = 5x 2 2λ = 0 q x 2 = 5x 1 λ = 0 # q λ =100 2x 1 x 2 = 0 &

17 Lagrange Mul#pliers: Example Solve the system, e.g. by Gaussian elimina#on q x 1 = 5x 2 2λ = 0 q x 2 = 5x 1 λ = 0 q λ =100 2x 1 x 2 = 0 # & Ay = ( ) +, 1 x 1 x 2 λ ( ) +, = ( ) +, ( ) +, x 1 x 2 λ ( ) +, = ( ) +, x 1 x 2 λ ( ) +, = ( ) +,

18 Lagrange Mul#pliers: Example The hessian matrix for this is example is " H(x 1, x 2, λ) = # & Note that it does not depend on any of the independent variables (why?). The determinant is posi#ve H = ( 2) = ( 1) 3 5 (5 0 ( 1) ( 2))+ ( 1) 2 ( 2) (5 ( 1) 2 0) = = 20

19 Convexity A func#on is convex in a region [a, b] if for any two points x 1, x 2 in [a, b] and t in [0, 1] f(t x 1 + (1 t) x 2 ) t f(x 1 ) + (1 t) f(x 2 ) f(x) f(t x 1 + (1 t) x 2 ) t f(x 1 ) + (1 t) f(x 2 x 1 x 2 x

20 Fixed point itera#ons Suppose we have a func#on f(x) Given x 0, define a sequence x 0 x 1 = f(x 0 ) x 2 = f(x 1 )=f(f(x 0 )) If the sequence converge, then the f has a fixed point x = f(x ) Idea: design func#ons with fixed points corresponding to the minimum f(x) x

21 Convergence Rate at which a fixed point algorithm approaches its fixed point lim f (x ) f (x ) k = µ k x k f (x ) q At q=1, 0<μ<1, linear convergence At q=1, μ=1, sublinear convergence E.g. logarithmically At q=1, μ=1, superlinear convergence q=2, μ>0, quadra?c convergence q=3, μ>0, quadra?c convergence

22 Classification of Optimization Methods Use function values only (simplex method) Use function + gradient (most descent methods) Use function + gradient + Hessian (Newton)

23 Newton s Method

24 Newton s Method

25 Newton s Method

26 Newton s Method

27 Newton s Method Suppose you want to minimize f(x), We use the Newton s method to find the roots of its deriva#ve, i.e. g(x)=f (x) = 0. At each step: x k+1 = x k g(x ) k g "(x k ) = x f "(x ) k k f "" (x k ) Requires 1 st and 2 nd deriva#ves Quadra#c convergence

28 Multidimensional Optimization

29 Mul# Dimensional Op#miza#on Important in many areas Fiqng a model to measured data Finding best design in some parameter space Hard in general Weird shapes: mul#ple extrema, saddles, curved or elongated valleys, etc. Can t bracket

30 Newton s Method in Replace 1 st deriva#ve with gradient, 2 nd deriva#ve with Hessian Mul#ple Dimensions # H = f (x 1,, x n ) # f = f x 1! f x n & ( ( ( ( 2 f 2 " 2 f x 1 x n x 1! #! 2 f x 1 x n " 2 f 2 x n & ( ( ( (

31 Newton s Method in Mul#ple Dimensions Replace 1 st deriva#ve with gradient, 2 nd deriva#ve with Hessian So,! x =! x H! ( x ) f! ( x 1 k + 1 k k k Tends to be extremely fragile unless func#on very smooth and star#ng close to minimum )

32 Shortest distance between two ellipses

33 Shortest distance between two ellipses

34 Shortest distance between two ellipses