Example - Newton-Raphson Method

Transcription

1 Eample - Newton-Raphson Method We now consider the following eample: minimize f( Since f ( and f ( we form the following iteration: + n 3 ( n 3 3( n n ( n 2 n ( n 2 6 n 3 n 2 n We can now try the iteration for different initial guesses. initial guess: ( 0 0 2( n 2 n n 2 initial guess: initial guess: ( ( 0.9 ( 0 ( stationary point ( 0 ( stationary point (.029 ( initial guess: ( 0.0 ( ( stationary point We can now evaluate f( at the two stationary points we ve found: f( 0 0, f( 4 and since f ( 3 > 0 local minimum therefore * local minimum. In fact you can convince yourself that it is the global minimum. (Note that the global maimum is at * ±. (.0

2 Interval Reduction Using Function Samples Only initial bracketed interval I 0 [a, b] f f 2 a c d b consider f : f ( d < f ( c I [,] cb, 2 consider f 2 : f 2 ( d > f 2 ( c I [,] ad. a L m u b Algorithm: Interval Halving Search. input a, b, min // input the initial interval 2. set: b a, m a + 2, f m f ( m 3. while > min repeat 4. set: L a + 4, f L f ( L 5. u b 4, f u f ( u 6. if f L < f m then 7. set b m, m L, f m f L, b a 8. else if f u < f m 9. set a m, m u, f m f u, b a 0. else. set a L, b u, b a 2. end if 3. end if 4. end while 5. output [a, b] // final interval size is less than min After 2n+ function evaluation the interval is reduced by I 2n + I 0 ( 2 n, n 23,, where I o is the size of the initial interval.

3 Fibonacci Search Technique Question How should we pick N successive points in an interval to perform function evaluations such that the final reduced interval is the smallest possible irrespective of the properties of f(? initial interval: [0, 2] new interval: [0, +δ] 0 +δ 2 Best location for two function evaluations N 3 initial interval: [0, 3] final interval: [, 2] 0 +δ 2 3 N 4 initial interval: [0, 5] final interval: [4, 5] δ 5 N 5 initial interval: [0, 8] final interval: [4, 5] δ 5 8 Eamples of optimal sampling locations Fibonacci sequence of numbers: F 0 F, F i F i + F i 2, i 2 (. Given an interval of proportion F i and choosing two sample points at F i 2, F i.

4 initial interval: [0, F i ] new interval: [0, F i- ] or [F i-2, F i ] F i- 0 F i-2 F i The new interval will be either Fibonacci type sampling [ 0, F i ] or [ F i 2, F i ] the first has size F i, the second has size F i F i 2 F i and therefore they are the same size. The net point is chosen as either F i 3 or F i F i 3. The interval reduction is equal to I N I 0 F N after N function evaluations. It has been shown that this is the best possible reduction for a given number of function evaluations. At each step the reduction is equal to F i F i +. The drawback of this procedure is that you must start with F N and work backwards down the sequence. Golden Section Search Given a normalized interval [ 0, ] where should two points and 2 be chosen such that: size of reduced interval is independent of function. 2 only one function evaluation per interval reduction. if f 2 < f then new interval is [0, ] 0 2 if f < f 2 then new interval is [ 2, ] 0 2 Golden section interval reduction, initial interval is [0, ].

5 Therefore by condition above 0 2 or 2 and by condition from τ θ τ τ 2 where we call τ the golden section and θ the golden mean. After each function evaluation and comparison θ of the interval is eliminated or τ of the original interval remains. Thus after N function evaluations an interval of original size L will be reduced to a size Lτ N Algorithm: Golden Section search. input a, b, tol 2. set c a + ( τ ( b a, F c Fc ( 3. d b ( τ ( b a, F d Fd ( 4. for K 2 repeat,, 5. if F c < F d then 6. set a k + a k, b k + d k, d k + c k 7. c k + a k + + ( τ ( b k + a k + 8. F d F c, F c Fc ( k + 9. else 0. set a k + c k, b k + b k, c k + d k. d k + b k + ( τ ( b k + a k + 2. F c F d, F d Fd ( k + 3. end 4. end until b k + a k + < tol Notes: iterate until interval < tol

6 Polynomial Interpolation Methods In an interval of uncertainty the function is approimated by a polynomial and the minimum of the polynomial is used to predict the minimum of the function. Quadratic Interpolation If F ( p 2 + q + r is the interpolating quadratic we need to find the coefficients p, q, and r given the end points of the interval [ ab, ] and c such that a < c< b with F a Fa ( F b Fb (, F c Fc (. The coefficients satisfy the matri equation a 2 a b 2 b c 2 c p q r F a F b F c which can be easily solved analytically. Once we have the coefficients we need to find the minimum of F( and thus d F ( g ( * 2p d * + q 0 and therefore we only need the q and p coefficients. These can be solved using Cramer s rule: * q 2p p F a a -- F b b F c c F a ( b c + F b ( c a + F c ( a b q a 2 F a -- b 2 F b c 2 F c F a ( b 2 c 2 + F b ( a 2 c 2 F c ( a 2 b and therefore * q p -- F a( b2 c 2 + F b ( c 2 a 2 + F c ( a 2 b F a ( b c + F b ( c a + F c ( a b

7 Notice that * must lie in [ ab, ] if F c < F a and F c < F b bracketing interval. Also note that f ( * 2p( * and therefore if p( * > 0 local minimum of quadratic whereas if p( * < 0 local maimum. We now evaluate f( * and compare it to f( c in order to reduce the interval of uncertainty. f( f(* > f c k a k * c k b k a k+ c k+ b k+ f(* > f c k new interval is [a k+, b k+ ] and new c value is c k+ f( f(* < f c k a k * c k b k a k+ c k+ b k+ f(* < f c new interval is [a k+, b k+ ] and new c value is c k+ k

8 Algorithm: Quadratic interpolation search. input a, b, c, tol, ftol 2. set F a Fa (, F b Fb (, F c Fc ( 3. for k,2,... repeat 4. set 5. * -- F a b ( k c k + F b ( c k a k + F c ( a k b k F a ( b k c k + F b ( c k a k + F c ( a k b k F F ( * 6. if * > c k and F < F c then 7. set a k + c k, b k + b k, c k + * 8. F a F c, F c F 9. else if * > c k and F > F c then 0. set a k + a k, b k + *, c k + c k. F b F 2. else if * < c k and F > F c then 3. set a k + *, b k + b k, c k + c k 4. F a F 5. else 6. set a k + a k, b k + c k, c k + * 7. F b F c, F c F 8. end 9. end until ( b k + a k + < tol or ( Fc ( k Fc ( k + Fc ( k < ftol Notes: stop when interval reduced to tol or when relative change in function value < ftol

9 Bounding Phase Question: how do we determine this initial interval? In general we perform a coarse search to bound or bracket the minimum *. This is also called interval location. We will discuss two types of interval location: function comparison (Swann s Bracketing Method; and 2 polynomial etrapolation (Powell s Method. Both of these methods are heuristic in nature but are about the best we can provide in terms of an algorithm. We start with a description of Swann s bracketing method. Function comparison: Swann s Bracketing Method Given an initial step length and starting point a we check the point a distance away on the right side of a. If f( a + < fa ( then we will move to the right. If f( a + > fa ( then we will move to the left (lets say we move to the right. We now magnify the step size by a magnification factor, say 2, and evaluate the function at the point 2 away from the latest point. As soon as any new function evaluation shows an increase we can say that the last two intervals provide a bound or interval of uncertainty for our minimum. F( move to right until function evaluation increase, then last two intervals are interval of uncertainty a b a 2 b 2 a 3 b 3 a 4 b 4 Eample of Swann s bracketing method, magnification factor is 2.0.

10 Algorithm: Swann s Bracketing Method - based on a heuristic epanding pattern.. input: 0, // initial point and step size 2. set: l 0 // lower test point. 3. u 0 + // upper test point. 4. f l f ( l // lower function value 5. f u f ( u // upper function value 6. f 0 f ( 0 // central function value 7. i // epanding eponent 8. if f l f 0 f u // case : move to the right 9. loop-up: i i+ 0. set: l 0, 0 u, f l f 0, f 0 f u. u u + 2 i // shift up in by a magnified amount 2. f u f ( u 3. if f u < f 0 go to loop-up 4. else output ( l, u 5. if f l f 0 f u // case 2: move to the left 6. loop-down: i i+ 7. set: u 0, 0 l, f u f 0, f 0 f l 8. l l 2 i // shift down in by a magnified amount 9. f l f ( l 20. if f l < f 0 go to loop-down 2. else output ( l, u 22. if f l f 0 f u // case 3: initial interval is bracket 23. output ( l, u 24. if f l f 0 f u error: non-unimodal function

11 Polynomial Etrapolation: Powell s Method we use polynomial etrapolate from the initial starting point. Given a starting point a, and two more points apart we etrapolate a polynomial through them (we will use a quadratic function. Now in this case the fitted quadratic may have either a maimum or minimum and this may be determined by evaluating the second derivative. As before if the polynomial is represented as F ( p 2 + q + r then the second derivative is given as F ( G ( 2p and in terms of the three function values F a, F b, F c, evaluated at a, b, and c we can evaluate p as ( c bf p a + ( a cf b + ( b af c. ( b c ( c a ( a b Now how do we use this to find an interval? Starting from three points, a, c, and b, a apart we fit a quadratic to these points, say F(. If we find that this quadratic has a maimum, that is p < 0, then the net point we take is ma away from the smallest function value. (See figure below where for sake of an eample we are assuming that we move to the right. f( case : F( has a maimum, i.e. p < 0 F( ma * a c b a 2 c 2 b 2 Etrapolated polynomial has a maimum therefore use the ma. We discard the point with smallest function value and take new point ma away from b. Discarding the lowest function value gives us more of a chance of fitting a polynomial with a minimum in the net iteration of the algorithm.

12 If the polynomial does have a minimum then we either use the minimum of the polynomial as shown in case 3 or we again use ma if the minimum is farther than ma from b as shown in case 2. f( case 2: F( has a minimum, i.e. p > 0 F( ma a c b a 2 c 2 b 2 * Etrapolated polynomial has a minimum but it is too far away so use ma. f( case 3: F( has a minimum, i.e. p > 0, and * - b < ma F( ma a c b * a 2 c 2 b 2 Etrapolated polynomial has a minimum and it is closer than ma from b. Algorithm: Interval location by Powell s Method. input a,, ma 2. set c a +, F a Fa (, F c Fc ( 3. if Fa > Fc then 4. set b a + 2, F b Fb ( 5. forward true 6. else 7. set b c, c a, a a 8. F b F c, F c F a, F a Fa ( 9. forward false

13 0. end. for K 2,, repeat 2. set p ( c K b K F a + ( a K c K F b + ( b K a K F c ( b k c K ( c K a K ( a K b K 3. if p > 0 then 4. set * -- b ( K c K F a + ( c k a K F b + ( a K b K F c 2( b K c K F a + ( c K a K F b + ( a K b K F c 5. end if 6. if forward then // moving forward 7. if p 0 then // quadratic has a maimum 8. set a K + a K, b K + b K + ma 9. c K + c K, F b Fb ( K else 2. if * b K > ma then // quadratic minimum is too far 22. set b K + b K + ma 23. else // quadratic minimum is O.K. 24. set b K + * 25. end 26. set a K + c K, c K + b K 27. F a F c, F c F b, F b Fb ( K end 29. else // moving backward (forward false 30. if p 0 then // quadratic has a maimum 3. set a K + a K ma, b K + b K 32. c K + c K, F a Fa ( K else 34. if a K * > ma then // quadratic minimum is too far 35. set a K + a K ma 36. else // quadratic minimum is O.K. 37. set a K + * 38. end 39. set b K + c K, c K + a K 40. F b F c, F c F a, F a Fa ( K + 4. end // if 42. end 43. end until F c < F a and F c < F b p 0