Lecture 8 Optimization

Transcription

1 4/9/015 Lecture 8 Optimization EE 4386/5301 Computational Methods in EE Spring 015 Optimization 1 Outline Introduction 1D Optimization Parabolic interpolation Golden section search Newton s method Multidimensional Optimization Powell s method Gradients, Hessians, and derivative tests Powell s method Steepest ascent method Newton s method or multiple variables Optimization 1

2 4/9/015 Recall Derivative Tests First order derivatives tell us slope and whether we have reached an etremum or not Second order derivatives tell us slow and whether our etremum is a minimum or a maimum. 0 0 minimum 0 maimum Optimization 3 Parabolic Interpolation Optimization 4

3 4/9/015 Formulation o the Method We it () to a polynomial and then use the irst derivative test to ind the etremum. Step 1 Pick three points that span an etremum and and and Step Fit the points to a polynomial a a a 0 1 a a a a a a a a a Step 3 Use the irst derivative test to ind etremum a0 1 a a 1 0 a a a a 1 e 1 e e a a a Step 4 Ater working through the algebra, we get a inal epression or the etremum e Optimization 5 a a Formulation o Newton s Method Recall that the Newton Raphson method was used to ind the zero o a unction. Starting with an initial guess i at the root, the ollowing equation is iterated until convergence. i i 1 i i We can easily convert this into an algorithm or inding an etremum instead o a zero. Deine an auiliary unction g() that is the irst derivative o (). d g d The auiliary unction g() will have a zero at an etremum o (). This mean we can perorm the Newton Raphson method on g() to ind an etremum o (). g i1 i g i i i1 i i i Optimization 6 3

4 4/9/015 Newton s Method Optimization 7 Golden Section Search Optimization 8 4

5 4/9/015 Step 1 Deine Interval L U We try to pick lower and upper limits, L and H, so that () has only one etremum inside the interval. Optimization 9 Step Evaluate the Function at the Bounds L L H H Optimization 10 5

6 4/9/015 Step 3 Pick Two Intermediate Points 1 and 1 U L d d d R U Optimization 11 L 51 R Step 4 Evaluate the Function at Points 1 and 1 1 Optimization 1 6

7 4/9/015 Step 5a Determine the Position o the Etremum 1 < Etremum is on right side I 1 >, then the etremum is on the let side o the interval between L and. I 1 <, then the etremum is on the right side o the interval between 1 and U. Optimization 13 Step 5b Adjust Points L L R L U L Optimization 14 7

8 4/9/015 Step 5c Determine the Position o the Etremum 1 > Etremum is on let side I 1 >, then the etremum is on the let side o the interval between L and. I 1 <, then the etremum is on the right side o the interval between 1 and U. Optimization 15 Step 5b Adjust Points U U 1 1 R 1 U U L 1 1 Optimization 16 8

9 4/9/015 Step 5c Repeat Until Convergence Converged i: U L tolerance Optimization 17 U L Step 6 Calculate Final Answer We estimate the inal etremum to be at the midpoint o the last interval. e U L Optimization 18 9

10 4/9/015 Algorithm Summary 1. Deine starting bounds L and U. L U. Evaluate the unction at the two bounding points. L L U U 3. Pick two intermediate points, 1 and, using Golden ratio. 51 R U RU L R L U L 4. Evaluate unction at the two intermediate points. 5. Update bounds by identiying position o the etrema. I 1 >, etrema is on let side. U U 1 1 R 1 U U L 1 1 I 1 <, etrema is on right side. L 1 L R L U L 6. Repeat Step 5 until convergence U L tolerance 7. Calculate inal answer 1 1 U L e e e Optimization 19 U L Animation o the Method Optimization 0 10

11 4/9/015 Derivation o Golden Ratio (1 o ) From this picture, we deine three length parameters. L 1 1 U 0 U 1 L L U 0 Recognizing that we want the points on the net iteration to lie on top o points rom the previous iteration, we deine two conditions to ensure this. 0 1 Condition 1 ensures l 1 + l covers entire span Condition ensures the net iteration has the same proportional spacing as the current iteration. Optimization 1 Derivation o Golden Ratio ( o ) Condition 1 ensures l 1 + l covers entire span. Condition ensures the net iteration has the same proportional spacing as the current iteration. L 1 1 U Substitute Condition 1 into Condition to eliminate l Deine the Golden ratio as R = l /l 1. 1 R R R R We solve or R using the quadratic ormula. b b 4ac R a 1 We pick the positive root to keep R positive Optimization R 11

12 4/9/015 Powell s Method Optimization 3 Conjugate Direction 1 a b Suppose we start at two dierent points, a and b, and use 1D optimizations along parallel directions to arrive at the two etremas 1 and. The direction o the line connecting 1 and is called the conjugate direction and is directed toward the maimum. Optimization 4 1

13 4/9/015 h 1 opt 0 Pick a starting point 0 and two dierent starting directions h 1 and h. Optimization 5 h 1 opt 0 1 Staring at 0, perorm a 1D optimization along h 1 to ind etremum 1. Optimization 6 13

14 4/9/015 h h 1 opt 0 1 Staring at 1, perorm a 1D optimization along h to ind etremum. Optimization 7 h 3 h h 1 opt 0 1 Deine h 3 to be in the direction connecting 0 to. Optimization 8 14

15 4/9/015 h 3 3 h 1 h opt 0 1 Staring at, perorm a 1D optimization along h 3 to ind etremum 3. Optimization 9 h h 3 3 h 1 h opt Staring at 3, perorm a 1D optimization along h to ind etremum 4. Optimization 30 15

16 4/9/015 h 3 h h 3 3 h 1 h 5 opt Staring at 4, perorm a 1D optimization along h 3 to ind etremum 5. Optimization 31 h 3 h h 3 3 h 1 h 5 opt h 4 Deine h 4 to be in the direction connecting 3 to 5. Optimization 3 16

17 4/9/015 h 3 h h 3 h 1 h opt This last 1D optimization is guaranteed to ind the maimum because Powell showed that h 3 and h 4 are conjugate directions. 0 1 h 4 Staring at 5, perorm a 1D optimization along h 4 to ind etremum opt. Optimization 33 Algorithm Summary 1. Pick a starting point 0 and two dierent starting directions h 1 and h.. Staring at 0, perorm a 1D optimization along h 1 to ind etremum Staring at 1, perorm a 1D optimization along h to ind etremum. 4. Deine h 3 to be in the direction connecting 0 to. 5. Staring at, perorm a 1D optimization along h 3 to ind etremum Staring at 3, perorm a 1D optimization along h to ind etremum Staring at 4, perorm a 1D optimization along h 3 to ind etremum Deine h 4 to be in the direction connecting 3 to Staring at 5, perorm a 1D optimization along h 4 to ind etremum opt. Optimization 34 17

18 4/9/015 Convergence Powell s method is quadratically convergent and etremely eicient. I iterated, it will converge in a inite number o iterations i the unction is quadratic. Most unctions are nearly quadratic near their etrema. Optimization 35 Gradient Methods Optimization 36 18

19 4/9/015 Scalar Field Vs. Vector Field Scalar Field,, magnitude,, y Vector Field, v, y yz magnitude, yz, direction, yz, Optimization 37 Isocontour Lines Isocontour lines trace the paths o equal value. Closely space isocontours conveys that the unction is varying rapidly. Optimization 38 19

20 4/9/015 Gradient o a Scalar Field (1 o 3) We start with a scalar ield y, Optimization 39 Gradient o a Scalar Field ( o 3) then plot the gradient on top o it. Color in background is the original scalar ield. y, Optimization 40 0

21 4/9/015 Gradient o a Scalar Field (3 o 3) The gradient will always be perpendicular to the isocontour lines. Optimization 41 The Multidimensional Gradient The 3D gradient can be written as,,,,,,,, yz ˆ yz yz a ˆ yz ˆ ay az y z When more dimensions are involved, we write it as 1 N 1 N Optimization 4 1

22 4/9/015 Properties o the Gradient 1. It only makes sense to calculate the gradient o a scalar ield*.. points in the direction o the maimum rate o change in. 3. at any point is perpendicular to the constant surace that passes through that point. 4. The gradient points toward big positive numbers in the scalar ield. * The gradient o a vector ield is a tensor called the Jacobian. It is commonly used in coordinate transormations, but is outside the scope o this course. Optimization 43 Numerical Calculation o the Gradient We may not always have a closed orm epression or our unction so it may not be possible to calculate an analytical epression or the gradient. We this is the case, we can calculate the gradient numerically N N N N N N This can be a very epensive computation! Optimization 44

23 4/9/015 Derivative Tests in Multiple Dimensions Suppose we have a D unction (,y). 0 y 0 Does this indicate a minimum? No! Optimization 45 The Hessian The Hessian describes curvature o multiple variable unctions. We will use it to determine whether we are really at a maimum or minimum. The Hessian is deined as N H 1 N N1 N N Optimization 46 3

24 4/9/015 Revised Derivative Tests In two dimensions, the Hessian is H y y y The revised derivative test using the Hessian is H determinant y y I H I H 0 and 0 and 0, then, y has a local maimum. 0, then, y has a local minumum. I H 0 then, y has a saddle point. Optimization 47 Steepest Ascent Method Optimization 48 4

25 4/9/015 Steepest Ascent Method We wish to minimize the number o times the gradient is calculated. Let s calculate it once and then move in that direction until () stops increasing. At this point, we reevaluate the gradient and repeat in the new direction. Algorithm 1. Pick a starting point.. Calculate the gradient at this point: g = () 3. I the gradient is zero, we are done! 4. Otherwise, move in small increments in the direction o g until () stops increasing: = + g Note: i we think o searching along this direction like a 1D optimization, we can improve eiciency greatly. 5. Go back to Step. Optimization 49 Newton s Method or Multiple Variables Optimization 50 5

26 4/9/015 Newton s Method with Multiple Variables (1 o ) We can etend Newton s method to multiple variables using the Hessian. We can write a second order Taylor series or () near = i. 1 H T T i i i i i i At an etremum, ()= 0,. To ind this point, we derive the gradient o the above epression. H i i i Optimization 51 Newton s Method with Multiple Variables ( o ) We set our gradient to zero and solve or. i i i ihii0 Hii i 1 i Hi i 1 H H 0 i i i From this, our update equation or the Newton s method is 1 H i1 i i i Optimization 5 6