Rotations in Curved Trajectories for Unconstrained Minimization

Transcription

1 Rotations in Curved rajectories for Unconstrained Minimization Alberto J Jimenez Mathematics Deartment, California Polytechnic University, San Luis Obiso, CA, USA 9407 Abstract Curved rajectories Algorithm (CA is a acage for the minimization of unconstrained functions of several variables with intervals on the variables he core algorithm is novel in that stes may follow olynomial sace curves instead of straight lines he sace curves result from truncations of a aylor series exansion of the Gradient inverse function When the series is convergent and the current guess of the solution is far, aroriate rotations of the sace curve may allow further rogress at a ste Imrovements are significant for some functions, reducing the number of Hessians required or imroving the accuracy of the solution his idea may be useful to other minimization acages Keywords Curved rajectory Minimization, Rotations of Sace Curves, Unconstrained Otimization Introduction We focus on the roblem: n Minimize f ( x, f : R R, ( n x R where f is a nonlinear continuous real scalar function, with continuous derivatives, of n variables reresented by the vector x We write the derivatives simly by: f x f x f x f x (4 (, (, (, (, etc is called the Gradient, which is is called the Hessian, which is an n n symmetric matrix he higher derivatives are tensors of higher dimension A minimum oint x * is a critical oint of the function satisfying: he first derivative f ( x an n vector function; f ( x f ( x* = 0 ( A necessary condition for a minimum is that the Hessian f ( x* must be ositive semi-definite at the critical oint (eg the Hessian has no negative eigenvalues at x *, otherwise the critical oint is not a minimum A minimum oint may not be a global minimum his aer addresses calculating a local minimum oint only yical methods for calculating a minimum start from a guess x 0 hen a direction v is selected at the th iteration ste such that the directional derivative is negative along v Now a search along this direction is made to find a suitable value of > 0 that yields f ( x + v << f ( x One can aroximate by solving the one-dimensional * Corresonding author: ajjimene@calolyedu (Alberto J Jimenez Published online at htt://journalsauborg/algorithms Coyright 0 Scientific & Academic Publishing All Rights Reserved scalar roblem, a line search: Minimize f ( x + v ( his line search can require significant comutation time and is usually unnecessary Instead, a limited number of trial values are tested until one is selected that roduces sufficient reduction in the function value his scalar minimization is a well nown roblem (see for examle [] Section 6, or [7] Chater he scalar search yields and the next guess of the solution is x + = x + v, and the iterations continue until the Gradient is as close to zero as desired, or there is lac of rogress Recent wor establishes a rust Region (see [] or [] for calculating x + using a quadratic model of the function Convergence (see for examle [8] age 5, or [7] age requires sufficient descent at each ste (in addition to other requirements, that is: f ( x+ = f ( x + v << f ( x (4 Brief Descrition of CA Algorithm A core comonent of the CA otimization acage is the addition of higher order terms to Newton's method (see [5] and [6] he higher order terms are derived by exanding the unnown minimum x* equal to the Gradient inverse function in a aylor series, and retaining u to two additional terms after the Newton term he derivation begins from ( f ( x = z x = f ( z After differentiation and much tedious algebra, the first four terms of the infinite aylor series exanded at z and evaluated at zero giving x* = ( f (0 are

2 where the vectors d, d, and d 4 are aroximated from the Hessian and Gradient at x, and from Gradients in the neighborhood of x by solving the linear systems: [ ] f ( x d = f ( x, (5 (7 he constant ε is a small ositive constant his results in an iteration, and at each ste a > 0 is selected such that x = x + + h ( satisfies (4, where h ( is a vector function reresenting the trajectory direction selected at the th ste he number of terms to use at each iteration deends on tests for convergence of the series he forms of the vector function are given by nd order trajectory h( [ ] x* = x d [ ] = d (Newton s Method rd order trajectory h( = (05 ( d d 4 th order trajectory + ( d d ( ( d ( d d 4, + 6 f ( x d = f ( x d d [ ] ( h( = 6 + d / 6 ( d d (6 4 When rd order or 4 th order is selected, the vector function h( is a olynomial curved trajectory, or sace curve, for > 0 with h(0 = 0 his algorithm converges raidly in the neighborhood of the solution when the Hessian is ositive semi-definite When far from the solution, the Hessian can be indefinite and h( may not be a descent direction for > 0 An earlier aer by this author [4] describes the factorization of the Hessian, its modification when it is not ositive definite, and storing it in sarse form In this aer we show that an aroriate rotation of the curved trajectory can often mae additional rogress when far away from the solution his rotation often trades additional Gradient and + ε [ f ( x ε d ( ε f ( x ] f x d f x d d f x d d d 6 ( 4 ( = ( ( 4 f x d d + f x ε ( ε ε ( ε ( Function evaluations to reduce the number of Hessian evaluations his tradeoff would be beneficial when the evaluation of the Hessian is very costly In some cases while there is no reduction in the number of Hessian evaluations, imrovements in the accuracy of the solution for badly scaled roblems have been observed Exloring Rotations For A wo Variable Problem We begin by focusing on a two variable roblem A rotation of the CA trajectory curve is robably best described by an angle θ, which when measured ositive counterclocwise, inserts a matrix multilying the iteration trajectory as follows: Where x = x + + R( θ h(, sinθ R( θ = sinθ Observe that when θ = 0, R(0 is the unit matrix Adding this rotation comlicates the th ste, if we were to search for the best and the best θ simultaneously Perhas this roduces the best overall results for some algorithms For this aer, since we are adding rotations to an already otimal algorithm, we first calculate the best using the algorithm as is hen we consider adding a rotation to the trajectory using the calculated to determine if greater rogress can be made towards the solution along a rotated trajectory; this can be viewed as rotating the trajectory and calculating a new value for along the rotated trajectory Before describing the calculation of a roductive angle of rotation, let us loo at an examle Consider Rosenbroc s [0] well nown banana-shaed valley function: f ( x, y 00( y x ( x = +, (8 with the usual starting guess oint, and solution: * x0 = [ x y] = [ ], x = [ ], (9 he CA acage first iteration is given by: 56 x=, 4, 0 f = f = f = ( d, d, d = = = he first iteration vector function h( is given by: ( ( h( = (06979 (

3 he CA algorithm calculates = 5, yielding: x = [ ], f = 59 Figure shows contours of f (asted from Male and this curved trajectory his figure shows Gradient vectors along the trajectory he trajectory angle with the Gradient vector lays a significant role in the imlementation of rotation as shown later Figure shows Rosenbroc s function value along this trajectory as a function of x 0 f = 4 f = 4 Original trajectory f = 4 f = 05 A rotated trajectory x x 0 f = 4 f = 4 x f = 05 Figure Contours of Rosenbroc [0] Banana Shaed Valley showing the original curved trajectory from the starting oint x 0 = (-,, and a rotated trajectory f = 4 Figure Contours of Rosenbroc [0] Banana Shaed Valley showing the first curved trajectory from the starting oint x 0 = (-, And showing x for = 5 Also shown are Gradient vector directions at some oints along the trajectory = 0, f = 4 = 5, f = 59 4 Calculating Angle Of Rotation For A wo Variable Problem After calculating the best = at the th ste with no rotation (this means θ = 0, we ursue calculating a suitable value for θ = θ that allows for further rogress along the rotated trajectory A general strategy is that calculating a roductive rotation angle should not require calculation of the Hessian he goal is to reduce the number of overall Hessians required, and using a new Hessian to determine a rotation angle would unliely hel this goal Using the Function and Gradient at = is reasonable since we need that to move on even without rotation Using an additional Function evaluation or two in the neighborhood of = also seems reasonable, but no additional Gradients will be used in the calculation of a rotation angle here are three useful angles that can be calculated We begin the derivations with the objective function as a function of θ and : f ( x = f ( θ, = f ( x + R( θ h( Figure Rosenbroc [0] Banana Shaed Valley function along the first ste trajectory for the CA algorithm as a function of he motivation for rotations is that the Gradient oints in the direction where the trajectory should be rotated to obtain smaller values of the function being minimized Given that the trajectory is a truncation of an infinite series, it is conjectured that a roductive rotation accounts for some of the excluded terms Figure shows the original trajectory, and a more roductive rotated trajectory beyond the ste oint And the artial derivative with resect to θ is given by: he above equation simlifies to: at = sin cos h ( θ θ = f R h = θ ( x x sinθ h ( = h ( h ( θ x x h ( + h ( sin θ x x And at θ = 0 and =, this artial derivative value indicates the imortance of θ :

4 = h ( h ( θ θ = 0 x x At the minimum with resect to θ, this derivative is zero, but calculating this θ * f for = 0 is not easy because the θ Gradient is changing with θ and we want to avoid more Gradient evaluations With the oor assumtion that the Gradient does not change much, we obtain a oor aroximation of θ * which we label θ r : h ( h ( x x π π θr = tan, θr h ( + h ( x x his θ r overshoots (it is larger than the actual minimum, since the Gradient terms are becoming smaller with a roductive rotation angle As shown later, we use a fraction of it as art of calculating an initial guess of a roductive θ A second useful angle is defined as θ, and this is the angle at = where the artial derivative with resect to is zero given by: sinθ h ( = f R h = ( x x sinθ h ( he above equation simlifies to: = h ( + h ( x x = h ( h ( sin θ x x At the minimum with resect to, the derivative is zero, and again with the oor assumtion that the Gradient does not change much, we obtain: h ( + h ( x x π π θ = tan, θ h ( h ( x x A third imortant angle is defined as θ G, and this is the angle between the trajectory and the Gradient at θ = 0 and = his angle is between the tangent vector to the trajectory, h (, and the Gradient he angle between two vectors is: f ih ( θg = cos, 0 θg π f h ( yically at =, the trajectory is no longer a descent direction and therefore θ G exceeds 90 degrees Recall that descending trajectories must have less than 90 degree angles with the Gradient (see for examle [9] ages 0- And the sign of θ I is equal to the sign of When the CA iteration was successful with, then we set θ = θ I Otherwise evaluate the objective function along the rotated trajectory with θ = θ I and = and then use a quadratic fit to obtain an imroved θ J When this θ J is significantly larger than the initial theta, further imrovement is obtained with another function evaluation along the rotated trajectory with θ = θ J and a second quadratic fit hus u to two function evaluations may be required to obtain a roductive theta, and some stes require none hese calculations yield the roductive rotation angle of 0885 radians shown in Figure for Rosenbroc function with no additional function samles he angles described above (in radians are: θ r = 0707, θ = and θ π / = he artial derivative is G Figures 4 and 5 show the rotation angle contours and a D grah for Rosenbroc s Banana function versus θ and Figure 4 Contours of the rotation angle θ for Rosenbroc [0] Banana Shaed Valley function along the first ste trajectory for the CA algorithm as a function of f θ θ = 0 = 859 θ = 0 θ = 0 θ = 0 θ = 0 θ θ = 0 With these three angles, we select an initial guess for θ to be the following: π π θi = 04max( θg, θ, min( θr, θr

5 f x 0 f = 4 f = 4 Current oint with = 5 on original trajectory he new better oint on rotated trajectory is with = 67 x f = 05 f = 4 Closest oint on rotated trajectory is with = 544 Figure 5 A D grah of Rosenbroc [0] Banana Shaed Valley function along the first ste trajectory for the CA algorithm as functions of and θ Figure 6 Contours of Rosenbroc [0] Banana Shaed Valley showing the original curved trajectory and rotated trajectory and starting guess to search for a better oint leading to = 67 After imlementing rotations in the CA acage, and testing hundreds of functions, it was determined that rotations should not be attemted unless the following is satisfied: CA trajectory was the result of a converging series (successive terms have smaller norm, Hessian is ositive definite when the order selected is two (Newton's direction, Current oint is far from the solution (Gradient norm is greater than, CA s th ste wored well ( was not too small, is not too small (greater than 0005 times Gradient norm, θ = θ= 0 Calculated θ is not too small (greater than 000, Function value on rotated trajectory is smaller than on the original trajectory he last item to discuss is the search for a new = along the rotated trajectory We want to start searching at the value on the trajectory which corresonds to the oint closest to the oint for on the non-rotated trajectory his is done by calculating smallest distance between the two trajectories without needing any objective function evaluations Figure 6 shows the results for Rosenbroc s function 5 More han wo Variable Problems Including more variables into rotating the trajectory starts by determining the most imortant coordinates to include in a rotation calculation We argue that they are the coordinates that imact the Gradient angle with the trajectory the most his means sorting the dot roduct terms of f h ( i Initially we only focused on the terms that are ositive in the dot roduct thining those coordinates are the ones maing the angle the largest However after exerimentation, sorting on the absolute value of the dot roduct terms roved to be the best determinant of the most imortant coordinates Also exerimentally we determined that terms that are smaller than 5% of the largest one are not worth considering, and no more than the to 4 imortant coordinates needed to be considered After raning the coordinates, the to two coordinates are used to calculate the first angle of rotation θ 0, as exlained in the revious section he only change to the revious section is the calculation of θ I when there are three imortant coordinates: θ = max(05max( θ π /, θ, 09min( θ, π / θ, I G r r and when there are four imortant coordinates: θ = max(06max( θ π /, θ, 08min( θ, π / θ, I G r r where the factors were otimized exerimentally hen all the imortant coordinates are added to the rotation as follows Each of these coordinates in order is aired one at a time, with all the receding ones, and the angle of rotation is calculated as described above For examle, the rotation matrix for a three variable roblem where all are imortant variables is effectively the roduct of these three rotation matrices: sinθ 0 sinθ0 0 R( θ = 0 sinθ 0 0 sinθ sinθ sinθ 0 0 0

6 he imlementation always handles rotation between two variables at a time; and in fact, it never oerates with any matrix larger than 6 Results Of Some Cuter Unconstrained Problems [] he Curved rajectories Algorithm (CA acage, including the otion of rotation, has been imlemented to run in the Cuter environment, and is available from the author for academic use he CA acage solves all the error-free unconstrained roblems, including ones with interval constraints on the variables able shows details on some roblems running with and without rotation Stoing criteria has been selected to achieve solutions as good as ublished results, tyically with Gradient norms less than able Summary of CA acage erformance on a samle of small unconstrained roblems from Cuter Column labels: Rot is number of iterations that used rotations; It is number of iterations, NF, NG, and NH are the number of Function, Gradient and Hessian evaluations; f and Gradient columns are the ending Function and Gradient norm values Problem size ROSENBR WOOD 4 BRYBND 0 CHNROSNB 0 DIXMAANJ 5 ENGVAL GROWHLS HEAR6LS 6 HEAR8LS 8 MARAOSB MEYER PFILS PFILS PFILS PFI4LS YFIU Rot It NF NG NH f Gradient norm E-4 7E E- 779E E-0 48E E-9 4E E-06 0E E-06 74E E-08 06E E- 0E E E E- 54E E- 40E E E E-04 4E E-0 48E E-4 65E E-6 669E E E E E E-04 4E E-04 8E E-06 08E E-05 49E E-06 0E E E E-07 8E E-06 47E E- 75E E- 4E-04 able summarizes the results on 7 roblems with less than 00 variables, giving a view of the liely imact of rotations for a large number of difficult roblems For sure, many of the roblems do not use rotations Running times are about the same Rotations tyically save Hessian evaluations at the exense of additional Gradient and Function evaluations his would be attractive when evaluating the Hessian is very costly able otals for 7 unconstrained Cuter roblems with less than 00 variables Pacage Iterations NF NG NH CA CA with rotation As the samles in the ables show, rotation maes significant imrovements for some roblems, no change to other roblems, and causes a few roblems to tae a longer ath to the solution; this is not an unexected result with many difficult roblems In many cases rotations imrove the solution found he singular and oorly scaled roblem SCOSINE in able is worth highlighting: rotations ended u needing one more Hessian, but the solution is more accurate able Summary of CA acage erformance on a samle of large unconstrained roblems from Cuter Column labels: Rot is number of iterations that used rotations; It is number of iterations, NF, NG, and NH are the number of Function, Gradient and Hessian evaluations; f and Gradient columns are the ending Function and Gradient norm values Problem size CHAINWOO,000 SBRYBND,000 SCOSINE 0,000 7 Conclusion Rot It NF NG NH f Gradient norm E E E-9 58E E- 05E E+0 46E E+0 799E-07 his aer resents the derivation and imlementation of adding a rotation otion to a minimization algorithm It has been imlemented in the already very efficient CA acage and roduced encouraging results, though in a few cases it caused more wor Using rotations changes the ath taen to arrive at the solution and this can sometimes end u in a region that is not smooth causing more wor to obtain the solution Rotations reduced the wor for the vast majority of the roblems and saved Hessian evaluations, and this would be of benefit when the Hessian is costly to evaluate

7 REFERENCES [] A R Conn, N I M Gould and P L oint, rust-region Methods, SIAM, 600 University City Science Center, Philadelhia, PA, 000 [] N I M Gould, D Orban and P L oint, CUEr and SifDec: A constrained and unconstrained testing environment, revisited, ACM ransactions on Mathematical Software, 9, 4 (00, 7 94 [] D M Himmelblau, Alied Nonlinear Programming, McGraw-Hill Boo Comany, NY, 97 [4] A J Jiménez, Sarse Hessian factorization in curved trajectories for unconstrained minimization, Otimization Methods and Software, London DOI:0080/ , 0 [5] A J Jiménez, A Variable-Order Nonlinear Programming Algorithm for Use in Comuter-Aided Circuit Design and Analysis, PhD Dissertation, University of Florida, Gainesville, Florida, 976 Available at the University of Florida, George A Smathers Libraries, htt://archiveorg/details/variableordernon00jim [6] AJ Jiménez and SW Director, A Variable-Order Algorithm for Unconstrained Minimization, in Proceedings 976 IEEE International Symosium on Circuit and Systems, Aril 976 [7] D Kincaid and W Cheney, Numerical Analysis Mathematics of Scientific Comuting, hird Edition, Brooc/Cole, Pacific Grove, Ca, 00 [8] D G Luenberger, Introduction to Linear and Nonlinear Programming, Addison-Wesley Publishing Comany, Menlo Par, CA, 965 [9] J Nocedal and S J Wright, Numerical Otimization, nd edition, Sriger, NY, NY, 006 [0] H H Rosenbroc, An automatic method for finding the greatest or the least value of a function, Comuter Journal, (960, [] D C Sorensen, Newton s method with a model trust region modification, SIAM Journal on Numerical Analysis, 9 (98,