Conjugate Gradient Methods

Size: px

Start display at page:

Download "Conjugate Gradient Methods"

Rosalyn Gibbs
5 years ago
Views:

1 Conjugate Gradient Methods Now we consider optimization in n-dimensional design space. The objective is to minimize function f(x), where the components of x are the n independent design variables. The basic strategy is 1. choose a point x in the design space. 2. loop with i = 1, 2, 3,... choose a vector v i minimize f(x) along the line through x i 1 in the direction of v i. let the minimum point be x i. If x i x i 1 < ε exit loop 3. end loop The minimization along a line can be accomplished with any one-dimensional optimization algorithm (such as the golden section search). The very important question is how to choose the vectors v i. Conjugate Directions Consider the quadratic function f(x) = c i Differentiation with respect to x i yields which can be written in vector notation as b i x i + 1 A ij x i x j = c b T x xt Ax. (1) i f x i = b i + j j A ij x j, (2) f = b + Ax, (3) where f is the gradient of f. Now consider the change in the gradient as we move from point x in the direction of a vector u. The motion takes place along the line x = x + su, (4) 1

2 2 where s is the distance moved. Substitution into Eq. (3) yields the expression for the gradient at x f x +su = b + A(x + su) = f x + sau. (5) Note that the change in the gradient is sau. If this change is perpendicular to a vector v, then v T Au =, (6) and the directions of u and v are said to be mutually conjugate (non-interfering). The implication is that once we have minimized f(x) in the direction of v, we can move along u without ruining the previous minimization. For a quadratic function of n independent variables it is possible to construct n mutually conjugate directions. Therefore, it would take exactly n line minimizations along these directions to reach the minimum point. If f(x) is not a quadratic function, Eq. (1) can be treated as a local approximation of the merit function, obtained by truncating the Taylor series expansion of f(x) about x f(x) f(x ) + f(x )(x x ) (x x ) T H(x x ). (7) Now the conjugate directions based on the quadratic form are only approximations, valid in the close vicinity of x. Consequently, it would take several cycles of n line minimizations to reach the optimal point. The various conjugate gradient methods use different techniques for constructing conjugate directions. The so-called zero-order methods work with f(x) only, whereas the first-order methods utilize both f(x) and f. The first-order methods are computationally more efficient, but the input of f can be very tedious, or it may be not available at all. Powell s Method Powell s method is a zero-order method, requiring the evaluation of f(x) only. involves n design variables, the basic algorithm is 1. Choose a point x in the design space. If the problem 2. Choose the starting vectors v i, i = 1, 2,..., n (the usual choice is v i = e i, where e i is the unit vector in the x i -coordinate direction). 3. Start cycle do i = 1, 2,..., n Minimize f(x) along the line through x i 1 in the direction of v i. Let the minimum point be x i. end do v n+1 x x n Minimize f(x) along the line through x in the direction of v n+1. Let the minimum point be x n+1. if x n+1 x < ε exit loop do i = 1, 2,..., n v i v i+1 (v 1 is discarded, the other vectors are reused). end do

3 3 4. End cycle Powell demonstrated that the vectors v n+1 produced in successive cycles are mutually conjugate, so that the minimum point of a quadratic surface is reached in exactly n cycles. In practice, the merit function is seldom quadratic, but as long as it can be approximated locally by Eq. (1), Powell s method will work. Of course, it takes more than n cycles to arrive at the minimum of a non-quadratic function. Example 1 Find the minimum of the function f = 1(y x 2 ) 2 + (1 x) 2 using Powell s method starting at the point (-1,1). This function has an interesting topology. There is a hump between the starting and minimum points. The minimum value of f occurs at the point (1,1). #!/usr/bin/python from powell import * from numpy import array def F(x): return 1.*(x[1] - x[]**2)**2 + (1 - x[])**2 xstart = array([-1., 1.]) xmin,niter = powell(f,xstart) print "x =",xmin print "F(x) =", F(xMin) print "Number of cycles =", niter raw_input ("") The minimum point is obtained after 12 cycles. x = [ 1. 1.] F(x) = e-29 Number of cycles = 12 Example 2 Use Powell s method to determine the smallest distance from the point (5,8) to the curve xy = 5. This is a constrained optimization problem: minimize f(x, y) = (x 5) 2 + (y 8) 2 (the square of the distances) subject to the equality constraint xy 5 =. The following program uses Powell s method with penalty function. #!/usr/bin/python from powell import * from numpy import array from math import sqrt def F(x):

4 4 lam = 1. # Penalty multiplier c = x[]*x[1] - 5. # Constraint equation return distsq(x) + lam*c**2 # Penalized merit function def distsq(x): return (x[] - 5)**2 + (x[1] - 8)**2 xstart = array([1.,5.]) x,numiter = powell(f,xstart,.1) print "Intersection point =", x print "Minimum distance =", sqrt(distsq(x)) print "xy =", x[]*x[1] print "Number of cycles =", numiter raw_input ("") The value of the penalty function multiplier µ can have profound effect on the result. With µ = 1 we obtain the following result: Intersection point = [ ] Minimum distance = xy = Number of cycles = 5 The small value of µ favored speed of convergence over accuracy. Because the violation of the constraint xy = 5 is clearly unacceptable, we run the script again with µ = 1 and change the starting point to ( ), the end point of the first run. The results shown next are now acceptable: Intersection point = [ ] Minimum distance = xy = Number of cycles = 5 A large µ often causes the algorithm to hang up, so that it is generally wise to start with a small µ. Example 3 The displacement formulation of the truss shown in Figure 1 results in the following system of equations for the joint displacements u E 2 2L 2 2A 2 + A 3 A 3 A 3 A 3 A 3 A 3 A 3 A 3 2 2A 1 + A 3 u 1 u 2 u 3 = P, (8) where E is the modulus of elasticity, and A i is the cross-sectional area of ith member. Use Powell s method to minimize the structural volume of the truss while keeping the displacement u 2 below a given value δ.

5 5 u 3 1 L L 3 u 2 2 P u 1 Figure 1: 1 Introducing the dimensionless variables v i = u i δ, the system of equations becomes 2 2x x 3 x 3 x 3 2 x 3 x 3 x 3 2 x 3 x 3 2 2x 1 + x 3 The structural volume to be minimized is x i = Eδ P L A i (9) v 1 v 2 v 3 = 1. (1) V = L(A 1 + A 2 + 2A 3 ) = P L2 Eδ (x 1 + x 2 + 2x 3 ) (11) In addition to the displacement constraint u 2 δ, we should also prevent the cross-sectional areas from becoming negative by applying the constraints A i. Thus, the optimization problem becomes minimize f = x 1 + x 2 + 2x 3 with the inequality constraints v 2 1, x i, i = 1, 2, 3. (12) Note that in order to obtain v 2 we must solve system (1). #!/usr/bin/python from powell import * from numpy import array from math import sqrt from gausselimin import * def F(x): global v, weight mu = 1. c = 2.*sqrt(2.)

6 6 A = array([[c*x[1] + x[2], -x[2], x[2]], \ [-x[2], x[2], -x[2]], \ [ x[2], -x[2], c*x[] + x[2]]])/c b = array([., -1.,.]) v = gausselimin(a,b) weight = x[] + x[1] + sqrt(2.)*x[2] penalty = max(.,abs(v[1]) - 1.)**2 \ + max(.,-x[])**2 \ + max(.,-x[1])**2 \ + max(.,-x[2])**2 return weight + penalty*mu xstart = array([1., 1., 1.]) x,numiter = powell(f,xstart) print "x = ", x print "v = ", v print "Relative weight F = ", weight print "Number of cycles = ", numiter raw_input ("") The first run of the program starts with x = [ ] T and uses µ = 1 for the penalty multiplier. The results are x = [ ] v = [ ] Relative weight F = Number of cycles = 1 Because the magnitude of v 2 is excessive, the penalty multiplier is increased to 1, and the program runs again using the output x from the last run as the input. x = [ ] v = [ ] Relative weight F = Number of cycles = 11 Now v 2 is much closer to the constraint value.

MVE165/MMG631 Linear and integer optimization with applications Lecture 13 Overview of nonlinear programming. Ann-Brith Strömberg

MVE165/MMG631 Linear and integer optimization with applications Lecture 13 Overview of nonlinear programming. Ann-Brith Strömberg MVE165/MMG631 Overview of nonlinear programming Ann-Brith Strömberg 2015 05 21 Areas of applications, examples (Ch. 9.1) Structural optimization Design of aircraft, ships, bridges, etc Decide on the material