10.3 Steepest Descent Techniques

Size: px

Start display at page:

Download "10.3 Steepest Descent Techniques"

Cameron Waters
5 years ago
Views:

1 The advantage of the Newton and quasi-newton methods for solving systems of nonlinear equations is their speed of convergence once a sufficiently accurate approximation is known. A weakness of these methods is that an accurate initial approximation to the solution is needed to ensure convergence. The Steepest Descent method considered in this section converges only linearly to the solution, but it will usually converge even for poor initial approximations. As a consequence, this method is used to find sufficiently accurate starting approximations for the Newton-based techniques in the same way the Bisection method is used for a single equation. Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

2 Consider the system of nonlinear equations of the form f 1 (x 1, x 2,, x n ) = 0, f 2 (x 1, x 2,, x n ) = 0,. f n (x 1, x 2,, x n ) = 0. Define the function g(x 1, x 2,, x n ) = n i=1 [ f i (x 1, x 2,, x n )] 2. The function g(x 1, x 2,, x n ) is nonnegative. If x = [x 1, x 2,, x n ] t is a solution to the system F(x) = 0, then the function g reaches its local minimum, and the minimum value is 0. Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

3 The method of Steepest Descent for finding a local minimum for g : R n R is as follows: 1 Evaluate g at an initial approximation x (0) = [ ] t. x (0) 1, x(0) 2,, x(0) n 2 Determine a direction from x (0) that results in a decrease in the value of g. 3 Move an appropriate amount in this direction and call the new value x (1). 4 Repeat step 1 through step 3 with x (0) replaced by x (1). Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

4 Illustration of the Method of Steepest Descent The figure is downloaded from Wikipedia.org Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

5 Review of Gradient of a Function For g : R n R, the gradient of g at x = [x 1, x 2,, x n ] t is denoted g(x) and defined by g(x) = [ g (x), g ] g t (x),, (x). x 1 x 2 x n For example, g(x 1, x 2 ) = x 2 1 2x 1x 2 + sin(2x 1 + x 2 ), g(x) = [2x 1 2x cos(2x 1 + x 2 ), 2x 1 + cos(2x 1 + x 2 )] t. Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

6 Review of Gradient of a Function The directional derivative of g at at x = [x 1, x 2,, x n ] t in the direction of a unit vector v measures the rate of change of g in the direction v. It is defined by g(x + hv) g(x) D v g(x) = lim = v g(x) h 0 h The direction that produces the maximum value for the directional derivative occurs when v is chosen to be parallel to g(x). This direction yields the greatest increase of the value of g. The direction of greatest decrease in the value of g at x is the direction given by g(x). Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

7 derivative occurs when v is chosen to be parallel to g(x), provided that g(x) = 0. Asa consequence, the direction of greatest decrease in the value of g at x is the direction given by g(x). Figure 10.3 is an illustration when g is a function of two variables. Illustration of greatest decrease direction for a function z = g(x 1, x 2 ). z z g(x 1, x 2 ) (x 1, x 2, g(x 1, x 2 )) Steepest descent direction x x (x x 2 1, x 2 ) t 1 g(x) Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

8 The object is to reduce g(x) to its minimum value of zero. We choose x (1) by moving away from x (0) in the direction g(x (0) ) that gives the greatest decrease at x (0). That is x (1) = x (0) α g(x (0) ), for some constant α > 0. The remaining task is to choose an appropriate value of α so that g(x (1) ) is significantly less than g(x (0) ). Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

9 To determine the value α, we consider the single-variable function h(α) = g(x (0) α g(x (0) )). The value of α that minimizes the function h is the value needed. Taking derivative of h(α) is computationally too costly. Instead, we consider an approximation of h(α). To do this, we choose three numbers α 1 < α 2 < α 3 that, we hope, are close to where the minimum value of h(α) occurs. We then construct a quadratic polynomial P (x) that agrees with the value of h at α 1, α 2, and α 3. Let α be the minimizer of P (x) in [α 1, α 3 ]. Then, α is used for approximating the minimum value of g is x (1) = x (0) α g(x (0) ). Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

10 First, we choose α 1 = 0, that is g 1 = g(x (0) α 1 x (0) ) = g(x (0) ) Next, we choose a number α 3 with g 3 = g(x (0) α 3 x (0) ) < g(x (0) ) = g 1. Since α 1 does not minimize h, such a number α 3 does exist. 1 First choose α 3 = 1, and calculate g 3 = g(x (0) α 3 x (0) ). 2 If g 3 < g 1, then this α 3 works. 3 If g 3 > g 1, then let α 3 = α 3 /2 and go back to Step 1. Finally, we choose α 2 = α 3 2. Compute g 2 = g(x (0) α 2 x (0) ) Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

11 Now we obtain three data points (α 1, g 1 ), (α 2, g 2 ), (α 3, g 3 ). Use the quadratic interpolation (coefficients h 1 and h 2 to be determined) P (x) = g 1 + h 1 (x α 1 ) + h 2 (x α 1 )(x α 2 ) h 1 = g 2 g 1, h 2 = g 3 g 1 h 1 (α 3 α 1 ) α 2 α 1 (α 3 α 1 )(α 3 α 2 ) Note that α 1 = 0, and α 2 = α 3 /2, then We can find that P (x) = g 1 + h 1 x + h 2 x(x α 2 ) h 1 = g 2 g 1 α 2 = 2(g 2 g 1 ) α 3 h 2 = g 3 g 1 h 1 α 3 α 3 (α 3 α 2 ) = 2(g 1 2g 2 + g 3 ) α3 2. Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

12 Note that P (x) = g 1 + h 1 x + h 2 x(x α 2 ) The critical point α 0 of P (x) can be obtained by P (x) = h 1 +2h 2 x h 2 α 2 = 0, = α 0 = h 2α 2 h 1 2h 2 = 1 2 (α 2 h 1 h 2 ). The minimum value of P on [α 1, α 3 ] occurs either at the only critical point α 0 of P or at the right endpoint α 3. Choose α from {α 0, α 3 } so that g(x α g(x (0) )) = min{g 0, g 3 }. Update the solution by x (1) = x (0) α g(x (0) ) Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

13 Example 5. Use the Steepest Descent method with x (0) = [0, 0, 0] t to find a reasonable starting approximating to the solution of 3x 1 cos(x 2 x 3 ) 1 2 = 0, x (x ) 2 + sin(x 3 ) = 0, e x 1x x π 3 = 0. 3 Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

14 Solution (1/5). g(x 1, x 2, x 3 ) = [f 1 (x 1, x 2, x 3 )] 2 + [f 2 (x 1, x 2, x 3 )] 2 + [f 3 (x 1, x 2, x 3 )] 2 2f 1 (x) f 1 (x) + 2f 2 (x) f 2 (x) + 2f 3 (x) f 3 (x) x 1 x 1 x 1 g(x) = 2f 1 (x) f 1 (x) + 2f 2 (x) f 2 (x) + 2f 3 (x) f 3 (x) x 2 x 2 x 2 2f 1 (x) f 1 (x) + 2f 2 (x) f 2 (x) + 2f 3 (x) f 3 (x) x 3 x 3 x 3 f 1 f 2 f 3 (x) (x) (x) x 1 x 1 x 1 = 2 f 1 f 2 f 3 f 1 (x) (x) (x) (x) x 2 x 2 x f 2 (x) 2 f 1 f 2 f 3 f 3 (x) (x) (x) (x) x 3 x 3 x 3 = 2 [ J(x) ] t F(x) Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

15 Solution (2/5). For x (0) = [0, 0, 0] t, we have g(x (0) ) = g(x (0) ) = 2 [ J(x (0) ) ] t F(x (0) ) = Normalize the gradient vector z = 1 g(x (0) g(x (0) ) = 1 ) g(x(0) ) = Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

16 Solution (3/5). Set α 1 = 0, we have g 1 = g(x (0) α 1 z) = Set α 3 = 1, we have g 3 = g(x (0) α 3 z) = Since g 3 < g 1, we let α 2 = α 3 2 = 0.5, g 2 = g(x (0) α 2 z) = The data points (α 1, g 1 ), (α 2, g 2 ), (α 3, g 3 ) are (0, ), (0.5, ), (1, ) We construct P (x) = g 1 + h 1 (x α 1 ) + h 2 (x α 1 )(x α 2 ) that interpolates the data points. P (α 1 ) = g 1 is automatically satisfied. P (α 2 ) = g 2 yields h 1 = 2(g 2 g 1 ) α 3 = P (α 3 ) = g 3 yields h 2 = 2(g 1 2g 2 + g 3 ) α 2 3 = Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

17 Solution (4/5). The interpolating polynomial is P (x) = x x(x 0.5) The critical point is α 0 = 1 2 (α 2 h 1 h 2 ) = Evaluating g(x) at α 0 yields g 0 = g(x (0) ˆαz) = Since g 0 < g 3, we choose α = α 0 = x (1) = x (0) αz = Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

18 be appropriate at this stage, since 70 iterations of the Steepest Descent method are re to find x (k) x < Solution (5/5). We use Matlab programming to finish the rest of computation. e 10.5 k x (k) 1 x (k) 2 x (k) 3 g(x (k) 1, x(k) 2, x(k) 3 ) ll Rights Reserved. AMay true not be copied, solution scanned, or duplicated, to the in whole nonlinear or part. Due to electronic system rights, some isthird x party = content (0.5, may be 0, suppressed ) t from the ebook and/or echapter, ressed content does not materially affect the overall learning experience. Cengage Learning reserves the right to remove additional content at any time if subsequent rights restrictions re so the second iterative solution x (2) would likely be adequate as an initial approximation for Newtons method or Broydens method. One of these quicker converging techniques would be appropriate at this stage, since 70 iterations of the Steepest Descent method are required to find x x (k) Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

19 Chapter 3/27/ Numerical 6:11 PM Solution /Users/xuzhang/Dropbox/T.../ex10_3_1_new.m of Nonlinear Systems of Eqns 1 of 2 % ex10_3_1: Steepest Descent Method clc clear %% Initialization MaxIt = 100; % max number of iterations tol = 10^(-2); % tolerance x0 = [0; 0; 0]; % initial approximation %% Steepest Descent Method for i = 1:MaxIt disp(['iteration = ', int2str(i)]); F = Fun(x0); J = jacobian(x0); grad = 2*J'*F; z = (1/norm(grad))*grad; % normalize the vector a1 = 0; g1 = FunG(x0-a1*z); a3 = 1; g3 = FunG(x0-a3*z); while g3 > g1 a3 = a3/2; g3 = FunG(x0-a3*z); end a2 = a3/2; g2 = FunG(x0-a2*z); h1 = 2*(g2-g1)/a3; h2 = 2*(g1-2*g2+g3)/(a3*a3); a0 = (1/2)*(a2-h1/h2); % a0 is the critical point g0 = FunG(x0-a0*z); if g0 < g3 % select the smaller one from g0 and g3 a = a0; elseif g0 >= g3 a = a3; end g = FunG(x0-a*z) x = x0 - a*z if abs(g) < tol % Stopping Criteria break; end x0 = x; end %% Function F function F = Fun(x) y1 = 3*x(1)-cos(x(2).*x(3))-1/2; 3/27/19 6:11 PM /Users/xuzhang/Dropbox/ %% Function F function F = Fun(x) y1 = 3*x(1)-cos(x(2).*x(3))-1/2; y2 = x(1).^2-81*(x(2)+0.1).^2+sin(x(3)) ; y3 = exp(-x(1).*x(2))+20*x(3)+(10*pi-3)/3; F = [y1;y2;y3]; end %% Jacobian Function function J = jacobian(x) n = length(x); J = zeros(n); % initialization of J J(1,1) = 3; J(1,2) = x(3).*sin(x(2).*x(3)); J(1,3) = x(2).*sin(x(2).*x(3)); J(2,1) = 2*x(1); J(2,2) = -162*(x(2)+0.1); J(2,3) = cos(x(3)); J(3,1) = -x(2).*exp(-x(1).*x(2)); J(3,2) = -x(1).*exp(-x(1).*x(2)); J(3,3) = 20; end %% Function g function u = FunG(x) y1 = 3*x(1)-cos(x(2).*x(3))-1/2; y2 = x(1).^2-81*(x(2)+0.1).^2+sin(x(3)) ; y3 = exp(-x(1).*x(2))+20*x(3)+(10*pi-3)/3; u = y1.^2 + y2.^2 + y3.^2; end Xu Zhang (Mississippi State University) MA4323/6323 Numerical Analysis II Spring / 60

Numerical solutions of nonlinear systems of equations

Numerical solutions of nonlinear systems of equations Tsung-Ming Huang Department of Mathematics National Taiwan Normal University, Taiwan E-mail: min@math.ntnu.edu.tw August 28, 2011 Outline 1 Fixed points