A new initial search direction for nonlinear conjugate gradient method

Transcription

1 International Journal of Mathematis Researh. ISSN Volume 6, Number 2 (2014), pp International Researh Publiation House A new initial searh diretion for nonlinear onjugate gradient method Sumit Saha and Biprajit Nath Department of Mathematis, Assam University, Silhar, India sumitsaha_math@yahoo.o.in, bipra4u@gmail.om Abstrat The general nonlinear onjugate gradient method always onsiders a ritial initial searh diretion to find a onvergent solution. In the present paper our aim is to establish the fat that the onvergent solution an also be obtained for nonlinear funtion, even if the initial searh diretion is different from the initial searh diretion taken in most of the algorithms of the nonlinear onjugate gradient method. The present work we try to establish the above mentioned fat theoretially. AMS subjet lassifiation: Keywords: Conjugate Gradient method, Line searh, desent property, suffiient desent ondition, global onvergene. 1. Introdution Conjugate Gradient (C.G) methods omprise a lass of unonstrained optimization algorithms whih are haraterised by low memory requirements and strong loal and global onvergene properties. In the seminal 1952 paper 1 of Hestenes and Stiefel, the algorithm is presented as an approah to solve symmetri, positive-definite linear systems. A non-linear unonstrained optimization problem an be stated as min { (x) : x R n} (1.1) Where f : R n R is a ontinuously differentiable funtion espeially if the dimension n is large. The onjugate gradient method to solve the general nonlinear problem defined by (1) is of the form x k+1 = x k + α k d k (1.2)

2 184 Sumit Saha and Biprajit Nath Where α k is a step size obtained by a line searh and d k is the searh diretion defiend by { g k,k = 0 d k = (1.3) g k + β k d k 1,k 1 Where β k is a parameter and g k denotes f(x k ) where the gradient of f at x k is a row vetor and g k is a olumn vetor. Different C.G methods orrespond to different hoies for the salar β k. It is known from (2) and (3) that only the step size α k and the parameter β k remain to be determined in the definition of Conjugate Gradient method. In this ase that if f is a onvex quadrati, the hoie of should be suh that the method (2) and (3) redues to the linear Conjugate Gradient method if the line searh is exat namely α k = arg min{(x k + αd k ); α>0} (1.4) For non linear funtions, different formulae for the parameter β k result in different 3 Conjugate Gradient methods and their properties an be signifiantly different. To differentiate the linear Conjugate Gradient method, sometimes we all the Conjugate Gradient method for unonstrained optimization by nonlinear Conjugate Gradient method and the parameter β k is alled Conjugate Gradient parameter. The equivalene of the linear system to the minimization problem of 1 2 xt Ax b T x. Motivated Flether and Reeves to extend the linear Conjugate Gradient method for nonlinear optimization. This work of Flether and Reeves in 1964 not only opened the door of nonlinear C.G Field but greatly stimulated the study of nonlinear optimization. In general the nonlinear Conjugate Gradient method without restarts is only linearly onvergent(crowder and Wolfe) while n-step quadrati onvergene rate an be established if the method is restarted along the negative gradient every n-step. In 1964 the method has been extended to nonlinear problems by Flether and Reeves 2, whih is usually onsidered as the first nonlinear Conjugate Gradient algorithm. Sine then a large number of variations of Conjugate Gradient algorithms have been suggested. A survey on their definition inluding 40 nonlinear Conjugate Gradient algorithms 3 for unonstrained optimization is given by Andrei. Sine the exat line searh is usually expensive and impratial, the strong Wolfe line searh is often onsider the implementation of the nonlinear Conjugate Gradient methods. It aims to find a step size satisfying the strong Wolfe onditions. f(x k + α k d k ) f(x k ρα k g T k d k (1.5) g(x k + α k d k ) T σg T k d k (1.6) where 0 <ρ<σ <1 The strong Wolfe line searh is often regarded as a suitable extension of the exat line searh sine it redues to the latter if σ = 0. In pratial omputation a typial hoie for σ that ontrols the inexatness of the line searh is σ = 0.1. On the other hand

3 Initial searh diretion for nonlinear onjugate gradient method 185 general non linear funtion, one may be satisfy with a step size satisfying the standard wolf onditions, namely (5) and g(x k + α k d k ) T σg T k d k (1.7) where 0 <ρ<σ <1 As is well known the standard Wolf line searh is normally used in the implementation of Quasi-Newton methods, another important lass of methods for unonstrained optimization. The work of Dai and Yuan indiates that the use of standard Wolfe line searh is possible in the nonlinear Conjugate Gradient field. A requirement for an optimization method to use the above line searhes is that, the searh diretion d k must have desent property namely g T k d k < 0 (1.8) For Conjugate Gradient method, by multiplying (3) with gk T, we have g T k d k = g k 2 + β k g T k d k 1 (1.9) Thus if the line searh is exat, we have g T k d k = g k 2 sine g T k d k 1 = 0. Consequently d k is desent provided g k = 0. In this paper we say that a Conjugate Gradient method is desent if (8) holds for all k and is suffiient desent if the suffiient desent ondition g T k d k g k 2 (1.10) holds for all k and some onstant >0.However we have to point out that the borderlines between these Conjugate Gradient methods are not strit. If s k = x k+1 x k and in the following y k = g k+1 k k. Different Conjugate Gradient algorithms orresponds to different hoies for the parameter β k. Therefore a ruial element in any Conjugate Gradient algorithm in the formula definition of β. Any Conjugate Gradient algorithm has very simple and the pototype of the general nonlinear onjugate gradient algorithm an illustrated as follows: Step1: Selet the initial starting point x 0 domf and ompute: f 0 and g 0 = f(x 0 ). Set d 0 = g 0 and k = 0 Step2: Test a riterion for stopping the iteration. For example, if s g k ɛ, then stop; otherwise ontinue with step 3. Step 3: Determine the Step length α k = gt k d k d T k d k Step 4: Update the variables as: x k+1 = x k + α k d k Compute f k+1 and g k+1 Compute y k = g k+1 g k and s k = k k+1 x k Step 5: Determine β k

4 186 Sumit Saha and Biprajit Nath Step 6: Compute the searh diretion as: d k+1 = g k+1 + β k s k Step 7: Restart riterion. For example if the restart riterion of Powell gk+1 T g k > 0.2 g k+1 2 is satisfied, then set d k+1 = g k+1 Step 8: Compute the initial guess α k 1 d k 1 d k, set k = k + 1 and ontinue with step 2. This is a prototype of the Conjugate Gradient algorithm, but some more 5,10,11,17 sophistiated variants are also known e,g.conmin, SCALCG, ASCALCG, ACGHES, ACGMSEC, CG_DESCENT et.these variants fous on parameter β k omputation and on the step length determination. Objetive: The objetive of this paper is to searh an initial searh diretion other than d 0 = g 0 for general unonstrained nonlinear onjugate gradient method and to establish the fat that the general unonstrained nonlinear onjugate gradient algorithm an be used with this new searh diretion. The other objetive of the present paper is to test the onvergene of onjugate gradient algorithm for this new initial searh diretion. Already the Conjugate Gradient method has been devised by Magnus Hestenes ( ) and Eduard Stiefel ( ) in their seminal paper where an algorithm for solving symmetri, positive-definite linear algebrai systems has been presented and after a relatively short period of stagnation, the paper by Reid brought the Conjugate Gradient method as a main ative area of researh in unonstrained optimization and later in 1964 the method has been extended to nonlinear problems by Flether and Reeves. Present Work: In the prepresent work, we will assume that the initial searh diretion for the nonlimear onjugate gradient algorithm is slightly deflet from the diretion g 0. In our work, instead of d 0 = g 0 we have taken the initial searh diretion as d 0 = g 0 + γg 0 (1.11) where γ is a very small and 0 <γ <1. The values of α k an be obtained by using (11) in α k = gt k d k dk T d. Therefore for k = 0 we have k α 0 = gt 0 d 0 d T 0 d 0 g0 T = g 0 (1 γ) 2 g0 T g 0 = 1 1 γ

5 Initial searh diretion for nonlinear onjugate gradient method 187 Putting the value of α 0 in (2), for k = 1 x 1 x 0 = g 0 x 1 = x γ (1 γ)g 0 Therefore, d 1 = g 1 + β 1 d 0 = g 1 + β 1 (1 γ)g 0 = g 1 g 1 2 g 0 2 (1 γ)g 0 = g 1 Lg 0 Where L = g 1 2 g 0 2 (1 γ)g 0 Therefore g T 1 d 1 = g 1 2 g 1 2 g 0 2 (1 γ)gt 1 g 0 g 1 2 α 1 = gt 1 d 1 d T 1 d 1 = g 1 2 Lg T 1 g 0 g L(g T 1 g 0 + g T 0 g 1) + (1 γ)l g 1 2 Negleting the remaining positive terms Again for k = 2 x 2 = x 1 + α 1 d 1 g 1 2 Lg1 T = x 1 g 0 g L(g1 T g 0 + g0 T g 1) + (1 γ)l g 1 2 (g 1 + Lg 0 ) g 1 2 Lg1 T x 2 x 1 = g 0 (g 1 + Lg 0 ) g L(g1 T g 0 + g0 T g 1) + (1 γ)l g 1 2

6 188 Sumit Saha and Biprajit Nath Continuing as above we have x k+1 x k = ( g k 2 Lg T k g k 1)(g k + Lg k 1 ) g k L(g T k g k 1 + g T k 1 g k) + (1 γ)l g k 2 g k 2 Lgk T = g k 1)(g k + Lg k 1 ) g k L(gk T g k 1 + gk 1 T g k) + (1 γ)l g k 2 ; gk T g k 1 = 0,gk 1 T g k = 0 1 g k (1 γ ) 2 g k 4 g k (1 γ ) 2 g k 1 x k+1 x k + (1 γ ) 2 g k 1 (1 γ ) 2 g k 1 < (1 γ ) 2 1 = ε Therefore x k onverges Again from (2) and (3), x k+1 = x k + α k d k d k = g k + β k d k 1 f(x k + α k d k f(x k ) = α k d k g T k + R n Here R n is the remainder after n terms. Therefore, f(x k + α k d k ) f(x k ) α k d k g T k Applying Taylor s series, we have f(x k + α k d k ) = f(x k ) + α kd k f(x k ) + α2 k d kdk T 1! 2! 2 f(x k ) +

7 Initial searh diretion for nonlinear onjugate gradient method 189 Again from Wolfe onditions, f(x k + α k d k ) f(x k ) ρα k d k gk T = ρα k gk T {( g k) + β k d k 1 } = ρα k g k 2 + ρα k gk T β kd k 1 f(x k + α k d k ) f(x k )< ρα k g k 2 f(x k + α k d k ) f(x k )<0 Therefore f(x k+1 )<f(x k )<f(x k 1 )< <f(x 1 )<f(x 0 ) From above we an observe that the funtion satisfies the Wolfe Conditions and it is onvergent. Disussion and Conlusion From the above investigation it is lear that the onvergent solutions an be obtained for unonstrained non-linear programming problems using nonlinear onjugate gradient methods by taking an initial searh diretion other then the diretion d 0 = g 0. In the present paper we have taken the initial searh diretion slightly defleted from d 0 = g 0 and with this defleted diretion also we have sueeded to ahieve a onvergent solution. The future prospet of the present work is open as the nature of γ needs more investigation. Referenes [1] M.R. Hestenes and E. Stiefel, Methods of Conjugate Gradients for solving linear systems, J. Researh Nat. Bur. Standards 49(1952), (1953). [2] A. Cohen. Rate of onvergene of several Conjugate Gradient method algorithms, SIAM Journal on Numerial Analysis 9(1972), pp [3] N. Andrei, 40 onjugate Gradient algorithms for unonstrained optimization. A survey on their definition. ICI Tehnial Report No.13/08, Mah 14, [4] N. Andei, Saled Conjugate Gradient algorithms for unonstrained optimization, Comput. Optim. Appl. 38 (2007), no. 3, [5] N. Andei, saled memoryless BFGS preonditioned Conjugate Gradient algorithm for unonstrained optimization, Optim Methods Software 22(2007), no. 4, [6] N. Andei, a saled BFGS preonditioned Conjugate Gradient algorithm for unonstrained optimization, Appl. Math. Lett. 20(2007), no

8 190 Sumit Saha and Biprajit Nath [7] Y. H. Dai, Convergene analysis of nonlinear onjugate gradient methods, Researh report, LSEC, ICMSEC, Aademy of Mathematis and System Siene, Chinese Aademy of Sienes, [8] Y. H. Dai and Y. Yuan nonlinear Conjugate Gradient with a strong global onvergene property, SIAM Journal on Optimization 10:1(1999), pp [9] W. W. Hager and H. Zhang, A new Conjugate Gradient method with guaranted desent and an effiient line searh, SIAM Jouranl on Optimization 16:1(2005), pp [10] W. W. Hager and H. Zhang, Algorithms 851: CG_DESCENT,AConjugate Gradient with guaranted desent, ACM Transations on Mathematial Software 32:1 (2006), pp [11] W. W. Hager and H. Zhang: A survey of nonlinear Conjugate Gradient methods, Paifi Journal of Optimization. 2:1 (2006), pp [12] G. Y. Li. C. M. Tang and Z. X. Wei, New Conjugay ondition and related new Conjugate Gradient methods for unonstrained optimization, Journal of Computational and Applied Mathematis 202(2007), pp [13] J. Noedal, Theory of algorithms for unonstrained optimization, Ata Numeria (1991), pp [14] Z. J. Shi and J. Guo, A new family of Conjugate Gradient methods, Journal of Computational and Applied Mathematis 224 (2009), pp [15] Y.H. Dai and Y. Yuan, An effiient Hybrid Conjugate Gradient method for unonstrained optimization, Ann. Oper. Res., 103(2001), pp [16] Y. H. Dai and Q. Ni, Testing different Conjugate Gradient methods for large sale unonstrained optimization, J. Comput. Math., 21 (2003), pp [17] Y.H. Dai and Y. Yuan, Some properties of a new Conjugate Gradient method, in advanes in nonlinear programming, Y. Yuan ed., Kluwer Publiations, Boston, 1998, pp