New hybrid conjugate gradient algorithms for unconstrained optimization

Transcription

1 ew hybrid conjugate gradient algorithms for unconstrained optimization eculai Andrei Research Institute for Informatics, Center for Advanced Modeling and Optimization, 8-0, Averescu Avenue, Bucharest, Romania, Abstract ew hybrid conjugate gradient algorithms are proposed and analyzed In these hybrid algorithms the famous parameter is computed as a convex combination of the Pola-Ribière-Polya and Dai-Yuan conjugate gradient algorithms In one hybrid algorithm the parameter in convex combination is computed in such a way that the conjugacy condition is satisfied, independent of the line search In the other, the parameter in convex combination is computed in such a way that the conjugate gradient direction is the ewton direction he algorithm uses the standard Wolfe line search conditions umerical comparisons with conjugate gradient algorithms using a set of 750 unconstrained optimization problems, some of them from the CUE library, show that the hybrid computational scheme based on conjugacy condition outperform the nown hybrid conjugate gradient algorithms MSC: 49M07, 49M0, 90C06, 65K Keywords: Unconstrained optimization, hybrid conjugate gradient method, conjugacy condition, numerical comparisons Introduction Let us consider the nonlinear unconstrained optimization problem n min f( x): x R, () { } n where f : R R is a continuously differentiable function, bounded from below For solving this problem, starting from an initial guess x0 generates a sequence { x } as R n, a nonlinear conjugate gradient method, x = x + + αd, () where α > 0 is obtained by line search, and the directions d are generated as d+ = g+ + s, d0 = g0 (3) In (3) is nown as the conjugate gradient parameter, s = x+ x and g = f( x) Consider the Euclidean norm and define y = g+ g he line search in the conjugate gradient algorithms often is based on the standard Wolfe conditions: f ( x + α d ) f ( x ) ρα g d (4), g+ d σ gd, (5) where d is a descent direction and 0< ρ σ < Plenty of conjugate gradient methods are nown, and an excellent survey of these methods, with a special attention on their global convergence, is given by Hager and Zhang [9] Different conjugate gradient algorithms correspond to different choices for the scalar parameter Some of these methods as

2 Fletcher and Reeves (FR) [6], Dai and Yuan (DY) [] and Conjugate Descent (CD) proposed by Fletcher [5]: FR g+ g+ DY g+ g+ CD g+ g+ =, =, =, g g gs have strong convergence properties, but they may have modest practical performance due to jamming On the other hand, the methods of Pola Ribière [3] and Polya (PRP) [4], Hestenes and Stiefel (HS) [0] or Liu and Storey (LS) []: PRP g+ y HS g+ y LS g+ y =,, =, = g g gs in general may not be convergent, but they often have better computational performances In this paper we focus on hybrid conjugate gradient methods hese methods are combinations of different conjugate gradient algorithms, mainly they being proposed to avoid the jamming phenomenon One of the first hybrid conjugate gradient algorithms has been introduced by ouati-ahmed and Storey [8], where the parameter is computed as: PRP g+ y PRP FR =, if 0, S g = FR g + =, otherwise g he PRP method has a built-in restart feature that directly addresses to jamming Indeed, when the step s is small, then the factor y in the numerator of tends to zero herefore, descent direction PRP becomes small and the search direction g + PRP d + is very close to the steepest Hence, when the iterations jam, the method of ouati-ahmed and Storey uses the PRP computational scheme Another hybrid conjugate gradient method was given by Hu and Storey [], where is: { { }} = max 0, min, HuS PRP FR in (3) As above, when the method of Hu and Storey is jamming, then the PRP method is used instead he combination between LS and CD conjugate gradient methods leads to the following hybrid method: { { }} = max 0, min, LS CD LS CD he CD method of Fletcher [5] is very close to FR method With an exact line search, CD method is identical to FR Similarly, for an exact line search, LS method is also identical to PRP herefore, the hybrid LS-CD method with an exact line search has similar performances with the hybrid method of Hu and Storey Gilbert and ocedal [7] suggested a combination between PRP and FR methods as: Since FR { { }} G max FR, min PRP, FR = is alwa nonnegative, it follows that G can be negative he method of Gilbert and ocedal has the same advantage of avoiding jamming Using the standard Wolfe line search, the DY method alwa generates descent directions and if the gradient is Lipschitz continuously the method is global convergent In an effort to improve their algorithm, Dai and Yuan [3] combined their algorithm with other conjugate gradient algorithm, proposing the following two hybrid methods: { c { }} hdy max DY, min HS, DY =,

3 { { }} = max 0, min,, hdyz HS DY where c = ( σ ) /( + σ ) For the standard Wolfe conditions (4) and (5), under the Lipschitz continuity of the gradient, Dai and Yuan [3] established the global convergence of these hybrid computational schemes In this paper we propose another hybrid conjugate gradient as a convex combination of PRP and DY conjugate gradient algorithms We selected these two methods to combine in a hybrid conjugate gradient algorithm because PRP has good computational properties, on one side, and DY has strong convergence properties, on the other side Often PRP method performs better in practice than DY and we speculate this in order to have a good practical conjugate algorithm he structure of the paper is as follows In section we introduce our hybrid conjugate gradient algorithm and prove that it generates descent directions satisfying in some conditions the sufficient descent condition Section 3 presents the algorithms and in section 4 we show its convergence analis In section 5 some numerical experiments and performance profiles of Dolan-Moré [4] corresponding to this new hybrid conjugate gradient algorithm and some other conjugate gradient algorithms are presented he performance profiles corresponding to a set of 750 unconstrained optimization problems in the CUE test problem library [6], as well as some other unconstrained optimization problems presented in [] show that this hybrid conjugate gradient algorithm outperform the nown hybrid conjugate gradient algorithms ew hybrid conjugate gradient algorithms he iterates x, x, x, the stepsize directions where d 0 of our algorithm are computed by means of the recurrence () where α > 0 is determined according to the Wolfe conditions (4) and (5), and the are generated by the rule: d = g + s, d0 = g0, (6) + + = ( θ ) + θ PRP DY g y g g = ( θ ) + g g θ (7) and θ is a scalar parameter satisfying 0 θ, which follows to be determined Observe PRP DY that if θ = 0, then =, and if θ =, then = On the other hand, if PRP DY 0< θ <, then is a convex combination of and Referring to the PRP method, Pola and Ribière [3] proved that when function f is strongly convex and the line search is exact, then the PRP method is global convergent In an effort to understand the behavior of the PRP method, Powell [5] showed that if the step length s = x x approaches to zero, the line search is exact and the gradient f ( x) is + Lipschitz continuous, then the PRP method is globally convergent Additionally, assuming that the search direction is a descent direction, Yuan [9] established the global convergence of the PRP method for strongly convex functions and a Wolfe line search For general nonlinear functions the convergence of the PRP method is uncertain Powell [6] gave a 3 dimensional example, in which the function to be minimized is not strongly convex, showing that even with an exact line search, the PRP method may not converge to a stationary point Later on Dai [7] presented another example this time with a strongly convex function for which the PRP method fails to generate a descent direction herefore, theoretically the convergence of the PRP method is limited to strongly convex functions For general nonlinear functions the convergence of the PRP method is established under restrictive conditions (Lipschitz continuity, exact line search and the stepsize tends to zero) However, the numerical experiments presented, for example, by Gilbert and ocedal [7] proved that the PRP method is one of the best conjugate gradient methods, and this is the main motivation to consider it in (7) 3

4 On the other hand, the DY method alwa generates descent directions, and in [8] Dai established a remarable property for the DY conjugate gradient algorithm, relating the descent directions to the sufficient descent condition It is shown that if there exist constants γ and γ such that γ g γ for all, then for any p ( 0, ), there exists a constant c > 0 such that the sufficient descent condition gi di c gi holds for at least p indices i [ 0, ], where j denotes the largest integer j herefore, this property is the main reason we consider DY method in (7) It easy to see that: y g g g d g ( θ) + s θ = s (8) g g y s d d0 Supposing that is a descent direction ( (8) we can prove the following result = g ), then for the algorithm given by () and heorem Assume that α in algorithm () and (8) is determined by Wolfe line search (4) and (5) If 0< θ <, and then direction d + θ Proof Since 0< <, from (8) we get But, + + g + g 0 g s ( g y )( g s ), (9) given by (8) is a descent direction y g g g g d g g s g s = + + ( θ) + + θ + g g y g g g g+ + g + s + g + s g g g s y g = + + gs y g+ = g + + g + s gg + + g+ g + s gg gs > 0 by (5) and since 0, it follows that g s g 0 + herefore, from (9), it follows that g d + + 0, ie the direction d + is a descent one heorem Suppose that ( g+ y)( g+ s) 0 If 0< < then the direction d + given by (8) satisfies the sufficient descent condition g s + g+ d+ θ g + (0) Proof From (8) we have: g+ y g+ g+ g+ d+ = g+ + ( θ) g + s + θ g + s g g y s θ 4

5 g s ( g y )( g s ) = g + + θ + g ( θ) gg g s + θ g 0 + Observe that, since > 0 by (5) and since g + s = + gs <, then / g+ s > herefore, if 0< θ <, it follows that θ < / g+ s herefore g+ s θ > 0 proving the theorem o select the parameter θ we consider the following two possibilities In the first hybrid conjugate gradient algorithm the parameter θ is selected in such a manner that the conjugacy condition Hence, from yd + = 0 0 is satisfied at every iteration, independent on the line search yd + = after some algebra, using (8), we get: ( y g )( y s ) ( y g )( g g + + θ = ( y g+ )( ) g+ g In the second algorithm the parameter θ is selected in such a manner that the direction d + from (8) is the ewton direction, ie y g g g f ( x ) g g ( θ) + s θ = s () g g y s Having in view that f ( x ) s = y, from () we get: θ + ) ( y g+ s g+ ) g ( g+ y)( = g+ g ( g+ y)( ) Observe that the parameter θ given by () or (3) can be outside the interval [0,] However, in order to have a real convex combination in (7) the following rule is considered: PRP if θ 0, then set θ = 0 in (7), ie = ; if θ, then tae θ = in (7), ie DY = herefore, under this rule for properties of PRP and DY algorithms ) () (3) θ selection, the direction d + in (8) combines the 3 he ew Hybrid Algorithms (CCOMB, DOMB) n Step Initialization Select x0 R and the parameters 0< ρ σ < Compute f ( x 0) and g 0 Consider d 0 = g 0 and set the initial guess: α 0 = / g0 Step est for continuation of iterations If g 0 6, then stop Step 3 Line search Compute α > 0 satisfying the Wolfe line search condition (4) and (5) and update the variables x = x + + αd Compute f ( x + ), g + and s = x+ x, y = g g + Step 4 θ parameter computation If otherwise compute θ as follows: ( y g )( y s ) g g =, then set θ = 0, 5

6 CCOMB algorithm ( θ from Conjugacy Condition): ( y g+ )( ) ( y g+ )( g g θ = ( y g )( y s ) ( g g )( g g DOMB algorithm ( θ from ewton Direction): θ ) ) ( y g+ s g+ ) g ( g+ y)( = g+ g ( g+ y)( ) Step 5 conjugate gradient parameter computation If 0< θ <, then compute (7) If θ, then set = DY Step 6 Direction computation Compute is satisfied, then set α PRP If θ 0, then set = d = g + s + g+ g 0 g+, d = + g+ otherwise define d = α d / d, set = + and continue with step ) as in If the restart criterion of Powell (4) d + = Compute the initial guess It is well nown that if f is bounded along the direction then there exists a stepsize α satisfying the Wolfe line search conditions (4) and (5) In our algorithm when the Powell restart condition is satisfied, then we restart the algorithm with the negative gradient g + More sophisticated reasons for restarting the algorithms have been proposed in the literature [0], but we are interested in the performance of a conjugate gradient algorithm that uses this restart criterion, associated to a direction satisfying the conjugacy condition or is equal to the ewton direction Under reasonable assumptions, conditions (4), (5) and (4) are sufficient to prove the global convergence of the algorithm We consider this aspect in the next section he first trial of the step length crucially affects the practical behavior of the algorithm At every iteration the starting guess for the step α in the line search is computed as α d / d his selection was considered for the first time by Shanno and Phua in COMI [7] It is also considered in the pacages: SCG by Birgin and Martínez [5] and in SCALCG by Andrei [,3,4] 4 Convergence analis hroughout this section we assume that: S = x R n : f( x) f( x ) is bounded (i) he level set { 0 } (ii) In a neighborhood of S, the function f is continuously differentiable and its gradient is Lipschitz continuous, ie there exists a constant L > 0 such that f ( x) f( y) L x y, for all x, y Under these assumptions on f, there exists a constant Γ 0 such that f( x) Γ, for all x S In [] it is proved that for any conjugate gradient method with strong Wolfe line search the following general holds: Lemma Suppose that the assumptions (i) and (ii) hold and consider any conjugate gradient method () and (3), where d is a descent direction and α is obtained by the strong Wolfe line search If d 6

7 then =, (5) d lim inf g = 0 (6) For uniformly convex functions which satisfy the above assumptions we can prove that the norm of generated by (8) is bounded above hus, by lemma we have the following result d + heorem 3 Suppose that the assumptions (i) and (ii) hold Consider the algorithm () and (8), where is a descent direction and α is obtained by the strong Wolfe line search If for 0, that d + f ( x + α d ) f ( x ) ρα g d (7), g + d σ g d (8) s tends to zero and there exists the nonnegative constants η and η such g η s and g+ η s, (9) and the function f is a uniformly convex function, ie there exists a constant µ 0 such that for all x, y S then Proof From (0) it follows that ( f ( x) f( y)) ( x y) µ x y, (0) lim g = 0 () µ s ow, since 0 θ, from uniform convexity and (9) we have: g y g g g g y s But y Hence, L s, therefore + ΓL η + η s µ s d g s g+ y η s + η s µ s Γ+ +, η µ which implies that (5) is true herefore, by lemma we have (6), which for uniformly convex functions is equivalent to () Powell [5] showed that for general functions the PRP method is globally convergent if the steplengths s = x+ x tend to zero, ie s s is a condition of convergence For convergence of our algorithms from (9) we see that along the iterations, for, the gradient must be bounded as: η s g η s If the Powell condition is satisfied, ie s tends to zero, then s s ΓL η and therefore the norm of gradient can satisfy (9) 7

8 In the numerical experiments we observed that (9) is alwa satisfied in the last part of the iterations For general nonlinear functions the convergence analis of our algorithm exploits insights developed by Gilbert and ocedal [7], Dai and Liao [9] and that of Hager and Zhang [8] Global convergence proof of these new hybrid conjugate gradient algorithms is based on the Zoutendij condition combined with the analis showing that the sufficient descent condition holds and d is bounded Suppose that the level set L is bounded and the function f is bounded from below Lemma Assume that d is a descent direction and f satisfies the Lipschitz condition f ( x) f( x) L x x for all x on the line segment connecting x and x +, where L is a constant If the line search satisfies the second Wolfe condition (5), then σ gd α () L d Proof Subtracting Since g d from both sides of (5) and using the Lipschitz condition we have ( ) ( ) α (3) σ gd g+ g d L d d is a descent direction and σ <, () follows immediately from (3) heorem 4 Suppose that the assumptions (i) and (ii) holds, 0< θ, ( g+ y)( g+ s) 0, for every 0 there exists a positive constant ω, such that ( g s )/( y s ) ω > 0 and there exists the constants γ and Γ, such that for all, θ + γ g Γ hen for the computational scheme () and (8), where α is determined by the Wolfe line search (4) and (5), either g = 0 for some or liminf g = 0 (4) Proof By the Wolfe condition (5) we have: ( ) ( ) = g g s σ gs = ( σ) gs + By heorem, and the assumption θ( g+ s)/( ) ω, it follows that herefore, Combining (5) with (6) we get g d g s g g θ ω y s (5) g d ω g (6) σ ωα γ ( ) On the other hand y = g+ g L s Hence g y g y Γ L s With these, from (7) we get But, + + g y g g g g 8

9 g y g y ΓL s + ΓLD g g γ γ γ where D max { y z : y, z } On the other hand, herefore, +, = S is the diameter of the level set S ow, we can write g g + + Γ ( ) σ ωα γ ΓLD Γ + E (7) γ ( σ) ωα γ + + d g + s Γ+ ED (8) Since the level set L is bounded and the function f is bounded from below, using Lemma, from (4) it follows that ( gd) 0 < <, (9) = 0 d ie the Zoutendij condition holds herefore, from heorem using (9), the descent property yields: 4 4 γ g ( gd) = 0 d = 0 d = 0ω d <, which contradicts (8) Hence, γ = liminf g = 0 herefore, when 0< θ our hybrid conjugate gradient algorithms are globally convergent, meaning that either g = 0 for some or (4) holds Observe that in conditions of heorem the direction satisfies the sufficient descent condition independent on the line search d + 5 umerical experiments In this section we present the computational performance of a Fortran implementation of the CCOMB and DOMB algorithms on a set of 750 unconstrained optimization test problems he test problems are the unconstrained problems in the CUE [6] library, along with other large-scale optimization problems presented in [] We selected 75 large-scale unconstrained optimization problems in extended or generalized form Each problem is tested 0 times for a gradually increasing number of variables: n = 000, 000,, 0000 At the same time we present comparisons with other conjugate gradient algorithms, including the performance profiles of Dolan and Moré [4] All algorithms implement the Wolfe line search conditions with ρ = 0000 and σ = 09, and the same stopping criterion g 0 6, where is the maximum absolute component of a vector ALG he comparisons of algorithms are given in the following context Let f i and ALG f i be the optimal value found by ALG and ALG, for problem i =,, 750, respectively We say that, in the particular problem i, the performance of ALG was better than the performance of ALG if: 9

10 ALG ALG 3 fi fi <0 (30) and the number of iterations, or the number of function-gradient evaluations, or the CPU time of ALG was less than the number of iterations, or the number of function-gradient evaluations, or the CPU time corresponding to ALG, respectively All codes are written in double precision Fortran and compiled with f77 (default compiler settings) on an Intel Pentium 4, 8GHz worstation All these codes are authored by Andrei he performances of these algorithms have been evaluated using the profiles of Dolan and Moré [4] hat is, for each algorithm we plot the fraction of problems for which the algorithm is within a factor of the best CPU time he left side of these Figures gives the percentage of the test problems, out of 750, for which an algorithm is more performant; the right side gives the percentage of the test problems that were successfully solved by each of the algorithms Mainly, the right side represents a measure of an algorithm s robustness In the first set of numerical experiments we compare the performance of CCOMB to DOMB Figure shows the Dolan and Moré CPU performance profiles of CCOMB versus DOMB Fig Performance based on CPU time CCOMB versus DOMB Observe that CCOMB outperforms DOMB in the vast majority of problems Only 730 problems out 750 satisfy the criterion (30) Referring to the CPU time, CCOMB was better in 575 problems, in contrast with DOMB which solved only 7 problems in a better CPU time In the second set of numerical experiments we compare the performance of CCOMB to the PRP and DY conjugate gradient algorithms Figures and 3 show the Dolan and Moré CPU performance profiles of CCOMB versus PRP and CCOMB versus DY, respectively 0

11 Fig Performance based on CPU time CCOMB versus Pola-Ribière-Polya (PRP) Fig 3 Performance based on CPU time CCOMB versus Dai-Yuan (DY) When comparing CCOMB to PRP (Figure ), subject to the number of iterations, we see that CCOMB was better in 34 problems (ie it achieved the minimum number of iterations in 34 problems), PRP was better in 96 problems and they achieved the same number of iterations in 9 problems, etc Out of 750 problems, only for 7 problems the criteria (30) holds Similarly, in Figure 3 we see the number of problems for which CCOMB was better than DY Observe that the convex combination of PRP and DY, expressed as in (7), is far more successful than PRP or DY algorithms he third set of numerical experiments refers to the comparisons of CCOMB to hybrid conjugate gradient algorithms: hdy, hdyz, G, HuS, S and LS-CD Figures 4-9 presents the Dolan and Moré CPU performance profiles of these algorithms, as well as the

12 number of problems solved by each algorithms in minimum number of iterations, minimum number of function evaluations and minimum CPU time, respectively Fig 4 Performance based on CPU time CCOMB versus hybrid Dai-Yuan (hdy) Fig 5 Performance based on CPU time CCOMB versus hybrid Dai-Yuan (hdyz)

13 Fig 6 Performance based on CPU time CCOMB versus Gilbert-ocedal (G) Fig 7 Performance based on CPU time CCOMB versus Hu-Storey (HuS) 3

14 Fig 8 Performance based on CPU time CCOMB versus ouati-ahmed - Storey (S) Fig 9 Performance based on CPU time CCOMB versus Liu-Storey Conjugate Descent (LS-CD) From these Figures above we see that CCOMB is top performer Since these codes use the same Wolfe line search and the same stopping criterion they differ in their choice of the search direction Hence, among these conjugate gradient algorithms we considered here, CCOMB appears to generate the best search direction In the fourth set of numerical experiments we compare CCOMB to CG_DESCE conjugate gradient algorithm of Hager and Zhang [8] he computational scheme implemented in CG_DESCE is a modification of the Hestenes and Stiefel method which 4

15 satisfies the sufficient descent condition, independent of the accuracy of the line search he CG_DESCE code, authored by Hager and Zhang, contains the variant CG_DESCE (HZw) implementing the Wolfe line search and the variant CG_DESCE (HZaw) implementing an approximate Wolfe line search here are two main points associated to CG_DESCE Firstly, the scalar products are implemented using the loop unrolling of 6 depth 5 his is efficient for large-scale problems (over 0 variables) Secondly, the Wolfe line search is implemented using a very fine numerical interpretation of the first Wolfe condition (4) he Wolfe conditions implemented in CCOMB and CG_DESCE (HZw) can compute a solution with accuracy of the order of the square root of the machine epsilon Fig 0 Performance based on CPU time CCOMB versus CG_DESCE with Wolfe line search (HZw) Fig Performance based on CPU time CCOMB versus CG_DESCE with approximate Wolfe line search (HZaw) 5

16 In contrast, the approximate Wolfe line search implemented in CG_DESCE (HZaw) can compute the solution with accuracy of the order of machine epsilon Figures 0 and present the performance profile of these algorithms in comparison to CCOMB We see that CG_DESCE is more robust than CCOMB 5 Conclusion We now a large variety of conjugate gradient algorithms he nown hybrid conjugate gradient algorithms are based on projection of classical conjugate gradient algorithms In this paper we have proposed new hybrid conjugate gradient algorithms in which the famous DY parameter is computed as a convex combination of and, ie PRP DY = ( θ) + θ he parameter θ is computed in such a manner that the conjugacy condition is satisfied, or the corresponding direction in hybrid conjugate gradient algorithm is the ewton direction For uniformly convex functions if the stepsize approaches zero, the gradient is bounded in the sense that η s g η s and the line search satisfy the strong Wolfe conditions, then our hybrid conjugate gradient algorithms are globally convergent For general nonlinear functions if the parameter θ from definition is bounded, ie 0< θ <, then our hybrid conjugate gradient is globally PRP convergent he Dolan and Moré CPU performance profile of hybrid conjugate gradient algorithm based on conjugacy condition (CCOMB algorithm) is higher than the performance profile corresponding to the hybrid algorithm based to the ewton direction (DOMB algorithm) he performance profile of CCOMB algorithm was higher than those of the well established conjugate gradient algorithms (hdy, hdyz, G, HuS, S and LS-CD) for a set consisting of 750 unconstrained optimization test problems, some of them from CUE library Additionally the proposed hybrid conjugate gradient algorithm CCOMB is more robust than the PRP and DY conjugate gradient algorithms However, CCOMB algorithm is outperformed by CG_DESCE References [] Andrei, est functions for unconstrained optimization [] Andrei, Scaled conjugate gradient algorithms for unconstrained optimization Accepted: Computational Optimization and Applications, 006 [3] Andrei, Scaled memoryless BFGS preconditioned conjugate gradient algorithm for unconstrained optimization Accepted: Optimization Methods and Software, 006 [4] Andrei, A scaled BFGS preconditioned conjugate gradient algorithm for unconstrained optimization Accepted: Applied Mathematics Letters, 006 [5] E Birgin and JM Martínez, A spectral conjugate gradient method for unconstrained optimization, Applied Math and Optimization, 43, pp7-8, 00 [6] I Bongartz, AR Conn, IM Gould and PL oint, CUE: constrained and unconstrained testing environments, ACM rans Math Software,, pp3-60, 995 [7] YH Dai, Analis of conjugate gradient methods, PhD hesis, Institute of Computational Mathematics and Scientific/Engineering Computing, Chinese Academy of Science, 997 [8] YH Dai, ew properties of a nonlinear conjugate gradient method umer Math, 89 (00), pp83-98 [9] YH Dai and LZ Liao, ew conjugacy conditions and related nonlinear conjugate gradient methods Appl Math Optim, 43 (00), pp 87-0 [0] YH Dai, LZ Liao and Duan Li, On restart procedures for the conjugate gradient method umerical Algorithms 35 (004), pp s 6

17 [] YH Dai, Han, JY, Liu, GH, Sun, DF, Yin, X and Yuan, Y, Convergence properties of nonlinear conjugate gradient methods SIAM Journal on Optimization 0 (999), [] YH Dai and Y Yuan, A nonlinear conjugate gradient method with a strong global convergence property, SIAM J Optim, 0 (999), pp 77-8 [3] YH Dai and Y Yuan, An efficient hybrid conjugate gradient method for unconstrained optimization, Ann Oper Res, 03 (00), pp [4] ED Dolan and JJ Moré, Benchmaring optimization software with performance profiles, Math Programming, 9 (00), pp 0-3 [5] R Fletcher, Practical Methods of Optimization, vol : Unconstrained Optimization, John Wiley & Sons, ew Yor, 987 [6] R Fletcher and C Reeves, Function minimization by conjugate gradients, Comput J, 7 (964), pp49-54 [7] JC Gilbert and J ocedal, Global convergence properties of conjugate gradient methods for optimization, SIAM J Optim, (99), pp -4 [8] WW Hager and H Zhang, A new conjugate gradient method with guaranteed descent and an efficient line search, SIAM Journal on Optimization, 6 (005) 70-9 [9] WW Hager and H Zhang, A survey of nonlinear conjugate gradient methods Pacific journal of Optimization, (006), pp35-58 [0] MR Hestenes and EL Stiefel, Methods of conjugate gradients for solving linear stems, J Research at Bur Standards, 49 (95), pp [] YF Hu and C Storey, Global convergence result for conjugate gradient methods J Optim heory Appl, 7 (99), pp [] Y Liu, and C Storey, Efficient generalized conjugate gradient algorithms, Part : heory JOA, 69 (99), pp9-37 [3] E Pola and G Ribière, ote sur la convergence de directions conjuguée, Rev Francaise Informat Recherche Operationelle, 3e Année 6 (969), pp35-43 [4] B Polya, he conjugate gradient method in extreme problems USSR Comp Math Math Ph, 9 (969), pp94- [5] MJD Powell, Restart procedures of the conjugate gradient method Mathematical Programming, (977), pp4-54 [6] MJD Powell, onconvex minimization calculations and the conjugate gradient method in umerical Analis (Dundee, 983), Lecture otes in Mathematics, vol 066, Springer-Verlag, Berlin, 984, pp-4 [7] DF Shanno and KH Phua, Algorithm 500, Minimization of unconstrained multivariate functions, ACM rans on Math Soft,, pp87-94, 976 [8] D ouati-ahmed and C Storey, Efficient hybrid conjugate gradient techniques J Optim heory Appl, 64 (990), pp [9] YYuan, Analis on the conjugate gradient method, Optimization Methods and Software, (993), pp9-9 September 6, 007 Sent to Panos Pardalos 7