A Diagnostic Treatment of Unconstrained Optimization Problems via a Modified Armijo line search technique.

Transcription

1 IOSR Journal of Mathematics IOSR-JM e-issn: , p-issn: X. Volume 10, Issue 6 Ver. VI Nov - Dec. 2014, PP A Dianostic reatment of Unconstraine Optimization Problems via a Moifie Armijo line search technique. Bamibola O.M. 1, Aewumi A. O. 2, Aelee O. J. 3, Aarana M. C. 4 1 Department of Mathematics, Universit of Ilorin, Ilorin. Nieria 2 School of Mathematics, Statistics an Computer Science, Universit of KwaZulu-Natal Durban, South Africa. 3 Corresponin AuthorDepartment ofmathematics, Covenat Universit, Ota. Nieria. 4 Department of Mathematics, Covenat Universit, Ota. Nieria.. Abstract: he stu of optimization is ettin broaer with ever passin a. Optimization is basicall the art of ettin the best results uner a iven set of circumstances. A ver simple but important class of optimization problems is the unconstraine problems. Several techniques have been evelope to hanle unconstraine problems, one of which is the conjuate raient metho CGM, which is iterative in nature. In this wor, we applie the nonlinear CGM to solve unconstraine problems usin ineact line search techniques. We emploe the well-nown Armijo line search criterion an its moifie form. Kewors:Line search, conjuate raient metho, unconstraine problems, step lenth. I. Introuction A class of methos available to solve optimization problems is the escent methos. his metho which involves the evaluation of the raient of the objective function, are available to solve unconstraine minimization problems.his raient metho calle the Conjuate Graient Metho CGM was propose initiall as man believe, in 1952 b estenes an Stiefel [6] for solvin problems of linear alebra. Base on the concept of conjuac, several CGMs have been evelope b ifferent researchers. he conjuate raient metho CGM is use to solve the problem Optimize f = f 0 + f f where, 0 N, f 0, f 0 N, an f 0 N N with an iterative scheme: + 1 = + 2 where is the step size an is the irection of ecent. In orer to obtain an for the iterative scheme inicate b 2 above, researchers have resorte to ifferent methos. One of such methos is to fin such an which minimizes f +. i.e., = armin f + his close form of obtainin is nown as eact line search. Another metho nown as an ineact line search fins an approimate value of numericall. his is achieve b emploin various criteria which satisf the escent propert 0 3 II. Descent Methos for Unconstraine Optimization In aition to function evaluation, the escent techniques require the evaluation of first an possibl hiher orer erivatives of the objective function. Due to this aitional information rearin the objective function, these methos, also nown as raient methos, are enerall more efficient than the irect search methos. he raient of a function, as efine b Rao [10], is the collection of partial erivative of a function with respect to each of its n variables. It is enote b f i.e., DOI: / Pae

2 A Dianostic reatment of Unconstraine Optimization Problems via a Moifie Armijo line. f = f 1 f 2.. f If there is a movement alon the raient irection from an point in n-imensional space, the function value increases at the fastest rate. hus, the raient irection is calle the irection of steepest ascent. Since the raient vector 4 represents the irection of steepest ascent, the neative of 4 enotes the irection of steepest escent. hus, an metho which maes use of the raient vector can be epecte to iel the minimum point faster than the one which oes not mae use of the raient metho. III. Conjuate Graient Metho he converence rate of the steepest escent metho escribe above can be reatl improve b moifin it into a conjuate raient metho CGM. he CGM is a particular case of the conjuate irection metho CDM. One of the remarable properties of CDM is its abilit to enerate, in a ver economical fashion, a set of vectors i, i = 1, 2,,n havin a propert nown as conjuac, when applie to a quaratic functional. herefore an n-imensional quaratic function can be minimize in at most n-steps iven n mutuall conjuate irections. he proceure use in the evelopment of the CGM is analoous to the Gram-Schmit orthoonalization proceure. he proceure sets up each new search irection as a linear combination of the previous search irections an the newl etermine raient. i.e. i, i = 0 D = n - i + i 1 i 1, i > where i = f i an i 1 is a scalar nown as the parameter of the metho. 3.1Properties of Conjuate Graient Metho he conjuate raient metho is a conjuate irection metho with a ver special propert: in eneratin its set of conjuate vectors, it can compute a new vector 1 b usin onl the previous vector 1 1 of the conjuate set; is automaticall conjuate to these vectors. Now, for a simple case of a quaratic objective function, the first search irection from the initial point 0 is in the irection of steepest escent; that is 0 = 0 hus Where 0 is the step lenth iven b In a close form, this can be show to ive i = = ar min f α 0 α = 0 0 he eneral epression for eqn. 6 i.e 6 DOI: / Pae

3 A Dianostic reatment of Unconstraine Optimization Problems via a Moifie Armijo line. DOI: / Pae = 7 can be shown as follows: Let s be linearl inepenent, then we can write 0 = n 1 n 1 8 for certain s. ere is the solution of = b B multiplin the above b an tain the scalar prouct with, we have = 0 9 Usin the iterative scheme +1 = + we have 1 = = = = herefore, = B multiplication with an tain the scalar prouct with we obtain from orthoonalit of s 0 = 0 From 6, we can write as = 0 = 0 herefore, = = = b Since = b, we et = In the net stae, 1 is taen as a linear combination of 1 an i.e. 1 = +1 +, = 0. 1, 2, 10 he coefficients, = 0, 1, 2 nown as conjuate raient parameters are chosen in a such a wa that 1 an are conjuate with respect to. B premultipln 7 b an imposin the conition

4 A Dianostic reatment of Unconstraine Optimization Problems via a Moifie Armijo line. we obtain 1 = 0, 1 11 he above iscussion completel escribes the staes involve in the conjuate raient alorithm for quaratic functional which we now outline below. IV. Conjuate Graient Alorithm for Nonquaratic s In the previous sections, we have shown that the conjuate raient alorithm is a conjuate irection metho, an therefore minimizes a positive efinite quaratic function on n variable in n steps. he conjuate raient alorithm for a quaratic function can be etene to eneral nonlinear functions b interpretin f = b + c as a secon-orer alor series approimation of the objective function. Near the 2 solution such functions behave approimatel as quaratics, as sueste b the alor series epansion. For quaratic, the essian matri,, is constant. owever, for a eneral nonlinear function the essian is a matri that has to be reevaluate at each iteration of the alorithm. his can be computationall ver epensive. ence, an efficient implementation of the conjuate raient alorithm that eliminates the essian evaluation at each step is esirable. o erive the nonlinear conjuate raient metho, there are enerall two chanes to the conjuate raient alorithm: it becomes more complicate to compute the step lenth α, an there are several ifferent choices for, the conjuate raient parameter Because = ar min f α, 12 α 0 the close-form or the eact formula for in the CG alorithm can be replace b a numerical line search proceure. he iea is as in the CG for quaratic function whereb a value of that minimizes f + is foun b ensurin that the raient is orthoonal to the search irection. Since we are ealin with nonquaratic function f, an is etermine from a one-imensional optimization that minimizes f +, we can o this numericall usin line search techniques. he onl consieration will then be how the line search metho with can be applie to the minimization of nonquaratic functions. In the CGA for quaratic function, there are several equivalent epressions for the value of. In nonquaratic CG, these ifferent epressions are no loner equivalent. Since it is possible to eliminate from 11, the resultin alorithms now onl epen on function an raient values at each iteration.b interpretin 5 as the secon-orer alor series approimation of the nonquaratic objective function with the approimation of the essian an etermine from, we procee as follows: = b 1 an = 1 = 1 for the quaratic function, we replace the calculation for 1 1 With several researchers worin etensivel on the moification of eqn 13, several formula for has emere corresponin to ifferent CGMs. Anrei [1] has ientifie as man as fort 40 CGMs an et the number of CGMs available now is on the increase. he most recent of the CGMs is that of Bamibola et al. [2]. he followin are few of the s available in literature. 1. estenes Stiefel 1952: 1 s [5] 13 DOI: / Pae

5 A Dianostic reatment of Unconstraine Optimization Problems via a Moifie Armijo line. 2. Pola-Ribiere-Pola 1969: 3. Fletcher-Reeves 1964: PRP 1 [8, 9] FR 1 1 [4] 4. Dai-Yuan 2000: 5. Liu-Store 1992: DY 1 1 [3] LS 1 [7] 6. Conjuate Descent 1997: CD 1 1 [5] 7. Bamibola-Ali-Nwaeze 2010: 1 BNA 1 [2] DOI: / Pae where = he corresponin methos for the above s are here enote respectivel as S, PRP, FR, DY, LS, CD an BAN. V. Armijo Line Search Criterion Amon the simplest line search alorithms is Armijo s metho. his metho fins a step size α that satisfies a conition of sufficient ecrease of the objective function an implicitl a conition of sufficient isplacement from the current point. his Armijo s conition is iven as <0 + 0 In man cases it ma be necessar to impose stroner conitions on the step size. 5.1Armijo Line Search Step 1: Choose 0, ½; set = 1 an = 0.5 Step 2: While f + α >f + f, tae = Step 3: erminate with =, set 1 = + α 5.2Moifie Armijo Line Search Our moification of the above Armijo line search alorithm is what follows. he basic ieal was to specif an interval for the rate,, at which the step size,, reuces. Step 1: Choose 0, ½ 0, 1 set = 1 Step 2: While f + α >f + f, tae = for some 1, 2 = 0.1, 0.5 i.e., is chosen a new ranoml from the open set 0.1, 0.5. Step 3: erminate with =, set 1 = + α VI. Computational Details he computational eperiments carrie out in this research incorporate the ineact Alorithms in section 5 into the nonlinear CGM Alorithm iscusse above. Our aim is to perform eperiments that ai in measurin the effectiveness of two of thecgms i.e., the Fletcher-Reeves an Pola-Ribiere-Polaan the ineact alorithms in the last section. o achieve this, ten 10 nonlinear unconstraine optimization test functions b Anrei [1] were use for computational illustrations. hese functions rane from polnomial functions to nonpolnomial objective functions. Our computational instrument was a Matlab Coe esine for the purpose of this research.

6 A Dianostic reatment of Unconstraine Optimization Problems via a Moifie Armijo line. VII. Results he tables below present the outcome of the computational eperiments on the test functions. he followin notations were use; f optimal value of the objective function, the norm of the raient of f, n imension, CE computational eample, Eec proram eecution time, an Iter number of iterations. able 1: Numerical Result usin Armijo Line Search CE FR PRP Iter f Eec Iter f Eec Etene Rosenbroc Funtion e e e e e e e e Diaonal e e e e e e e e Etene immelblau e e e e e e e e Etene Beale e e e e e e e e Moifie Etene Beale e02 8.0e e02 1.2e e02 1.4e e02 1.4e Etene Bloc Diaonal BD e e e e e e e e Generalize riiaonal e e e e e e e e Generalize White olst e00 5.5e e00 6.2e e00 5.5e e00 6.2e Etene riiaonal e03 3.4e e03 7.9e e04 1.3e e04 1.2e Etene Freuenstein an Roth e05 8.4e e e e05 2.2e e e able 2:Numerical Results Usin Moifie Armijo Line Search. CE FR PRP Iter ƒ Eec Iter ƒ Eec Etene Rosenbroc Funtion e e e e e e e e Diaonal e e e e e e e e Etene immelblau e e e e e e e e Etene Beale e e e e e e e e Moifie Etene Beale e02 8.0e e02 1.2e e02 2.8e e02 1.4e Etene Bloc Diaonal BD e-9 1.6e e e e e e e Generalize riiaonal e e e e e e e e Generalize White olst e00 5.5e e00 6.2e e00 5.5e e00 6.2e Etene riiaonal e03 3.4e e03 7.9e e04 1.3e e04 1.2e Etene Freuenstein an Roth e05 8.4e e e e05 2.2e e e VIII. Discussion of Result From the above tables, it is clear that our moification of the Armijo line search criterion is still ver much in place. At a specifie tolerance level of = 1.0e-6, thouhwe obtaine ientical results but the rate of converence was moeratel improve in the moifie alorithm. IX. Conclusion We conclue b affirmin that the moifie alorithm in this wor was effective in the computational treatment of unconstraine optimization problems. DOI: / Pae

7 A Dianostic reatment of Unconstraine Optimization Problems via a Moifie Armijo line. References [1]. Anrei, N Unconstraine optimization test functions. Unpublishe manuscript; Research Institute for Informatics. Bucharest 1, Romania. [2]. Bamiibola, O.M., Ali, M. An Nwaeze, E An efficient an converent metho for unconstraine nonlinear optimization. Proceeins of International Conress of Matthematicians. eraba, Inia. [3]. Dai, Y an Yuan, Y A nonlinear conjuate raient with a stron lobal converence properties: SIAM Journal on Optimization. Vol. 10, pp [4]. Fletcher, R. an Reeves, C.M minimization b conjuate raients. Computer Journal. Vol 7, No. 2. [5]. Fletcher, R Practical metho of optimization. Secon e. John Wile, New Yor. [6]. estenes, M.R. an Stiefel, E Metho of conjuate raient for solvin linear equations. J. Res. Nat. Bur. Stan. 49. [7]. Liu, Y. an Store, C Efficient eneralize conjuate raient alorithms. Journal of Optimization heor an Application. Vol. 69, pp [8]. Pola, E. an Ribiere, G Note sur la converence e irections conjuees. Rev. FrancaiseInformatRechercheOperationelle. 3e Annee 16, pp [9]. Pola, B he conjuate raient in etreme problems. USSR Comp.Math. Math. Phs [10]. Rao, S.S Optimization theor an application. Secon eition, Wile Eastern Lt., New Delhi. DOI: / Pae