An Efficient Conjugate- Gradient Method With New Step-Size

Transcription

1 Australa Joural of Basc a Ale Sceces, 5(8: 57-68, ISSN A Effcet Cojugate- Graet Metho Wth New Ste-Sze Ba Ahma Mtras, Naa Faleh Hassa Deartmet of Mathematcs College of Comuters Sceces a Mathematcs Mosul Uversty, Ira Abstract: I ths aer, we erve a ew formula for calculatg the ste-sze for cojugate graet metho. he ew formula ee o the two formulas for Barzla Borwe(BB metho for covex quaratc fucto a the value of the varable whch s geerate by usg uform strbuto. he covergece roerty of the ew metho a the umercal result of the ew metho comaratve wth the Barzla -Borwe metho are stue by usg several test fucto. Key wors: cojugate graet metho. BB metho, Ste-sze, uform strbutato. INRODUCION he steeest escet metho, whch ca be trace bac to Cauchy (Cauchy A. 847, s the smlest graet metho for ucostrae otmzato m f(x x R ( where f(x s a cotuous eferetal fucto R. he metho has the followg form: x + = x g ( where g = g(x = f(x s the graet vector of f(x at the curret terate ot x a > s the ste-sze. Because the search recto the metho s the ooste of the graet recto, t s the steeest escet recto locally, whch gves the ame of the metho. Locally the steeest escet recto s the best recto the sese that t reuces the objectve fucto as much as ossble. he ste-sze ca be obtae by exact le search: argmf{(x g } (3 or by some le search cotos, such as Golste cotos or Wolfe cotos (Da Y.H. a H. Zhag.. It s easy to show that the steeest escet metho s always coverget. hat s, theoretcally the metho wll ot termate uless a statoary ot s fou.. he Barzla a Borwe ' s Metho: I 988 Barzla a Borwe (Barzla, j a J.M, Borwe,. 988 roose a graet escet metho that uses a fferet strategy for choosg the ste legth. hs s base o a terretato of the quas-newto methos a very smle maer. he stelegth alog the egatve graet recto s comute from a two-ot aroxmato to the secat equato from quas-newto methos. he ma ea of Barzla-Borwe, s aroach (Yua Ya-xag,. 6 s to use the formato the revous terato to ece the ste-sze the curret terato. he terato s vewe as x x D g (4 Where D =α I. I orer to force the matrx D to have certa quas-newto roerty, t s reasoable to requre ether m D y (5 or m D y (6 where x x a y g g, because a quas-newto we have that x x B g a the quas Newto matrx B satsfes the coto Corresog Author: Ba Ahme Mtras, Deartmet of Mathematcs College of Comuters Sceces a Mathemat Mosul Uversty, Ira 57

2 Aust. J. Basc & Al. Sc., 5(8:57-68, B y (7 Now, from D I a the relato (5(6 two ste-szes ca be obtae y y (8 y (9 Resectvely. For covex quaratc fucto two varables, Barzla a Borwe []shows that the graet metho ( wth gve by (8 coverges R-suer learly a R-orer s. It s rove by Raya (Raya, M. 993 that the BB metho s global coverget for ay f the objectve fucto s a covex quaratc. However, for > o suerler covergece results have bee establshe for the BB metho, though umercal results cates qute ofte the BB metho coverges suerlearly (Yua,Ya-xag. he BB metho (Yua Ya-xag,. 6 erforms qute well for hgh mesoal roblems. he BB metho s ot mootoe, a t s ot easy to geeralze to geeral olear fuctos uless certa omootoe techques beg ale. herefore, t s very esrable to f ste-sze formula whch eables fast covergece a ossesses the mootoe roerty. he result of Barzla Borwe (Yua,Ya-xag has trggere off may researches o the steeest escet metho. For examle, see(da, Y.H. 3, (Da, Y. H. Yua, J. Y. a Y, Yua,., (Da, Y.H. a Y,Yua. 3, (Frelaer, A. J. M, Martez. B, Mola a M, Raya,. 999, (Noceal, J. A, Sarteaer. a C, Zhu.,( Fletcher, R., 987,( Raya, M. 997,( Fletcher, R. 5. I ths wor, we suggest the metho for the ucostrae mmzato of a fferetable fucto f ( : R R s efe by x x g, Where g s the graet vector of f at x a the scalar y s gve by ( y ( y ( Where x x, y g g. t t Whe f ( x x Gx b x c s a quaratc fucto a G s a symmetrc ostve efte matrx a by usg ELS the ( becomes t G ( t I ths case, s the Raylegh quotet of G at the vector. Sce G s symmetrc ostve efte the m max (3 Where m a max are resectvely the smallest a the largest egevalues of G, Hece,there s o ager of vg by zero eq. (. 3. Dervato of the ew ste Sze We ow that QN-le coto s H y - = - (4 multlyg eq. (4 by the covex combato ( ( y we get 58 (

3 Aust. J. Basc & Al. Sc., 5(8:57-68, I( y ( y y ( ( y Smlfy eq.(5a to get ( y ( y ( y I Solvg eq. (5b for α to have the equato for the ew ste-sze as t efe eq.(. (, s a arameter geerate by Uform strbuto, a H = I. Whe eq. ( reuce to y a whe eq. ( reuce to y, y otherwse a for ay value of the arameter θ (, whch s geerate by uform strbuto we wll get the ew ste-sze formula as t efe eq.(. 4. Covergece Proerty: For ay quaratc fucto: f(x = x t Gx b t x + c wth a SPD(symmetrc ostve efte Hessa matrx G, let x be the uque mmzer of f, {x } the sequece geerate by the ew metho from a gve vector x, a e = x - x for all (6 he, usg eq.( a the fact that g =Ax - b, where b = Ax, we have Ge = for all (7 Substtutg = e e + eq.(7 we obta for ay e +l = ( I-Ge. (8 Now for ay tal error e, there exst costats w, w,,..., w such that e w v w3 Where {v,v,, v }are orthogoal egevectors of G assocate wth egevalues {,,..., }. Usg eq.(4 we obta for ay teger, e w v (9 j Where w w w. ( j j he covergece roertes of the sequece {e } wll ee o the behavor of each of the sequece { w },. I geeral, these sequece wll crease at some teratos. However, the followg lemma shows that the sequece { w } wll ecrease Q-learly to zero. (5a (5b Lemma 4.: he sequece { w } coverges to zero Q-learly wth coverges factor ĉ =-( Proof: For ay ostve teger, m max. 59

4 Aust. J. Basc & Al. Sc., 5(8:57-68, m w w Sce satsfy eq. (3, we have w m w c w max Where ĉ =-(, m max Lemma 4.: f the sequece },{ w },...,{ w } lmf w. l { w all coverges to zero for a fxe teger l, l Proof: Suose, by the way of cotracto, that there exsts a costat such that 6. he, ( wl l for all ( By eq.( a eq.(7 t follows that the Raylegh quotet ca be wrtte as l 3 eg e. l eg e he, by eq. (9 a the orthoormalty of the egevectors{v,v,, v }, we obta 3 ( w. ( ( w Sce the sequece w },{ w },...,{ w } all covergece to zero, there exst ˆ suffcetly large such that { l ( w l for all ˆ. (3 By eq.( a eq.(3,for ay ˆ l Sce l ( w l ( w ( w, max he t follows from eq.(4 that max for all ˆ, 3 l Whch mles the bou l l max, max Fally, usg eq.( a the frst art of eq.(, we obta, for all ˆ, w l cˆ w l wl l, (4 for all ˆ, (5

5 Aust. J. Basc & Al. Sc., 5(8:57-68, Where m ĉ =max(,- < max (6 Because ths cocluso cotracts the hyothess ( we f that the lemma s true. heorem (4.3 establshes the covergece of the ew metho whe ale to quaratc fucto wth a symmetrc ostve efte Hessa matrx. heorem 4.3: Let f(x be a strctly quaratc covex fucto.let {x }be the sequece geerate by the ew graet * metho a x uque mmze of f. he, ether x j = x * for some fte j, or the sequece {x } coverges to * x. Proof: We ee oly coser the case whe there s o fte j such that x j = x *. Hece, t suffces to rove that the sequece {e }coverges to zero. From eq.( 6 a the a the orthoormalty of the egevectors we have ( e w herefore, the sequece of errors{e }coverges to zero f a oly f each oe of the sequeces { w }, for =,,, coverges to zero. Lemma (4. shows that { w } coverges to zero. We rove that { w } coverges to zero for by ucto o. herefore we let be ay teger from ths terval, a we assume that w },...,{ w } all te to zero. he for ay gve there exsts ˆ { suffcetly large such that ( w for all ˆ (7 From eq.( a eq.(7, we obta ( w max ( w For all tegers ˆ. Moreover, by lemma 4., there exsts ˆ such that ( w Now, let us say that s ay teger for whch ( w a ( w. Clearly, ( w ( w for j (8 (9 j Where j s the frst tha for whch ( w. he, by (8 a (9, we have max for j (3 3 hus usg eq.(3 a the frst art of eq. (, we obta w ˆ c w for j Where ĉ s a costat as eq.(6, whch satsfes c ˆ. Fally, usg the bou 6

6 Aust. J. Basc & Al. Sc., 5(8:57-68, m max w w m Whch s mles by exresso (3 a the frst art of eq.(, we coclue that 4 4 max m max m ( w ( w m m For all j ( w [( max m ] ( w. It follows from the cotos o a j that ( w s boue above by a costat multle of for all. Hece, sce. Further, eq.( roves the equalty chose arbtrarly small, we euce lm w as requre, whch comlete the roof. 5. Outles of the New Metho: Ste (: Gve x R. Set =, g. Ste( : Set x + =x +α. where α s the ste sze s comute by Usg eq.(. Ste (3: Chec for covergece.e. f g <, where s small Postve tolerace, Sto; otherwse cotue. Ste (4: Comute ew search recto efe by =-g +β - where β s Comute by the followg g y formula y Ste (5: If = or f the restart crtero Set =+, go to ste(. 6. Statstcs est: g g. g, [Powell, M.J.D., 977] ca be 6. Uform Dstrbuto: he uform strbuto s smle robablty strbuto. It ca be ether screte or cotuous. I the screte case, they ca be characterze by sayg that all ossble values are equally robable. I the cotuous case, oe says that all tervals of the same legth are equally robable. he cotuous uform strbuto s calle the rectagular strbuto because of the shae of ts robablty esty fucto. It s arameterze by smallest a largest values that the uformly-strbute raom varable ca tae, a a b. he geeral form of the formula of robablty esty fucto of the uform raom strbuto s efe as {f(x=(/(b-a, f a<x<b, otherwse }. he case where a= a b=s calle the staar uform, so that the raom varable ca taes values oly betwee zero a oe (Steel, G.D. a J. H, orre. 98, (Uform strbutato efto. 6. Ch-Square Dstrbuto: he strbuto of (Gree ch: rea ch square because of ts relato to s a the very mortat stuet ' s t strbuto. Ch-square s efe as a sum of squares of eeet staar ormally strbute varables wth zero meas a ut varace[uform strbutato efto]. 6.3 Gooess-of-Ft est: It s oe of ch -square alcatos whch s use to eterme whether or ot ata "ft" a artcular strbuto. he strbuto, whe assocate wth screte ata,s usually cojucto wth a test of gooess of ft. he test crtero s ( observe ex ece ex ecte 6

7 Aust. J. Basc & Al. Sc., 5(8:57-68, he sum s tae over all cells the classfcato system. Observe refers to the umbers observe the cells; execte refers to the average umbers of execte values whe the hyothess s true, that s the theoretcal values [Susa Dea a Barbara Illowsy]. we use ths alcato of ch -square to eterme whether the results of the ew ste-szes are uformly strbute or ot Dscusso: he results of the ew metho are reorte ables (-5 term of umber of fucto evaluato, the umber of teratos, the value of ste sze, a the mmum ot. he comarso of the ew metho wth BB-metho volve fve well ow test fuctos. All the result are obtae by usg ouble recso usg rogram wrtte FORRAN(. he comaratve Performace of the algorthm are evaluate by coserg the umber of fucto evaluato, the umber of teratos, the value of ste sze, a the mmum ot. he stog crtero s tae to be : 6 g x he umercal results of the ew metho showe that t s as effcet as BB-metho a there are mrovg the results for some values of the arameter. Each tables (-5 cosst of (5 values for the arameter θ, the frst two values of θ, that meas whe θ=,a θ= gve results of the two formulas for BB-Metho, a the other values for θ (, whch s geerate by uform strbuto gve the results of the reset wor. he results table ( ue to the Powell fucto wth =. he best results at θ= he results table ( ue to the Wolf fucto wth =. he best results at θ=.93459, θ=.938 he results table (3 ue to the Woo fucto wth =. he best results at θ=.938, θ= he results table (4 ue to the Shallow fucto wth =. he best results at θ=.46577, θ= he results table (5 ue to the Rece fucto wth =. he best results at θ=.7795 we use gooess - of- ft test whch s oe of ch square alcato to eterme whether the results of the ew ste-szes are uformly strbute or ot. We choose Powell fucto to chec f the 5 values of the ew ste-szes for ths fucto strbute uformly or ot. he results of usg SPSS acage to eterme whether these ste-szes results are uformly strbute or ot are eclare table (6. o exla ths let H : ew ste- szes results of Powell fucto strbute uformly(null hyothess H : ew ste-szes results of Powell fucto are ot strbute uformly (Alterate hyothess =.5 (Level of sgfcace he Asym.sg. s (>.5, t meas we accet ull hyothess a the ew ste-szes results of Powell fucto strbute uformly. Cocluso: New formula for calculatg ste-szes for cojugate graet metho s roose ths wor wth ts umercal results whch showe that t s a effcet as BB-metho a there are mrovg the results for some values of the arameter. I future research we shoul see more aroaches for estmatg the ste-sze as exactly as ossble a f some avalable techque to guaratee both the global covergece a quc covergece rate of cojugate graet methos. able :Comaratve betwee BB-Metho a the New Metho, (Powell Fucto(N= θ NOF NOI SEP- SIZE Excee oe E E E E E E E E E E E E E- MIN E E E E E E E E E E E E E E-8 63

8 Aust. J. Basc & Al. Sc., 5(8:57-68, E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E-9 able : Comaratve betwee BB-Metho a the New Metho,( Wolf Fucto(N= θ NOF NOI SEP- SIZE MIN E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E-3 64

9 Aust. J. Basc & Al. Sc., 5(8:57-68, E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E-3 able 3: Comaratve betwee BB-Metho a the New Metho, (Woo Fucto(N= θ NOF NOI SEP- SIZE MIN E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E-4 65

10 Aust. J. Basc & Al. Sc., 5(8:57-68, E E E E E E E E-4 able 4: Comaratve betwee BB-Metho a the New Metho, (Shallow Fucto(N= θ NOF NOI SEP-SIZE MIN E E E E E E E E E E E E E E E E fal E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E-9 66

11 Aust. J. Basc & Al. Sc., 5(8:57-68, able 5 :Comaratve betwee BB-Metho a the New Metho, (Rece Fucto(N= θ NOF NOI SEP-SIZE MIN E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E- able 6: est Statstcs VAR Ch-square Df Asym. Sg. REFERENCE Barzla, J. a J.M. Borwe, 988. wo ot ste sze graet methos. IMA J. Numercal. Aal., 8:

12 Aust. J. Basc & Al. Sc., 5(8:57-68, Cauchy, A., 847. Methoe geerale our la resoluto es systems equatos multaees,com. Re. Sc. Pars, 5: Da, Y.H., 3. Alterate ste graet metho. Otmzato, joural of mathematcal rogrammg a oeratos research. 5: Da, Y.H. J.Y. Yua a Y. Yua,. Mofe two-ot ste-sze graet methos for costrae Otmzato. Comutatoal Otmzato a Alcatos, : 3-9. Da, Y.H. a Y. Yua, 3. Alterate mmzato graet metho. IMA Joural of Numercal aalyss, 3: Da, Y.H. a H. Zhag,. A Aatve wo-pot Ste sze Graet Metho. Numercal Algorthms. Fletcher, R., 987. Practcal Methos of Otmzato. E., Joh Wley a Sos, Chchester, ISBN: , :45. Fletcher, R., 5. O the Barzlar-Borwe metho. Ale otmzato, 96: Frelaer, A.J.M., B. Martez, Mola a M. Raya, 999. Graet metho wth retars a Geeralzatos. SIAM J. Numer. Aal., 36: Noceal, J., A. Sarteaer a C. Zhu,. O the behavor of the graet orm the steeest escet metho. Joural of comutatoal otmzato a alcato, (. Powell, M.J.D., 977. Restart roceures for the cojugate graet metho. Mathematcal Programmg, : Raya, M., 993. O the Barzla a Borwe choce of ste legth for the graet metho. IMA J. Numercal Aal., 3: Raya, M., 997. he Barzla a Borwe graet metho for the large scale ucostrae mzato roblem. SIAM J. Otm., 7: Steel, G.D. a J.H. orre, 98. Prcles a roceures of statstcs. seco eto, Mcgraw-Hll Boo comay. ISBN: Susa Dea a Barbara Illowsy. he Ch-Square: Dstrbutato: eacher ' s Gue. htt://cx.org/cotet/m76/latest/ Uform strbutato efto, htt:// Yua Ya-xag, 6. A ew ste- sze for the steeest escet metho. Joural of Comutatoal Mathematcs, 4(: Yua,Ya-xag. Ste-Szes for Graet Metho. Acaemy of Mathematcs a Systems Sceces, Chese Acaemy of Sceces, Cha 68