A ew type of optmzato method based o cojugate drectos Pa X Scece School aj Uversty of echology ad Educato (UE aj Cha e-mal: pax94@sacom Abstract A ew type of optmzato method based o cojugate drectos s proposed ths paper It ca be proved that ths type of method has quadratc termato property wthout exact le search he ew method requres oly the storage of 4 vectors such that t s sutable for large scale problems Numercal expereces show that the ew method s effectve Keywords-optmzato; large scale problems; cojugate drectos; quadratc termato property I INRODUCION Optmzato s a mportat tool decso scece aalyss of physcal systems ad cotrol egeerg Cojugate gradet (CG methods are some of the most useful algorthms for ucostraed optmzato problem m f ( x x R ( by a sequece of le searches x x αd ( from a user suppled estmate x he step legth α ca be obtaed by the le search methods Cojugate gradet algorthms are wdely used for the two otable propertes: Frst they eed ot store ay matrx thus are sutable for large scale problems; Secod whe exact le search (ELS s tae CG algorthms have the quadratc termato property whch reflects the effcecy of the algorthms he search drectos of CG methods are geerated as follows d g d ( where g f( x d gad s geerated by the defto of cojugated drectos d G d (4 where G s the Hessa of f at the pot x Varous formulas for are gve for dfferet CG algorthms[][][] g g (5 g g g ( g g (6 g g g ( g g (7 d ( g g I practce a exact le search (ELS s ot usually possble for t requres too may evaluatos of objectve fucto f ad possbly the gradet f More practcal strateges perform a exact le search (ILS to detfy a step legth However for a geeral quadratc fucto CG methods ca t guaratee global covergece whe perform ILS[4] Eve for a postve quadratc fucto whe perform ILS CG methods wll lose the quadratc termato property Example : Cosder a -dmesoal postve quadratc fucto f ( x x Ax b x whch gradet s g( x Ax b Suppose that { d d} s a set of ozero cojugate vectors Case : Gve a startg pot x let us geerate the sequece { x * } gve explctly by by Eq( where α s obtaed by ELS g d α (8 d Ad hus x x αd x x αd d d Sce g( x s lear ad d d are cojugate we have ( α d Ad d d So ad x s the mmzer of the fucto f ( x hs llustrates the cojugate drectos are good search drectos whe perform ELS Case : Startg from the same x let us perform ILS alog the search drecto d to get the step legth α α ad mproved x x α d he startg from the same x perform ELS alog d to get the step legth α ad x x αd By the property of ELS we have d But x s ot the mmzer of f ( x for ( α ( g( x d g( x Ad d g( x A( x x d (9 d A( x x Case llustrates that whe perform ILS CG methods wll lose the quadratc termato property However some Publshed by Atlats Press Pars Frace the authors
modfcato ca be tae o the secod search drecto Let p d γ d ( be the search drecto the secod step ad α be the step legth x x α p the g( x d g( x d αd A( d γd ( ( d A x x αγ d Ad If we correctly select α ad γ we ca get d ad d ad x s the mmzer hs llustrates that whe perform ILS f some modfcato s tae o the curret cojugate drecto we wll get a better search drecto he ew method ths paper s based o ths truth II DERIVAION OF HE NEW MEHOD Cosder -dmesoal postve quadratc fucto f ( x x Ax b x c ( Suppose p s the th search drecto x s the pot after the th le search he aylor expaso of f ( x ca be wrte as f ( x f( x g p p G p m( p (4 where p x x G A he best search drecto th step s p arg m mp ( It s dffcult to get the best drecto R space f the Hessa s uow We ca tae the dea of CG methods that uses lear combato of two lear depedet vectors to approxmate p Let accessoral vector d cojugate wth p thus d Ap the from the ht of Example the search drecto th step ca be wrtte as p ap ad d ca be geerated form FRCG or PRCG as d g p (5 Obvously f a p s cojugate gradet drecto We amed the ew method Modfed Cojugate Method (MCG he ey to obta MCG search drecto s the selecto of combato coeffcets a ad a whch s smlar to the dea of Yua [5] LetU [ p d ] y [ a a] he p Uy (6 mp ( f( x g p p G p (7 f ( x ( U g y y U G Uy ϕ( y Let gˆ U g ˆ G U G U he we ca rewrte If y ϕ ( y f( x ˆ ˆ g y y Gy (8 arg m ϕ( y s obtaedwe ca get the mmzer of mpo ( the mafold { p ap ad a R a R} to approxmate p wrtte as p Uy p s the search drecto of MCG If G s postvethe Ĝ s also postve So we ca get p Uy (9 where y Gˆ gˆ If Ĝ s ot postvewe ca tae p ( sg g d d ( he algorthm s as follows Step : Gve x tolerace ε set ; Step : Calculate g f( x f g ε stop ad x s the mmzer; Otherwse go to step ; Step : If set p g go to step 6;Otherwse g ( g g set d g p where ; g g Step 4: Calculate s p Gp t d Gd ; Step 5: If t the p sg( d g d; Else f s the p ( sg p g p ; p g d g Otherwse p p dgo to step 6; s t Step 6: Searchα whch satsfes Wolfe codto: f ( x α p f( x cα f( x p f( x α p p c f( x p < c < c < Step 7: Set x x αp go to step Remar: Usually α s always tred frst whether t satsfes Wolfe codto whe performg ELS hat s true MCG III PROPERIES OF MCG heorem he search drectos geerated by MCG are descet drectos Proof: For the drecto geerated by Eq( p g sg( g d g d < For the drecto geerated by Eq(9 p g y U g ˆ g UG U g ( g p ( g d < p G p d G d So the search drectos geerated by Eq(9 ad Eq( are descet drectos Publshed by Atlats Press Pars Frace the authors
Lemma Suppose that MCG s used to solve -dmesoal postve quadratc problem defed by Eq( the teratve pots are x x ad search drectos geerated by MCG are p p p the { p p p } s a set of ozero cojugate vectors ad for x s the mmzer of the objectve fucto o the mafold { x αp α R} Proof: Sce the objectve fucto s postve quadratc form Eq(8 ad Eq(9 we ca coclude that whe usg MCG the step legth α wll be for For p Uywhere y arg m f( x U y U g x Uy p g y U g We have ( e U g hus It shows that x s the mmzer o { x αp α R} whch meas that x s obtaed by performg ELS alog p from x p U y where y arg m f( x U y p g s d g d g t t It shows that p d We ca coclude that { } p d d s a set of ozero cojugate vectors from Eq(5 ad p d herefore { } p p p s a set of ozero cojugate vectors Lemma Suppose that f ( x s a -dmesoal postve quadratc fucto { d d d } s a set of ozero cojugate vectors Let x R be ay startg pot f ELS s performed alog d d d to get the sequece x x x the x s the mmzer of f ( x [6] heorem MCG has the quadratc termato property Proof: From Lemma we ow that for postve quadratc fucto MCG equals to the cojugate gradet methods startg from the secod teratve pot whch performg ELS Combed wth Lemma t ca be cocluded that MCG has the quadratc termato property MCG ca be vew as a d of Iterated-subspace mmzato methods [7] he frame wor of Iteratedsubspace mmzato methods (ISM s as follows Step : If the th teratve pot x satsfes the toleracestop ad output x ; Otherwse cotue; t Step : Costruct Z R ad solve t -dmesoal subproblem y arg m f( x Z y ; t Step : x x Zy go to step MCG s a d of specal Iterated-subspace mmzato method whch Z [ p d] t ad Newto method s used to solve the -dmesoal sub-problems So the covergece of MCG s based o the covergece of Iterated-subspace mmzato methods whch show as follows [8] heorem Suppose that f ( x s a dfferetable fucto the level set L { x R f( x f( x } s bouded f ( x s Lpschtz-cotuous ad for some ε > < α < the sequece of pots geerated by the ISM satsfes Goldste codto f ( x g( x Zdz f( x f( x αg( x Zdz ( ad gx ( Zd z ε ( Z g( x d z where d s the soluto of subproblem m f ( x Z d z t d z R z the the ISM algorthm globally coverges to a statoary pot for problem( from ay startg pot Co [7] proofed that f egatve gradet vector s the subspace spaed by Z ad whe performg ILS wth Wolfe codto sub-problems ISM wll satsfes Eq( ad Eq( thus covergece ca be guarateed Obvously MCG satsfes the codtos above IV IMPLEMENAION OF MCG MEHOD he ey to mplemet MCG method s to calculate s p Gp ad t d Gd For the Hessa G s ot easy to calculate ad store t s reasoable to calculate s ad t approxmately We ca use the approxmato: g( x hp g( x Gp h ( g( x hd g( x Gd h (4 8 for some small dfferecg terval h h s farly close to optmal terval value[6] he s ad t ca be calculated he algorthm usg ths approxmato s amed MCG o reduce evaluatos of gradet aother approxmato ca be used: Gp α (5 where α s step legth he algorthm usg ths approxmato s amed MCG Publshed by Atlats Press Pars Frace the authors 4
he algorthms MCG MCG ad PRCG are mplemeted by Matlab software wth the same stoppg 5 crtero < he test fuctos are as follows F:Exteded Rose-broc / f( x ( x x ( x * [ ] f x x ( F:rda f( x ( x x x ( F:Power f( x x * x [ ] f( x F4:Exteded Beale [5 x ( x ] / f( x [5 x ( x] [65 x ( x] x [ ] F5:Noda f ( x ( x x ( x x - * [ ] f( x he umercal results of MCG MCG ad PRCG are show ABLE I he results are form of the umber of fucto calls Nf the umber of gradet calls Ng the umber of teratos NI the cetral processor ut tme CPU the orm of gradet at mmzer Norm(g ad the mmum of the fucto Fvl he results show that both MCG ad MCG outperform PRCG Ether MCG or MCG eeds o more tha 4 -dmesoal vectors to mplemet hus they are sutable for large scale problems such as -dmesoal problems or eve hgher dmesoal problems Comparg wth MCG MCG eeds less fucto calls but more gradet calls herefore whe evaluato of the objectve fucto s much easer tha ts gradet MCG s the prorty opto V CONCLUSIONS he ew optmzato method amed MCG preseted ths paper ca be vew as a d of Iterated-subspace mmzato methods whch global covergece s guarateed It has the quadratc termato property wthout ELS Numercal expermets show that MCG s more effcet ad precse tha PRCG Furthermore MCG s also sutable for large scale problems ACKNOWLEDGMEN hs wor s supported by NSFC (No 5975 No 57598 he foudatos support s greatly apprecated REFERENCES [] Fletcher R ad Reeves CM Fucto Mmzato by Cojugate Gradets[J] Computer Joural 964 7 : 49-54 [] Soreso H W Comparso of some cojugate drecto procedures for fucto mmzato [J] Joural of the Fral Isttute 969 88 :4-44 [] Lu Y Storey C Effcet geeralzed cojugate gradet algorthms[j] Joural of optmzato theory ad applcatos 9969: 9-5 [4] Da Yad Yua Y Nolear Cojugate Gradet Methods [M] Shagha: Shagha Scece ad echology Press [5] Yua Y ad Stoer J A subspace study o cojugate algorthms[j] ZAMM Z agew Math Meth 995 75(:69-77 [6] Nocedal J ad Wrght S J Numercal Optmzato[M] Sprger Seres Operatos Research New Yor: Sprger-Verlag 999 [7] [o A R Gould N L M ad Sarteaer Aetal O Iterated- Subspace mmzato methods for olear optmzato[a] L Adams ad L Nazareth Lear ad Nolear cojugate gradet related methods[c] AMS-IMS-SIAM 996:5-78 [8] Des J E ad Schabel RB Numercal methods for ucostraed optmzato ad olear equatos[m] Pretce-Hall Eglewood Clffs NJ 98 ABLE I Performace of MCG MCG ad PRCG Fucto Dmeso Algorthm Nf Ng NI CPU Norm(g Fvl MCG 76 7 47 6E-7 45E-7 MCG 79 9 6 47 5E-7 59E-7 F PRCG 8 54 8 47 84E-7 54E- MCG 76 7 78 47E-7 6E-6 MCG 79 9 6 78 765E-7 59E-6 PRCG 8 54 8 9 54E-6 59E- MCG 55 8 77 6 79E-6 7E- F MCG 55 77 6 75E-6 E- PRCG 9 56 74E-6 549E- MCG 57 44 86 469 97E-6 5E-4 Publshed by Atlats Press Pars Frace the authors 5
F F4 F5 MCG 57 859 86 47 98E-6 5E-4 PRCG 9 7 57 46 896E-6 64E- MCG 7 5 47 998E-6 947E- MCG 7 6 5 46 998E-6 947E- PRCG 86 6 89E-6 76E- MCG 57 7 78 97 977E-6 75E- MCG 57 55 78 5 977E-6 75E- PRCG 784 765 4 4 78E-6 67E- MCG 5 9 9 8E-6 6E- MCG 6 9 8E-7 54E-6 PRCG 69 59 7 47 4E-6 95E- MCG 5 9 9 78 8E-6 64E- MCG 6 9 78 59E-7 8E-5 PRCG 7 6 8 5 94E-6 94E- MCG 5 75 6 46 65E-7 44E-8 MCG 64 6 6 6 96E-8 69E-9 PRCG 79 6 79 95E-6 6E- MCG 59 8 7 94 6E-7 4E-9 MCG 7 9 4 9 68E-8 79E- PRCG 98 79 7 5 848E-6 79E-6 Publshed by Atlats Press Pars Frace the authors 6