Parallel Newtonian Optimization without Hessian Approximation. Khalil K. Abbo College of Computer sciences and Mathematics University of Mosul

Transcription

1 f. J. of Comp. & Mth s., Vol., No., 6 Pllel Newto Optmzto wthout ess Appomto Collee of Compute sceces d Mthemtcs Uvesty of Mosul eceved o: /9/5 Accepted o: 6//5 الملخص الغرض من هذا البحث هو اقتراح خوارزمية تعتمد على طريقة التوازي تستند ا لى طريقة نيوتن لحل مساي ل الامثلية اللاخطية باستخدام حاسبات متوازية من النوع (MIMD) وذلك بحل نظام معادلات خطية بطريقة الحذف لكاوس بشكل متواز بدلا من ا يجاد معكوس مصفوفة هيسيان لكي نتجنب الخطا الناتج من حساب معكوس المصفوفة ولتزيد من قوة العمليات الحسابية وكذلك لتقليل الزمن اللازم لحل المسالة. ABSAC he pupose of ths ppe s to toduce pllel lothms bsed o the Newto method fo solv o-le ucosted optmzto poblem (MIMD) pllel computes by solv le system pllel us Guss Elmto method the th fd vese ess mt to vod the eos cused by evlut the vese mt d lso to cese comput powe d educe u tme..itoducto: hs ppe s coceed wth fd locl mmum * of the ucosted optmzto poblem m f () whee d the objectve f: s twce cotuously dffeetble. Methods fo ucosted optmzto e eelly tetve methods whch the use typclly povdes tl estmte of *, d possbly some ddtol fomto. A sequece of tetes { } s the eeted ccod to some lothm. Usully the lothm s such tht the sequece of fucto vlues f s mootoclly deces (f deotes f( )). Due to pctcl pplctos of the ucosted optmzto poblem, cosdeble mout of effot hs bee epded o the developmet of effcet sequetl lothms fo the soluto of ths poblems. ecet dvces pllel comput techoloy hve mde t 69

2 possble to solve the optmzto poblem moe effectvely d ces comput powe fo ths eso t s tul tht thee s cesed teest des pllel lothms fo vous types of pplctos. I ths ppe, we wll dscuss pllel umecl lothm mely Newto lothm fo the eel ucosted optmzto poblem us multpocessos o (MIMD) pllel mches whch cosst of umbe of fully pommble pocessos cpble of smulteously pefom etely dffeet opetos o dffeet dt, whee ech pocesso hs ts ow locl memoy. Cocuet computto c be doe t dffeet levels. Ou focus s o us (MIMD) computes whee umbe of pocessos commucte d coopete to solve commo computtol poblem. Multpocesso pllel computto volves thee key edets hdwe, softwe (pomm lue, opet system d comple) d pllel lothms (See [8] o []). A popul clss of methods fo solv poblem s the Qus- Newto (QN) methods. Assume tht t the th teto, ppomto pot d mt e vlble, the the methods poceed by eet sequece of ppomto pots v the equto α d whee α > s the step- sze whch s clculted to stsfy cet le sech codtos d d s -dmesol el vecto epeset the sech decto. Fo QN methods, d s defed by : d whee f ) s the det vecto of f() evluted t pot ( d s ppomto of () G ( whee G f ( ) ) whch s coected o updted fom teto to teto eel s symmetc d postve defte, thee e dffeet choces of, we lst hee some must popul foms (See [5],[6]o[4]). ( )( ) (4) ( ) Whee d s clled k- oe fomul, the othe foms e BFGS d DFP whee BFGS (5) 7

3 Pllel Newto Optmzto DFP (6) QN methods metoed bove hve two dsdvtes: oe of whch QN lothms eque le sech whch s epesve pctce d he secod s the ccumulto of eo evlut the ppomte usse mt t ech teto. o ovecome these dsdvtes we use pllel Newto method wthout ppomt usse mt d elect le sech -Newto Method Newto method uses fst d secod devtves, the de behd ths method s s follows: Gve stt pot, we costuct qudtc ppomto to the objectve fucto tht mtches the fst d the secod devtve vlues t the pot. We the mmze the ppomte (qudtc) fucto sted of the ol objectve fucto. We use the mmze of the objectve fucto s the stt pot the et step d we epet the pocedue tetvely. If the objectve fucto s qudtc, the the ppomto s ect, d the method yelds the tue mmze oe step. If o the othe hd, the objectve fucto s ot qudtc the the ppomto wll povde oly estmte fo the posto of the tue mmze. We c obt qudtc ppomto to the ve twce cotuously dffeetble fucto f() us the ylo sees epso of f bout the cuet pot, elect tems of ode thee d hhe fo smplcty we let f ( ) q( ) f G (7) whee, d the mt G usully postve defte whe s some ehbohood of *. A uque mmze of q( ) ests f d oly f G s postve d Newto method s oly well defed ths cse. (See [5]). he ( ) s obted by fd the sttoy pot t q( ), whch eques the soluto of the le system G (8) the et step s the defed by (9) 7

4 We the pove tht the sequece { } coveece d the ode of coveece s two (See []). hese locl coveece popetes epeset the del locl behvo whch othe lothms m be evluted s f s possble (See [6]). I fct, supe le coveece of y lothm s obted f d oly f the step s symptotclly equl to the step ve by solv (8). hs fudmetl esults due to Des d Moe (974) emphses the mpotce of the Newto step fo locl coveece. wo qute dffeet clsses of methods fo solv the le system (8) e t teest: dect methods d tetve methods. I dect method, the system s tsfomed to system of smple fom e.. tul o dol fom, whch c be solved elemety wy. he most mpott dect method s the Guss elmto, we wll use t to solve the le system (8).. Guss Elmto Method A fudmetl obsevto Guss elmto s the elemety ow opetos, whch c be pefomed o the system wthout ch the set of soluto. hese opetos e: - dd multple of the th equto to the jth equto - Iteche two equtos. he de behd Guss elmto s to use such elemety opetos to elmte the ukows the system (8) symmetc wy, so tht t the ed equvlet uppe tul system s poduced, whch s the solved by bck substtuto. If. (whee G j ). he the fst step we elmte fom the lst (-) equto by subtct the multple L,,..., of the fst equto fom the th equto. hs poduces educes system of (-) equtos the (-) ukows,,...,, whee the ew coeffcets e ve by : ~ j j L ~ j ; L,,,..., If, we c elmte fom the lst (-) equtos. Afte m- steps, m of Guss elmto the mt G hs bee educed to the fom: 7

5 Pllel Newto Optmzto ( m) G M Whee (m) K K O j steps m,,,. ; M ( m) M ( m) ( m) s the educed elemet poduced fom ow opeto t ( ) he elemets m (,..., ) e clled pvotl elemets. hee e two dffcultes the Guss elmto method. he fst f zeo ( m) pvotl elemet s ecouteed.e fo some m, the we cot poceed. But ths cse does ot occu ou poblem, whch s stted (8). Sce G s sque mt d postve defte, ths mes tht ll dol elemets of G e o-zeo. Secod zeo pvot ect thmetc wll lmost vbly be polluted by oud eos such wy tht t equls some smll o-zeo umbe, ufotutely, thee s o eel ule whch c be used to decde whe pvot should be tke to be zeo. Wht tolece to use such test should deped o the cotet. hs questo d the stblty (wthout these two dffcultes) of Guss method teted [Lus, ] Guss Elmto Alothm Gve mt G G d vecto the follow lothm educes the system G to uppe tul fom,,,..,, sce G s postve d symmetc) Fo k:- Fo k: L k fo jk: ed ed ed ( k) k ( k) kk L ( k ) ( k) ( k) j j k kj L ( k ) ( k) ( k) k k ; 7

6 4- Pllel lothms fo fd Newto decto: I ode fo the lothm to be s effcet s possble, we must be ble to eplot the memoy hechy of the chtectue we e u o. he memoy hechy efes to dffeet kds of memoes sde compute. hese e fom vey fst, epesve d theefoe smll memoy t the top of the hechy, dow to slow. Chep d vey le memoy t the bottom (fo moe detl see []). Useful flot pot opetos c oly be doe t the top of hechy, the estes. Sce ete le mt cot ft sde the estes, t must be moved up d dow thouh the hechy. hs tkes tme d fo mmum pefomce should be optmzed fo ech chtectue. he Bsc le Aleb Subpoms BLAS e hh pefomce outes fo pefom bsc vecto d mt opetos. he BLAS e specfclly mche optmzed to eplot the memoy hechy. Level BLAS does vecto-vecto opeto, Level BLAS does mt-vecto opetos d Level BLAS does mt-mt opetos (see [7]). I ode to speed up Guss elmto, we wsh to use s hh level of BLAS s possble. hee e dffeet wys to pllelze the Guss elmto method. I ths ppe, we cosde two types of pllel Guss elmto, the fst s bsed o the ows of the mt G d the secod uses colums of G. 4-: Pllel Guss Elmto Bsed o ows (PGE) he most popul fom of Guss elmto s defed by subtct multple of ow of the mt G fom othe ows ode to educe (8) to uppe tul system, whch s the solved dectly by bck substtuto. We ssume tht the comput system cossts of (whee s the dmeso of the poblem) detcl depedet pocessos lso ssume tht the th ow of G s ssed to pocessos. At the fst ste, the fst ow of G s set to ll pocessos d the the elmto of the fst elemets of ows to c be doe pllel the pocessos P,,P. he computto s cotued by sed the ew secod ow of the educed mt fom pocessos P,,P, the do the clcultos pllel ; d so o [see fue, ] 74

7 Pllel Newto Optmzto P P P - sve,, G - sve secod ow -sve d eceve.fd ( ) j l j, j,..., d sed l j to p j - eceve d to fd L fom p s follows d L - do fo,, to fd the educed fom L sed fst to p,..., p - 4.,..., do fo - do fo,, L -eceve,..., fom pocessos P... P L d sed the vecto to P 5..eceve L j fom p d epet (4) fo 6.eceve the vecto to fd d fd ew fom 4. eceve L (,..., ) d fd sed t to P,,P G 4- If G uppe tul use bck substtuto to fd, ( ) ( ),..., d 5- sed ( ) to P fom d G G 6.- f < ε stop, othewse ;Go to Fue tsks of pocessos PGE hs ppoch hs two mjo dwbcks (See [9]) () thee s cosdeble commucto of dt betwee the pocessos t ech ste () the umbe of ctve pocessos deceses by oe t ech ste. 4-: Pllel Guss Elmto Bsed o Colums (PGEC) Let ( ), d G G e ve d ssume tht the comput system cossts of detcl depedet pocessos d fully commucted pocessos, lso ssume tht the jth colum (C j ). of the 75

8 mt G s ssed to pocesso j d colum he fst step s to fd L ;,... ssed to P. P the the vlues of L c be set to pocesso P j (j,..,), the the computto c be doe pllel ll pocessos s follows : fo ech pocessos j,, j L jj j ;,..., d pocesso compute the vecto fom m Lm m ; m,..., fo et step (see fue we c fd L ;,... p d the sed to pocessos P,,P d the pocess s epeted utl the mt G s tsfomed to uppe tul mt, we deote t G (k) ltte k steps d the bcksubsttuto s used to fd the vecto. k he ew pot s foud fom equto (9). A ths lothm the umbe of ctve pocessos deceses by oe t ech ste. he m dvte of ths lothm s tht we optmzed dt tsfe betwee pocessos sce we eed oly tsfe elemets sted of vectos, ths wll educe the tme equed to complete the computtos. 76

9 Pllel Newto Optmzto p p p p Step.sve,, C - fd L ;,..., d sed t to P,,P - do fo,., L ; ; C 4.step (k): - eceve fom C (,..., ) k P,,P ; s - sve C - eceve L fom P - do L C,..., step - fd L ;,..., d epet the pocess - sed L to P,,P - epet () fo C - sve C - eceve L fom P - do fo,, L C sed t to P - step - eceve L fom P - epet () fo d L C - the pocess epeted utl G (k) tul ( ) 4- sed C k to P - sve ; - eceve L,, - fom P - do fo.,,- L : : step (k) - ( k) k C,..., ( k) - sed C to P k k G C C C tul? d L to fd [ ] () L - use bck substtuto to fd C to P 4- sed ; to fd ew t stop othewse < ε.if 4-,,, ; G G epet fom step ( k) Fue tsks of pocessos PGEBC Numecl Emple: 77

10 78 We ve emple fom [] to llustte the tsks of pocessos the lothms PGE d PGEC -Mmze the fucto (). 4 ),, ( f ] [ ; ] [ whee ; ) ( G. Us PGE: We eed pocessos, see fue ()

11 Pllel Newto Optmzto P P P - sve - fd,, G 4 L d L vlue to P, P - sed, sed these [ 4 ] to P, P 4- do fo,, --(-)(/)-4 --(-/)(/)5/4 5 [ 4 ] 4 5-eceve L fom P 6- do (5/4)-(-)(-4)-7/4 / 4 d 7 / 4 sed to P 7- / / 8- < ε stop - sve - eceve [ 4 4 ] [ 4 ] d L - - do,, -4 (-) -(-)()8 [ 4 8] 4 sed to P -sve - eceve [ ] - do fo,, -(/) -(/)(-4)8 -(/)()5/ 4 eceve d L d fd 8 L sed L to 4 P 5- epet step () fo 8-(-)(-4) (5/)-(-)87/ 6- G 7-eceve / fom P 8- use bck substtuto to fd d sed t to P Fue () sks of pocessos us PGE lothm - Us PGEC: 79

12 We eed 4 pocessos to solve the poblem ve () s show fue (4) P ` P P P 4 Step - sve -sve - sve -sve X C [ / ] [ / ] [ 4 ] - fd 4 L, L d sed the to P,,P,P4 - do 4 - -(/) [ ] C step - eceve [ 4 4 ] C - eceve L-,,L/ - do 4-(-)(-4)-4 -(/)(-4)8 [ ] C step - fd L8/-4- d sed t to P,,P4I - do 8-(-)(-4) sed t to P C [ ] -eceve L,L -do - (-)()8 -(/)()-5/ [ ] C 8 5/ step - eceve L - do -5/-(-)(8)7/ [ ] C 8 7/ sed t to P [ ] / -eceve L,L - do --(-)(-/)-4 --/(-/)5/4 [ ] / 4 5/4 step - eceve L - do 5/4 -(-)(-4)-7/4 [ / 4 7/4] () sed t to P C () () C,, () - G 4 4 / 4 7 / 4 - use bck 8 7/ substtuto to fd / Fue (4): sks of pocessos fo PGEC lothm 5-Numecl epemets: 8

13 Pllel Newto Optmzto Both the mout of tme equed to complete the clcultos d the subsequet oud-off eo deped o the umbe of flot thmetc opetos eeded to solve oute poblem. he mout of tme equed to pefom multplcto o dvso o compute s ppomtely the sme d s cosdebly ete th tht equed to pefom ddto o subtcto (see []). he totl umbe of thmetc opeto depeds o the sze s follows: Multplctos / dvsos: 5 Addtos / subtctos: 6 Fo le, the totl umbe of Multplctos d dvsos s ppomtely s s the totl umbe of ddtos d subtctos. ece f the ll opetos doe o sequetl compute the the ccumulto eo wll be le. O the othe hd, dvd the opeto o dffeet pocessos wll educe the mout of eos. As fo the mout of tme, If we ssume t s tme equed to complete the clcultos sequetl compute, we epect the tme equed to pefom the sme clcultos the pllel lothms poposed s t whee p umbe of pocessos. p I the lothms PGE d PGEC, thee s o eed to le sech d mt veso, ll opetos e doe us ody thmetc opetos. Also, the umbe of fucto evlutos e oly, sce the Guss method ves ect solutos show the ve emple. Cocluso: I ths ppe we poposed two pllel lothms fo solv ucosted optmzto poblem. hese methods c be eteded to y o-homoeous le system wth postve defte mt. hese methods do ot eque to vecto multplcto d mt veso s show the umecl emple. EFEENCES 8

14 [] Budy B.. (984) A Bsc optmzto methods Lodo.Edwd Aold Bdfod squ. [] Des J. d J. Moe (974) A chctezto of supe le covee d ts pplcto to qus-newto Methods Mths. of Computto. 8,pp [] Douls J. d B. chd () Numecl Methods thd edto [Books/ Cole] homso le: Ic [4] Edw K. d. Stslw () A toducto to optmzto Joh Wley & Sos. Ic [5] Fletche. (987) Pctcl optmzto secod edto, Joh Wley & Sos. Ltd [6] Fletche. (99) A ovevew of ucosted optmzto Numecl Alyss epot NA/49. [7] Jmes, W.;. Mchel,; A. ek, d V.V, (99) Pllel Numecl Le Aleb,Mthemtcl Isttute uvesty. [8] Jod,. d G. Alhbd, () Fudmetls of pllel pocess, USA: New Jesey, Petce ll. [9] Khlf M. 99 Pllel Numecl Alothms fo solv ODEs Ph.D. hess leds uvesty. [] Luss () Numecl Mthemtcs d Scetfc computto Ake-Bjolck Vol.. [] Ms O. (994) Pllel Gdet Dstbuto ucosted optmzto /sech?hl e&sx&ospell&esum&ctesult&cd &qms O. 994%E%8%9CPllelGdetDstb [] Mchel, W. 4 Pllel Guss Elmto http: // www. studet.cs.uwteloo.c/~cs775/poject/de.html [] obet, M. (4) Newto s Method fo ucosted optmzto, Cs775, Uvesty of Wteloo, Msschusetts Isttute of echoloy. 8