New Inexact Parallel Variable Distribution Algorithms

Transcription

1 Computatona Optmzaton and Appcatons 7, ) c 1997 Kuwer Academc Pubshers. Manufactured n The Netherands. New Inexact Parae Varabe Dstrbuton Agorthms MICHAEL V. SOLODOV soodov@mpa.br Insttuto de Matematca Pura e Apcada, Estrada Dona Castorna 110, Jardm Botanco, Ro de Janero, RJ, CEP , Braz Receved June 6, 1995; Revsed October 4, 1995 Abstract. We consder the recenty proposed parae varabe dstrbuton PVD) agorthm of Ferrs and Mangasaran [4] for sovng optmzaton probems n whch the varabes are dstrbuted among p processors. Each processor has the prmary responsbty for updatng ts bock of varabes whe aowng the remanng secondary varabes to change n a restrcted fashon aong some easy computabe drectons. We propose usefu generazatons that consst, for the genera unconstraned case, of repacng exact goba souton of the subprobems by a certan natura suffcent descent condton, and, for the convex case, of nexact subprobem souton n the PVD agorthm. These modfcatons are the key features of the agorthm that has not been anayzed before. The proposed modfed agorthms are more practca and make t easer to acheve good oad baancng among the parae processors. We present a genera framework for the anayss of ths cass of agorthms and derve some new and mproved near convergence resuts for probems wth weak sharp mnma of order and strongy convex probems. We aso show that nonmonotone synchronzaton schemes are admssbe, whch further mproves fexbty of PVD approach. Keywords: convergence parae optmzaton, asynchronous agorthms, oad baancng, unconstraned mnmzaton, near 1. Introducton We consder the genera unconstraned optmzaton probem mn x R n f x), 1) where f : R n R. We frst state the orgna PVD agorthm [4]. Let x R n be parttoned nto p bocks x 1,...,x p, such that x R n, p =1 n =n. These bocks of varabes are then dstrbuted among p parae processors. Each processor has the prmary responsbty for updatng ts bock of varabes by sovng the paraezaton probem see Agorthm 1 beow). The remanng secondary varabes are aowed to change n a restrcted fashon aong some easy computabe drectons. The dstnctve nove feature of ths agorthm s the presence of the forget-me-not term x ī + D ī µ n the parae subprobems ). Ths work was started when the author was wth the Computer Scences Department, Unversty of Wsconsn- Madson, U.S.A., and was supported by Ar Force Offce of Scentfc Research Grant F and Natona Scence Foundaton Grant CCR

2 166 SOLODOV The presence of ths term aows for a change n secondary varabes. Ths makes PVD fundamentay dfferent from the bock Jacob [1], coordnate descent [0] and parae gradent dstrbuton agorthms [10]. The drectons D ī are typcay easy computabe steepest descent or quas-newton drectons n the space of the correspondng varabes. The forget-me-not approach mproves robustness and acceerates convergence of the agorthm and s the key to ts success. The paraezaton phase s foowed by a smpe synchronzaton step whch pcks up a pont wth the objectve functon vaue at east as good as the smaest among a the new ponts computed by the parae processors. Agorthm 1 PVD). Start wth any x 0 R n. Havng x, stop f f x ) = 0. Otherwse, compute x +1 as foows: ) Paraezaton: For each processor {1,...,p}compute y,µ ī ) arg mn ψ x x,µ ):= f x,x ī + D ī µ ). ),µ ) Synchronzaton: Compute x +1 such that f x +1 ) mn {1,...,p} ψ y,µ ī ). 3) We w sometmes refer to x as the base pont at the + 1)-st teraton. In the above agorthm denotes the compement of n the set {1,...,p}and µ R p 1. The matrx D ī s an n p 1) bock dagona matrx formed by pacng the bocks d 1,...,d p 1 d t R n t,t =1,...,p 1) of an arbtrary drecton d R n aong ts bock dagona as foows: D ī := d 1 d... d 1 d d p In the orgna PVD agorthm the proposed synchronzaton step conssts of mnmzng the objectve functon n the affne hu of a the ponts computed n parae by the p processors. In [4] t was shown that every accumuaton pont of the PVD terates s a statonary pont of f ) f an exact goba souton to subprobems ) s computed at every teraton. It was aso estabshed that, n the strongy convex case, the terates converge to the probem souton at a near rate. We pont out that the goba souton requrement n the genera nonconvex) case s mpractca. In Secton 3 we show that t s possbe to get rd of ths requrement by mposng a certan suffcent descent condton nstead. Secton 3 aso contans some new convergence resuts for probems wth weak sharp mnma of order. We note that the orgna requrement of exact subprobem souton s aso undesrabe. In Secton we descrbe an

3 NEW INEXACT PARALLEL VARIABLE DISTRIBUTION ALGORITHMS 167 agorthm wth nexact subprobem souton n the convex case and derve a sharper near convergence resut than the one gven n [4]. We emphasze that the suffcent descent and nexact subprobem souton approaches provde a fexbe framework that aows for effectve oad baancng among the parae processors. In Secton 3 we aso exhbt that synchronzaton step can be combned wth nonmonotone stabzaton schemes, f needed. One of the keys to our anayss s mposng certan reasonabe condtons on the choce of drectons for the change n secondary varabes. The choce of those drectons s very mportant for the success of the PVD approach. Ths fact was emprcay observed n [4]. It can aso be vvdy ustarted by theoretca consderatons for the constraned optmzaton probems [19]. We brefy descrbe our notaton now. The usua nner product of two vectors x R n, y R n s denoted by x, y. The Eucdean -norm of x R n s gven by x = x,x. The cosed unt ba n R n s denoted by B := {x R n x 1}. For a nonempty cosed) set X R n,d,x)denotes the Eucdean dstance to the set X. For a rea-vaued matrx A of any dmenson, A denotes ts transpose. For a dfferentabe functon f : R n R, f w denote the n-dmensona vector of parta dervatves wth respect to x, and f w denote the n -dmensona vector of parta dervatves wth respect to x R n, =1,...,p. If a functon f ) has Lpschtz contnuous parta dervatves on R n wth some constant L > 0, that s f y) fx) L y x x,y R n, we wrte f ) C 1 L Rn ). By R-near convergence and Q-near convergence, we mean near convergence n the root sense and n the quotent sense, respectvey, as defned n [13]. We now state a cassca emma [16], p. 6), as we as another emma a sght modfcaton of [16], p. 44) that w be used ater. Lemma 1. Let ϕ ) C 1 L Rn ), then ϕy) ϕx) ϕx), y x L y x x, y R n. Lemma. Let {a } and {ɛ } be two sequences of rea numbers such that ɛ 0, =0 ɛ <, and a +1 a + ɛ for = 0, 1,... It foows that ether the sequence {a } s unbounded beow, or t converges.. PVD wth nexact subprobem souton In ths secton we propose a computatonay mportant modfcaton of the PVD agorthm n whch the subprobems ) n the Agorthm 1 are soved approxmatey. It s cear that n practce nsstng on exact souton of those subprobems s undesrabe, and often unreastc. Even when t s possbe to compute these soutons accuratey, t can be wastefu dong so, especay n the nta stages of the mnmzaton process. Our resuts show that there s no need to wat unt exact soutons to a the subprobems are found whch can resut n consderabe de tmes for processors that have aready

4 168 SOLODOV competed ther work). Instead, we can accept the current approxmatons to soutons of the subprobems and proceed to the synchronzaton step, provded those approxmatons are reasonaby good. Ths approach s more robust and aows for fexbe synchronzaton schemes thus makng t easer to acheve good oad baancng among the parae processors. In partcuar, we show that we can sove the subprobems to wthn ε-statonarty see 5)), and yet guarantee the near convergence rate f f ) s strongy convex. The toerance for an -th parae subprobem depends neary on the the norm of the correspondng porton of the gradent at the current base pont see 5) and 10)). By makng an expct use of the forget-me-not terms n the subprobems, we aso mprove on the near convergence resut gven n [4]. In [4] t s estabshed that, for the strongy convex case, the foowng estmate s vad x x c 1 1 c ), p where x s the unque) souton of the probem, p s the number of parae processors, and c 1, c are postve constants. Ths resut s not qute satsfactory because the presense of p n the denomnator suggests that the convergence speed goes down as the number of processors used ncreases. We pont out that the proof gven n [4] fas to make use of the forget-me-not terms whch are the key to the agorthm. By refnng the proof, we obtan a better convergence speed estmate x x c 1 1 c 3 ), where c 3 > 0 does not depend on p. Therefore convergence speed of the agorthm does not deterorate as the number of processors used ncreases, provded certan natura condtons are mposed on the forget-me-not terms. We consder the foowng agorthm. Agorthm. Start wth any x 0 R n. Havng x, stop f f x ) = 0. Otherwse, compute x +1 as foows: ) Paraezaton: For each processor {1,...,p}compute y,µī)as an ε,-approxmate souton see 5)) of mn ψ x x,µ ):= f x,x ī + D ī µ ).,µ ) Synchronzaton: Compute x +1 such that f x +1 ) y,µ ī ). 4) mn {1,...,p} ψ To make the paraezaton step precse, we say that the current approxmaton to the souton of a subprobem s admssbe f t beongs to an ε-statonary set [18] of ths subprobem. The paraezaton subprobems are therefore equvaent to computng a pont y,µ ) ī X, s ε,) := { x,µ ) R n +p 1 ψ x,µ ) } ε,. 5)

5 NEW INEXACT PARALLEL VARIABLE DISTRIBUTION ALGORITHMS 169 We frst estabsh some premnary resuts. Let A be an n n + p 1) matrx defned by ) I A 0 = 0 D ī, where I s an n n dentty matrx. We assume that every bock dt of D ī that s dt =1, t=1,...,p. Then for any y Rn+p 1 we have A y = = n j=1 n +p 1 j=1 = y, n +p 1 y j + y j j=n +1 y j d j s normazed, 6) where the frst equaty foows from the bock dagona structure of D ī. Hence A = A ) =1. Lemma 3. 1,...,p. If f ) C 1 L Rn ) then ψ, ) C1 L Rn +p 1 ) for any = 0, 1,... and = Proof: Note that ψ x f ) x,x ī +µ D ī,µ ) = D ī ) f x,x ī = A) f x,x ī + D ī ) µ +µ D ī ) ) 7) For any x,µ ), z,ν ) R n+p 1 we have ψ x,µ ) ψ z,ν ) = A) f x,x ī + D ī µ ) )) f z,x ī +D ī ν A ) f x, x ī + D ī µ ) f z,x ī +D ν ) ī L x z A µ ν ) = L x,µ ) z,ν ), where the second nequaty foows from the fact that A ) =1, and f ) CL 1Rn ); the ast equaty foows from 6). We thus estabshed that ψ, ) C1 L Rn+p 1 ), for a = 1,...,p, =0,1,... Lemma 4. If f ) s strongy convex wth moduus θ>0then ψ, ) s strongy convex wth moduus θ>0for any = 0, 1,...and = 1,...,p.

6 170 SOLODOV Proof: Makng use of 7), we have ψ x,µ ) ψ z,ν ), x,µ ) z,ν ) = A ) f x, x ī + D ī µ ) ) f z,x ī +D ī ))) x z ν µ ν = f x, x ī + D ī µ ) ) f z,x ī +D ī )) ν A x z µ ν = f x, x ī + D ī µ ) ) f z,x ī +D ī )) x z ν D ī µ ν ) ) x z θ D ī µ ν ) ) = θ x z A µ ν = θ x,µ ) z,ν ), where the nequaty foows from strong convexty of f ), and the ast equaty foows from 6). Hence ψ, ) s strongy convex wth moduus θ. For smpcty of presentaton, from now on we assume that d t = t f x ) t f x ), t = 1,...,p. For ths choce of drectons, we have f x ) d 1, 1 f x ) f x ) 1 f x ) ). A f x ) = d 1, 1 f x ). d +1, +1 f x ) = 1 f x ). +1 f x ). d p, p f x ). p f x ) Hence, by 7), ψ x, 0 ) = A ) f x ) = fx ). 8) The atter property enabes us to expcty reate soutons of the parae subprobems ) to the progress beng made towards sovng the orgna probem 1). Ths s the key to our generazatons as we as mproved convergence resuts.

7 NEW INEXACT PARALLEL VARIABLE DISTRIBUTION ALGORITHMS 171 We note that nstead of the scaed gradent drectons we coud take any other drectons satsfyng the natura condtons dt, t f x ) σt t f x ) ), t = 1,...,p, where σ t ) are forcng functons see [13], p. 479). Dependng on the partcuar forcng functons, some arguments n the subsequent anayss may need to be changed. We fnay state a usefu emma whch s the bass for devsng agorthms wth nexact subprobem souton n the convex case. Ths resut s a smpfcaton of [18] Lemma.4 for smooth unconstraned case. We ncude the smpfed proof for competeness. Lemma 5. Let ϕ ) be convex and dfferentabe. Let x X s := arg mn x R n ϕx) and x X s ε) := {x R n ϕx) ε},ε 0. Then ϕx) ϕx ) εdx, X s ). If ϕ ) s strongy convex wth moduus θ>0,then ϕx) ϕx ) ε θ. Proof: Let x X s ε) and x be the orthogona projecton of x onto X s. By convexty of ϕ ),wehave ϕx) ϕx ) ϕx), x x ϕx) x x εdx, X s ). For the second asserton, just note that [16], p. 4) for any x R n θϕx) ϕx )) ϕx). The proof s compete. We are now ready to prove our man resuts. Theorem 1. Suppose f ) s strongy convex wth moduus θ>0and f ) C 1 L Rn ).If max {1,...,p} ε, <, =0 9) then every sequence {x } generated by Agorthm converges to the souton x of1). Moreover, f ε, β f x ), 0 β< θ L 10)

8 17 SOLODOV then {x } converges to x R-neary: x x )1 θ fx0 ) f x)) 1 θθ Lβ ) L Proof: For any teraton = 0, 1,...and any processor = 1,...,p, by 5) and Lemma 5, we have that ). ψ y,µ ) ī ψ + ε, θ, 11) where ψ s the exact optma vaue of the correspondng subprobem. Defne an auxary pont R n +p 1 z,νī We further obtan f x ) f y, xī + D ī µī ) := x,0 ) 1 L ψ x,0 ). ) = ψ x, 0 ) ψ y,µ ) ī ψ x,0 ) ψ ε, ψ 1 L θ x,0 ) ψ z,ν) ī ε, θ ψ x,0 ) ε, θ = 1 L f x ) ε, θ, 1) where the frst nequaty foows from 11), the thrd nequaty from Lemma 1, and the ast equaty from 8). By 4), we have f x ) f x +1 ) f x ) f y, xī + D ī ) µī 1 L f x ) 1 max θ {1,...,p} ε,. 13) From 13) we have f x +1 ) f x ) + 1 max θ {1,...,p} ε,. Note that, by strong convexty of f ), the sequence { f x )} s bounded beow. Hence, by Lemma and 9), t foows that the sequence { f x )} converges. Therefore { f x )

9 NEW INEXACT PARALLEL VARIABLE DISTRIBUTION ALGORITHMS 173 f x +1 )} 0. Snce, by 9), m max {1,...,p} ε, = 0, we concude from 13) that { f x ) } 0. Snce x, the souton of 1), s the unque statonary pont, t foows that {x } converges to x. If 10) hods, then from 1) we obtan f x ) f x +1 ) 1 L f x ) β θ f x ) 1 L f x ) β θ f x ) = θ Lβ f x ), 14) Lθ where the second nequaty foows from monotoncty of the -norm. Note that by 10), θ Lβ Lθ > 0. The rest of the proof s standard. By Lemma 1, t foows that L x x fx ) f x) f x), x x = fx ) f x) 15) By the Cauchy-Schwartz nequaty and strong convexty of f ), t foows that Hence f x ) x x = fx ) f x) x x fx ) f x), x x θ x x. f x ) θ x x. Combnng the ast nequaty wth 14), we obtan f x ) f x +1 ) θθ Lβ ) x x. L Ths together wth 15) yeds f x ) f x +1 ) θθ Lβ ) fx ) f x)). L Rearrangng terms gves f x +1 ) f x) 1 θθ ) Lβ ) fx ) f x)). L

10 174 SOLODOV Hence the sequence { f x )} converges Q-neary. Successve appcaton of the ast nequaty yeds f x ) f x) By strong convexty of f ),wehave 1 θθ ) Lβ ) fx 0 ) f x)). L θ x x fx ) f x) f x), x x = fx ) f x). Hence the sequence {x } converges R-neary. In partcuar, we have and x x x x )1 θ fx ) f x)), )1 θ fx0 ) f x)) 1 θθ Lβ ) L ). Ths competes the proof. For the convex case, we have the foowng resut. Theorem. Suppose f ) s convex and f ) CL 1Rn ). Let L f, x 0 ) := {x fx) fx 0 )}. Suppose L f, x 0 ) x 0 + rb, r >0.If or ε, β f x ), 0 β< 1 Lr, max ε, <, {1,...,p} =0 then every accumuaton pont of any sequence {x } generated by Agorthm s a souton of 1). Proof: Frst note that under our assumptons L f, x 0 ) s bounded and hence X s s nonempty. Furthermore, for a dx, X s ) r.

11 NEW INEXACT PARALLEL VARIABLE DISTRIBUTION ALGORITHMS 175 Appyng Lemma 5, smary to the proof of Theorem 1, we obtan f x ) f x +1 ) 1 L f x ) rε,. The rest of the proof can be patterned after that of Theorem PVD wth a suffcent descent condton In ths secton, we present a practca verson of the PVD agorthm for the genera nonconvex) case. In partcuar, we show that there s no need to fnd an exact goba souton for the subprobems. Any pont that satsfes a natura suffcent descent condton can be accepted for the next teraton. We note, n the passng, that the proof gven n [4] makes use of exact goba soutons n an essenta way and breaks down f, for exampe, ony statonary ponts n the subprobems are avaabe. We further pont out that a certan degree of asynchronzaton among the p parae processors s possbe by aowng each of the p processors to take as many steps as desred by ndvduay updatng ts base pont. Synchronzaton can be performed at any tme provded every processor has acheved the suffcent descent condton. Furthermore, we show that synchronzaton step need not be monotone and can be combned wth nonmonotone stabzaton schemes smar to [6]. We aso derve some new convergence resuts for weaky sharp probems of order see Defnton beow). Ths cass of probems can be vewed as a generazaton of strongy convex probems and a certan unconstraned smooth anaogue of weak sharp mnma []. We begn by mposng a natura suffcent descent condton on an agorthm Agorthm A beow) used to sove the subprobems ) generated by the PVD Agorthm 1. Agorthm A. Gven any functon ϕ ) CL 1Rm )and any startng pont t 0 R m generate a pont t R m such that ϕt 0 ) ϕt ) + γ ϕt 0 ), 16) where γ>0depends on L and does not depend on t 0. Note that the above condton s satsfed by a snge teraton of any reasonabe descent agorthm [16], [10] apped to the probem of mnmzng ϕ ) wth t 0 as a startng pont. Hence t s aso satsfed for a mnmum or a statonary pont computed by some descent agorthm provded t uses t 0 as a startng pont. We now state our new PVD agorthm. Agorthm 3. Start wth any x 0 R n. Havng x, stop f f x ) = 0. Otherwse, compute x +1 as foows: ) Paraezaton: for each processor {1,...,p} generate y,µī ) by appyng Agorthm A one or more tmes to the probem mn ψ x x,µ ):= f x,x ī + D ī ) µ 17),µ usng x, 0) as the frst startng pont.

12 176 SOLODOV ) Synchronzaton: Compute x +1 such that f x +1 ) where λ 0, 1). max {1,...,p} ψ y,µ ī ) +λγ f x ), 18) Note that once the suffcent descent condton 16) wth respect to f x ) = ψ x, 0) s satsfed, each processor can ndependenty update ts base pont, generate new drectons D ī and proceed to fnd a pont wth better objectve functon vaue. After these parae steps are performed by each processor then an eventua synchronzaton step s taken. Note that our synchronzaton step may ncrease rather than decrease the objectve functon when compared to the vaues obtaned by the parae processors. Ths provdes the agorthm wth more fexbty and s known to be sometmes usefu n nonnear nonconvex optmzaton [5, 6]. Of course, ony computatona experments can gve an nsght nto the usefuness of nonmonotone synchronzaton schemes for PVD agorthms. We next ntroduce a noton of weak sharp mnma of order whch aows us to strengthen some of the tradtona convergence resuts. Defnton. We say that a set of oca) mnma X s s weaky sharp of order f there exst postve constants ρ and ɛ such that f x) f [x] + ) ρdx, X s ) x X s + ɛ B, 19) where [ ] + denotes the orthogona projecton map onto X s. The cass of probems wth weak sharp mnma of order can be thought of as a certan unconstraned smooth anaogue of weak sharp mnma of order 1) [16, ]. Note that t subsumes strongy convex programs. Let f ) be strongy convex wth moduus. Then ts unque optma pont x s gobay wth ɛ = ) weaky sharp of order. Ths can be easy verfed as foows. By strong convexty, for any x R n fx) f x) f x), x x + x x =ρ x x =ρdx,x s ). Hence the growth property of f ) near the souton set) n the above Defnton s a generazaton of strong convexty. It s cear that there exst functons wth weak sharp mnma of order whch are not strongy convex or even convex) n any neghborhood of ther souton sets. One exampe s f x) := x 1 + x 1), x R.

13 NEW INEXACT PARALLEL VARIABLE DISTRIBUTION ALGORITHMS 177 The statonary set of ths functon s X s = { x x 1 + x = 1} {0,0)}:= X 1 s X s wth Xs 1 beng the set of mnma. It s easy to see that X s 1 s a set of weak sharp mnma of order wth ρ = 1 and ɛ = 1/). Indeed, for any x Xs B d x, Xs 1 ) ) = 1 x 1 + x = x1 + x 1) = f x) f x) x X 1 s. Obvousy, even ocay n any neghborhood of Xs 1 ) f ) n ths exampe s nether strongy convex nor convex. However, we are abe to strengthen standard convergence resuts for probems of ths cass see Theorem 3 beow). As an asde, we note that Xs ={0,0)} s a set of weak sharp maxma n the sense of the same defnton wth the sgn of the eft-hand-sde of 19) reversed). A remark n the end of ths secton contans further exampes of probems wth weak sharp mnma of order. Theorem 3. Let f ) C 1 L Rn ). Suppose {x } s any sequence generated by Agorthm 3. Then ether f ) s unbounded from beow on R n or the sequence { f x )} converges, the sequence { f x )} converges to zero and for every accumuaton pont x of the sequence {x } t foows that f x) = 0. Suppose the sequence {x } s bounded ths hods, for exampe, f the eve set L f, x 0 ) := {x f x) f x 0 )} s bounded). Let the subset X s of statonary ponts of f ) that contans accumuaton ponts of {x } be a set of weak sharp mnma of order, and et 19) hod wth ρ>l/. Then the sequence { f x )} converges Q-neary, and the sequences { f x )} and {dx, X s )} converge to zero R-neary. Proof: By Lemma 3, for any teraton = 0, 1,... and any processor = 1,...,p, ψ, ) C1 L Rn+p 1 ) wth the same L). By 16) and 8), t foows that ψ x, 0 ) ψ y,µ ) ī γ ψ x,0 ) = γ f x ). Snce the ast nequaty hods for a = 1,...,p,wehave fx ) max {1,...,p} ψ y,µ ī ) γ f x ). Hence, by the synchronzaton step 18), f x ) f x +1 ) λγ f x ) ) γ f x )

14 178 SOLODOV and f x ) f x +1 ) 1 λ)γ f x ). 0) We mmedatey concude that { f x )} s a monotoncay nonncreasng sequence. If ths sequence s bounded from beow then t converges. In the atter case, { f x ) f x +1 )} 0 and consequenty { f x )} 0. Hence, by contnuty of f ), f there exst accumuaton ponts of {x }, a of them are statonary ponts of f ). Suppose now the sequence {x } s bounded. The precedng dscusson mmedatey mpes that the set of statonary ponts of f ) s nonempty. Denote by X s ts subset that contans accumuaton ponts of {x }. Ceary, {dx, X s )} 0. Hence x X s + ɛ B for suffcenty arge, say 0. Suppose X s s weaky sharp of order. Then 19) s satsfed for a 0. By Lemma 1, f x ) f [x ] + ) fx ), x [x ] + + L x [x ] + = fx ), x [x ] + + L dx,x s ), where [ ] + denotes the orthogona projecton onto X s. Hence for a 0, by 19), we obtan f x ), x [x ] + fx ) f[x ] + ) L dx,x s ) fx ) f[x ] + ) L fx ) f[x ] + )) = 1 L ) f x ) f [x ] + )), 1) By the Cauchy-Schwartz nequaty and 19), we further obtan f x ) dx, X s ) fx ), x [x ] + 1 L ) f x ) f [x ] + )) ρ 1 L ) dx, X s ). Hence f x ) ρ 1 L ) dx,x s ). )

15 NEW INEXACT PARALLEL VARIABLE DISTRIBUTION ALGORITHMS 179 By ), the Cauchy-Schwartz nequaty and 1) we have f x ) fx ) ρ ρ 1 L ρ 1 L Combnng 0) and 3) gves 1 L ) dx,x s ) ) f x ), x [x ] + ) f x ) f [x ] + )) 3) f x ) f x +1 ) γ1 λ) f x ) γρ1 λ) 1 L ) f x ) f [x ] + )). Rearrangng terms, we obtan f x +1 ) f [x ] + ) 1 γρ1 λ) 1 L ) ) f x ) f [x ] + )). We aready estabshed that the sequence { f x )} converges. Let f := m f x ). Snce a accumuaton ponts of the sequence {x } beong to the set X s and {x } s bounded, t foows that accumuaton ponts of the sequences {x } and {[x ] + } are the same. Therefore, by contnuty of f ), we obtan m f [x ] + ) = m f x ) = f. Because X s s a set of oca) mnma and [x ] + X s, t must be the case that f [x ] + ) = f for a suffcenty arge, say 1. Therefore, for max{ 0, 1 }, we obtan f x +1 ) f 1 γρ1 λ) 1 L ) ) f x ) f ). Hence the sequence { f x )} converges Q-neary. By 0), the sequence { f x )} converges R-neary to zero. Aso, by 19), the sequence {dx, X s )} converges R-neary to zero. Remark. At ths tme, t s an open queston whether the sequence {x } tsef converges neary under the assumptons of Theorem 3. Note that f we had a sera gradent descent method where x +1 x = η fx )

16 180 SOLODOV wth the sequence of stepszes {η } unformy bounded away from zero, then the near convergence rate of {x +1 x } and hence aso of {x }) woud mmedatey foow from the near convergence of { f x )}. The dffcuty wth the parae agorthm s that we cannot expcty reate { f x )} to {x +1 x }. Carefu re-examnaton of the proof of Theorem 3 shows that at the + 1)-st teraton every parae processor decreases the objectve functon f ) of the orgna probem by a factor of f x ) ths at east s true under our assumptons on the drectons dt, t = 1,...,p). Hence f the processors were to proceed wth updatng ther base ponts competey ndependenty wthout usng any nformaton from the other processors, we coud st guarantee the same convergence resuts for each of the p sequences of terates generated. Of course, ths approach essentay yeds p sera processes and therefore s a theoretca extreme. Ths observaton s however of sgnfcance because t mpes that we are aowed a ot of fexbty n devsng PVD agorthms and, n partcuar, n defnng the ponts of synchronzaton. Remark. A practcay mportant exampe of weak sharp mnma of order s provded by the mpct Lagrangan reformuaton [1] of the nonnear compementarty probem. Consder the foowng nonnear compementarty probem [3, 15] NCP) of fndng an x R n such that Fx) 0, x 0, x, Fx) =0, where F : R n R n s a contnuousy dfferentabe mappng. In [1] t was estabshed that the NCP can be soved va smooth) unconstraned mnmzaton of the foowng mpct Lagrangan functon: Mx,α) := α x,fx) + [x αfx)] + x + [Fx) αx] + Fx), where α>1 and [ ] + denotes the orthogona projecton onto the nonnegatve orthant R n +. In partcuar, the mpct Lagrangan s nonnegatve everywhere n R n and assumes the vaue of zero precsey at the soutons of the NCP. In [8] t was estabshed that α 1) rx) Mx,α) αα 1) rx), x R n, where rx) := x [x Fx)] +. Therefore the set of soutons X s of the NCP s a set of weak sharp mnma of order for the mpct Lagrangan whenever the projecton-type error bound hods: dx, X s ) ρ rx) x wth rx) ɛ,

17 NEW INEXACT PARALLEL VARIABLE DISTRIBUTION ALGORITHMS 181 where ρ and ɛ are postve constants ndependent of x). Ths error bound s known to hod when F ) s affne see [9, 17]) or F ) has certan strong monotoncty structure see [1], Theorem ). Moreover, under addtona assumptons on F ), ths condton hods gobay wth ɛ = see [7, 8, 11, 14]). Therefore our anayss shows that certan unconstraned mnmzaton technques apped to mnmzng the mpct Lagrangan attan near rate of convergence under certan condtons). Ths s an nterestng resut gven that the mpct Lagrangan s not known to be strongy convex n any neghborhood of ts zero mnma. 4. Concudng remarks New parae varabe dstrbuton agorthms wth nexact subprobem souton and wth a certan natura suffcent descent condton mposed on the parae subprobems were proposed and anayzed. The modfed agorthms present a fexbe framework and make t easer to acheve good oad baancng among the parae processors. New and mproved near convergence resuts were derved for strongy convex probems and probems wth weak sharp mnma of order. A study of partay asynchronous dstrbuted agorthms [1] that make use of PVD approach can be an nterestng subject of future research. References 1. D.P. Bertsekas and J.N. Tstsks, Parae and Dstrbuted Computaton, Prentce-Ha, Inc.: Engewood Cffs, New Jersey, J.V. Burke and M.C. Ferrs, Weak sharp mnma n mathematca programmng, SIAM Journa on Contro and Optmzaton, vo. 31, no. 5, pp , R.W. Cotte, F. Ganness, and J.-L. Lons eds.), Varatona Inequates and Compementarty Probems: Theory and Appcatons, Wey: New York, M.C. Ferrs and O.L. Mangasaran, Parae varabe dstrbuton, SIAM Journa on Optmzaton, vo. 4, no. 4, pp , L. Grppo, F. Lampareo, and S. Lucd, A nonmonotone ne search technque for Newton s method, SIAM Journa of Numerca Anayss, vo. 3, pp , L. Grppo, F. Lampareo, and S. Lucd, A cass of nonmonotone stabzaton methods n unconstraned optmzaton, Numersche Mathematk, vo. 59, pp , X.-D. Luo and P. Tseng, On goba projecton-type error bound for the near compementarty probem, Lnear Agebra and Its Appcatons to appear). 8. Z.-Q. Luo, O.L. Mangasaran, J. Ren, and M.V. Soodov, New error bounds for the near compementarty probem, Mathematcs of Operatons Research, vo. 19, pp , Z.-Q. Luo and P. Tseng, Error bound and convergence anayss of matrx spttng agorthms for the affne varatona nequaty probem, SIAM Journa on Optmzaton, vo., pp , O.L. Mangasaran, Parae gradent dstrbuton n unconstraned optmzaton, SIAM Journa on Contro and Optmzaton, vo. 33, no. 6, pp , O.L. Mangasaran and J. Ren, New mproved error bounds for the near compementarty probem, Mathematca Programmng, vo. 66, pp , O.L. Mangasaran and M.V. Soodov, Nonnear compementarty as unconstraned and constraned mnmzaton, Mathematca Programmng, vo. 6, pp , J.M. Ortega and W.C. Rhenbodt, Iteratve Souton of Nonnear Equatons n Severa Varabes, Academc Press, 1970.

18 18 SOLODOV 14. J.-S. Pang, A posteror error bounds for the neary-constraned varatona nequaty probem, Mathematcs of Operatons Research, vo. 1, pp , J.-S. Pang, Compementarty probems, n R. Horst and P. Pardaos eds.), Handbook of Goba Optmzaton, Kuwer Academc Pubshers: Boston, Massachusetts, B.T. Poyak, Introducton to Optmzaton, Optmzaton Software, Inc.: Pubcatons Dvson, New York, S.M. Robnson, Some contnuty propertes of poyhedra mutfunctons, Mathematca Programmng Study, vo. 14, pp , M.V. Soodov and S.K. Zavrev, Error-stabty propertes of generazed gradent-type agorthms, Mathematca Programmng Technca Report 94-05, Computer Scence Department, Unversty of Wsconsn, 110 West Dayton Street, Madson, Wsconsn 53706, U.S.A., June 1994 revsed Juy 1995). 19. M.V. Soodov, On the convergence of constraned parae varabe dstrbuton agorthms, Technca Report B-094, Insttuto de Matematca Pura e Apcada, Estrada Dona Castorna 110, Jardm Botanco, Ro de Janero, RJ, CEP 460, Braz, Oct SIAM Journa on Optmzaton, accepted for pubcaton. 0. P. Tseng, Dua coordnate ascent methods for non-strcty convex mnmzaton, Mathematca Programmng, vo. 59, pp , P. Tseng, On near convergence of teratve methods for the varatona nequaty probem, Journa of Computatona and Apped Mathematcs, vo. 60, pp. 37 5, 1995.