arxiv: v1 [math.oc] 6 Jan 2016
|
|
- Marcus Carroll
- 5 years ago
- Views:
Transcription
1 arxv: v1 [math.oc] 6 Jan 2016 THE SUPPORTING HALFSPACE - QUADRATIC PROGRAMMING STRATEGY FOR THE DUAL OF THE BEST APPROXIMATION PROBLEM C.H. JEFFREY PANG Abstract. We consder the best approxmaton problem (BAP) of proectng a pont onto the ntersecton of a number of convex sets. It s known that Dykstra s algorthm s alternatng mnmzaton on the dual problem. We extend Dykstra s algorthm so that t can be enhanced by the SHQP strategy of usng quadratc programmng to proect onto the ntersecton of supportng halfspaces generated by earler proecton operatons. By lookng at a structured alternatng mnmzaton problem, we show the convergence rate of Dykstra s algorthm when reasonable condtons are mposed to guarantee a dual mnmzer. We also establsh convergence of usng a warmstart terate for Dykstra s algorthm, show how all the results for the Dykstra s algorthm can be carred over to the smultaneous Dykstra s algorthm, and dscuss a dfferent way of ncorporatng the SHQP strategy. Lastly, we show that the dual of the best approxmaton problem can have an O(1/k 2 ) accelerated algorthm that also ncorporates the SHQP strategy. Contents 1. Introducton 1 2. Prelmnares: Dykstra s algorthm 4 3. Extended Dykstra s algorthm 6 4. Convergence rate of alternatng mnmzaton and Dykstra s algorthm 9 5. Smultaneous Dykstra s algorthm Usng the APG for (D ) Concluson 20 Appendx A. Proof of convergence of Algorthm References Introducton We consder the followng problem, known as the best approxmaton problem (BAP). Date: September 5, Mathematcs Subect Classfcaton. 41A50, 90C25, 68Q25, 47J25. Key words and phrases. alternatng mnmzaton, Dykstra s algorthm, best approxmaton problem. 1
2 SHQP STRATEGY FOR THE BAP 2 (BAP ) mn f(x) := 1 2 x d 2 (1.1) s.t. x C := C 1 C m, where d s a gven pont and C, = 1,..., m, are closed convex sets n a Hlbert space X. The BAP s equvalent to proectng d onto C. We shall assume throughout that C. We now recall some background on frst order methods, alternatng mnmzaton, and algorthms for the best approxmaton problem Frst order methods and alternatng mnmzaton. When presented wth a problem wth a large number of varables, frst order methods (whch use gradent descent and avod computatonally expensve operatons lke solvng lnear systems) and other methods that decompose the large problems nto smaller peces to be solved may be the only practcal alternatve. For these algorthms, the nonasymptotc or absolute rate of convergence of the functon values to the optmal obectve value hold rght from the very frst teraton of the algorthm, and are more useful than the asymptotc rates. These rates are typcally sublnear, lke O(1/k) for example. Classcal references on frst order methods nclude [NY83], and newer references nclude [Nes04, JN11a, JN11b]. As explaned n [NY83, Nes04], the nonasymptotc rates of convergence of frst order algorthms for smooth convex functons s at best O(1/k 2 ). Nesterov proposed varous O(1/k 2 ) nonasymptotc methods (whch are thus optmal) for such problems (frst method [Nes83], second method [Nes88] and thrd method [Nes05]), and other optmal methods were studed n [AT06, BT09, LLM11]. The paper [BT09] also descrbed an O(1/k 2 ) algorthm for solvng the sum of a smooth convex functon and a structured nonsmooth functon. These optmal methods are also known as accelerated proxmal gradent (APG) methods, and ther desgn and analyss are unfed n the paper [Tse08] (who also dealt wth convex-concave mnmzaton). An optmzaton problem wth a large number of varables can have ts varables dvded nto a number of blocks so that each subproblem has fewer varables. These subproblems are solved n some order (often n a cyclc manner) whle the varables n other blocks are kept fxed. See the formula (2.3) for an elaboraton. Ths s referred to as alternatng mnmzaton (AM), and sometmes referred to Cyclc Coordnate Mnmzaton (CCM). Another alternatve s to perform only gradent descent on each block, whch would reduce to what s descrbed as Cyclc Coordnate Descent, CCD. Methods lke AM and CCD are qute old. If a functon to be mnmzed were nonsmooth, then t s possble for the AM and CCD to be stuck at a non-optmal soluton. A O(1/k) rate of convergence for AM of a two block problem was establshed n [Bec15] wthout any assumpton of strong convexty. As mentoned n [Bec15], AM s also known n the lterature as block-nonlnear Gauss Sedel method or the block coordnate descent method (see for example [Ber99]). They also state the other contrbutons of [Aus76, BT13, Ber99, GS99, LT93]. There has been much recent research on stochastc/ randomzed CD. Snce we are not dealng wth stochastc CD n ths paper, we shall only menton the papers [Nes12, FR15], and defer to the ntroducton and tables n [FR15] for a summary
3 SHQP STRATEGY FOR THE BAP 3 of stochastc CD. A recent work [CP15] also dentfes some cases where the determnstc CD scheme can have an O(1/k 2 ) acceleraton The best approxmaton problem and the method of alternatng proectons. The BAP s often assocated wth the set ntersecton problem (SIP) (SIP ) Fnd x C := C 1 C m. A well studed method for the SIP s the method of alternatng proectons (MAP). We recall materal from [BC11, Deu01a, Deu01b, ER11] on materal on the MAP. As ts name suggests, the MAP proects the terates n a cyclc or some other manner so that the terates converge to a pont n the ntersecton of these sets. One acceleraton of the MAP for convex problems s the supportng halfspace and quadratc programmng strategy (SHQP): The proecton process generates supportng halfspaces of each C, and the set C s a subset of the polyhedron obtaned by ntersectng these halfspaces. Proectng onto the polyhedron can accelerate the convergence of the MAP, and may lead to superlnear convergence n small problems. The SHQP strategy was dscussed n [Pan15c]. See Fgure 4.1 for an llustraton. Ths dea was dscussed n less generalty n [BCK06] and other papers. Other methods of acceleratng the MAP nclude [GPR67, GK89, BDHP03]. As remarked by several authors, the MAP does not converge to the soluton of the BAP. Dykstra s algorthm [Dyk83] solves the best approxmaton problem through a sequence of proectons onto each of the sets n a manner smlar to the MAP, but correcton vectors are added before every proecton. The proof of convergence to P C (d) was establshed n [BD85] and sometmes referred to as the Boyle-Dykstra theorem. Dykstra s algorthm was redscovered by [Han88], who showed that Dykstra s algorthm s equvalent to AM on the dual problem. See also [GM89]. When the sets C are halfspaces, the convergence s asymptotcally lnear [DH94]. A nonasymptotc O(1/k) convergence rate of Dykstra s algorthm was obtaned n [CP15] usng the methods smlar to [BT13, Bec15] when a dual mnmzer exsts. (Ths does not dmnsh the sgnfcance of the Boyle-Dykstra theorem. In our opnon, a quck glance at the respectve proofs shows that the Boyle-Dykstra theorem, whch proves the convergence to the prmal optmal P C (d) even when a dual mnmzer does not exst, s techncally more sophstcated than the proof of the O(1/k) convergence rate of Dykstra s algorthm when a dual mnmzer exsts.) Dykstra s algorthm s qute old, so we refer the reader to the commentary n [Deu01b, ER11] for more on prevous work on Dykstra s algorthm. In contrast to the stuaton for the MAP, not much has been done on acceleratng Dykstra s algorthm for the BAP. The queston of how to accelerate Dykstra s algorthm has been explctly posed as an open problem n [Deu01b, Deu01a, ER11]. A method was proposed n [LR15]. The only property of Dykstra s algorthm needed for ther acceleraton s that Dykstra s algorthm generates a sequence convergng to P C (d), so there s stll be room for mprovng Dykstra s algorthm. See also [HS15]. A varant of Dykstra s algorthm that s more sutable for parallel computatons s the smultaneous Dykstra s algorthm proposed n [IP91] usng the product space formulaton of [Pe84]. Some specfc best approxmaton problems can be solved wth specalzed methods. The proecton of a pont nto the ntersecton of halfspaces can be solved by classcal methods of quadratc programmng. Other sets for whch the proecton
4 SHQP STRATEGY FOR THE BAP 4 onto the ntersecton s easy nclude the ntersecton of an affne space and the semdefnte cone [QS06, Mal04]. The subgradent algorthm can be used to solve a convex constraned optmzaton problem wth a convergence rate of O(1/ k). Hence the BAP can be solved at a rate of O(1/ k). See [Nes04]. In [Pan15b], we obtaned a convergence rate of O(1/k) n the case when the obectve functon s a strongly convex quadratc functon by adaptng a Haugazeau s algorthm [Hau68] (see also [BC11]), whch s another known method for solvng the BAP. We note however that the rate of O(1/k) n Haugazeau s algorthm s typcal, even when solvng a BAP nvolvng only two halfspaces Contrbutons of ths paper. The man contrbuton of ths paper s to extend Dykstra s algorthm so that the SHQP strategy can be ncorporated nto Dykstra s algorthm. (We try to reserve the use of the word acceleraton to mean an O(1/k 2 ) algorthm.) See Algorthm 3.1 for our extenson of Dykstra s algorthm. Recall that the Boyle-Dykstra theorem proves the convergence of Dykstra s algorthm to the prmal soluton of the BAP. We prove that the extended Dykstra s algorthm also converges to the prmal soluton of the BAP (even when there s no dual mnmzer). Next, we show that a commonly occurrng regularty assumpton guarantees the exstence of a dual mnmzer. The exstence of such a dual mnmzer would, by the results n [CP15], mply that Dykstra s algorthm converges at a O(1/k) rate. Ths analyss also carres over to our extended Dykstra s algorthm. We pont out that t s useful to use warmstart solutons for Dykstra s algorthm and our extenson. Whle t s recognzed that Dykstra s algorthm s the alternatng mnmzaton algorthm on the dual, t appears that every descrpton and proof of convergence of Dykstra s algorthm n the lterature starts wth the default zero vector. See further dscussons n Subsecton 2.1. We answer the natural queston of whether Dykstra s algorthm and our extenson converge to the optmal prmal soluton wth a warmstart terate by adaptng the proof of the Boyle-Dykstra Theorem [BD85]. See Appendx A. We show how all these deas mentoned earler can be mplemented for the smultaneous Dykstra s Algorthm n Secton 5. We also explan another way to ncorporate the SHQP strategy on the BAP n Subsecton 4.3 that works when a mnmzer to the dual problem exsts and s more natural to augment to the APG. Whle ths strategy s more natural than our extended Dyktra s algorthm, we were not able to prove ts global convergence usng the framework of the Boyle-Dykstra theorem Notaton. Our notaton s farly standard. For a closed convex set D, we let P D ( ) denote the proecton onto D. The normal cone of D at a pont x D n the usual sense of convex analyss s denoted by N D (x). We wll let ỹ = (y 1,..., y m ). When we dscuss the extended Dykstra s algorthm n Secton 3, we wll need ỹ = (y 1,..., y m, y m+1 ), but ths shouldn t cause too much confuson. 2. Prelmnares: Dykstra s algorthm In ths secton, we recall Dykstra s algorthm and some results. We also gve a dscusson of warmstartng Dykstra s algorthm.
5 SHQP STRATEGY FOR THE BAP 5 Algorthm 2.1. (Warmstart Dykstra s algorthm) Let X be a Hlbert space. Consder the problem of proectng a pont d X onto C X, where C = m =1 C and C are closed convex sets. Choose startng y (0) X for all {1,..., m}, and let x (0) m = d (y (0) y m (0) ). 01 For k = 1, 2, x (k) 0 = x m (k 1) 03 For = 1, 2,..., m 04 z (k) := x (k) 1 + y(k 1) 05 x (k) := P C (z (k) ) 06 y (k) := z (k) x (k) 07 End for 08 End for Let the vector ỹ X m be (y 1,..., y m ), where each y X. For each closed convex set D X, let δ (, D) : X R be defned by δ (y, D) = max x D y, x. (The functon δ (, D) s also the conugate of the ndcator functon δ(, D), thus explanng our notaton.) Defne the dual problem (D ) by (D ) nf y1,...,y m h(y 1,..., y m ) := f(y y m ) + where y X and the f : X R s as n (1.1). We revew some easy results on (D ). m δ (y, C ), (2.1) Proposton 2.2. Let X be a Hlbert space. Let C be closed convex sets n X for {1,..., m}, and let C = m =1 C. Let d X and x = P C (d). Let ỹ = (y 1,..., y m ). We have the followng: (1) nf y1,...,y m h(y 1,..., y m ) = 1 2 d d x 2. (2) Let v : X m R be defned by v(y 1,..., y m ) = 1 2 d (y y m ) x 2 + m δ (y, C x). (2.2) Then v(ỹ) = h(ỹ) d, x x 2, and nfỹ v(ỹ) = 0. (3) We have v(y 1,..., y m ) 1 2 d (y y m ) x 2. (4) If (y 1,..., y m ) s a mnmzer of v( ) (or equvalently, h( )), then x = d (y y m ). (5) If m = 1, then y 1 = d x s a mnmzer of v( ) (or equvalently, h( )). Proof. Statement (1) can be obtaned from [GM89, pages 32 33]. For Statement (2), note that =1 v(ỹ) = 1 2 d (y y m ) x 2 + m δ (y, C x) = 1 2 d [ (y y m ) 2 d (y y m ), x x 2 m ] + δ (y, C ) y y m, x =1 [ m ] = 1 2 d (y y m ) 2 d, x x 2 + δ (y, C ) =1 = h(ỹ) d, x x 2. =1 =1
6 SHQP STRATEGY FOR THE BAP 6 The rest of Statement (2) s elementary. Statements (3) and (4) follow easly from the fact 0 C x, whch gves δ (y, C x) y, 0 = 0. Statement (5) s easy. As explaned n [Han88, GM89] and perhaps other sources, alternatng mnmzaton n the order y (k) 1 = arg mn y (k) y 2 = arg mn y (k). y m = arg mn y h(y, y (k 1) 2, y (k 1) 3,..., y (k 1) m ) (2.3) h(y (k) 1, y, y(k 1) 3,..., y m (k 1) ) h(y (k) 1, y(k) 2,..., y(k) m 1, y), leads to the Dykstra s algorthm as presented n Algorthm 2.1 through Proposton 2.2(5). We also have the followng easly verfable facts: x (k) = d y (k) 1 y (k) 1 y(k) y (k 1) +1 y m (k 1) (2.4) and z (k) = d y (k) 1 y (k) 1 y(k 1) +1 y m (k 1) (2.5) 2.1. Warmstart Dykstra s algorthm. It appears that all descrptons and proofs of convergence of Dykstra s algorthm use the default startng pont y (0) = 0 for all {1,..., m}. We saw earler that Dykstra s algorthm s alternatng mnmzaton on the dual problem wth startng pont ỹ (0). In partcular, the terates ỹ (k) are such that {h(ỹ (k) )} k s a non-ncreasng sequence of real numbers to the dual obectve value. One may then choose a startng pont ỹ (0) such that h(ỹ (0) ) s closer to the dual obectve value than the default startng pont of all zeros. There are several ways to obtan a dfferent startng pont. (1) One can use greedy algorthms (that may not guarantee global convergence to the optmal soluton) to decrease the dual obectve values. A plausble strategy s to use the greedy algorthms tll they do not appear to acheve good decrease n the value h( ), then swtch to the warmstart Dykstra s algorthm, or our extended algorthm n Algorthm 3.1, to guarantee convergence to the optmal prmal soluton. (2) A warmstart soluton may be avalable after solvng a nearby problem. For example, one mght want to resolve a problem after a set has been added or removed, or after a perturbaton of parameters. Alternatvely, there may be a nearby structured problem that can be solved approxmately wth less effort than the orgnal problem. The proof of convergence of Dykstra s algorthm wth a dfferent startng pont s not too dfferent from the Boyle-Dykstra theorem. We defer the proof to Appendx A, where we also prove the convergence of our extended Dykstra s algorthm to be ntroduced n Secton Extended Dykstra s algorthm As mentoned n Subsecton 1.2, the SHQP strategy (of collectng halfspaces contanng C generated by earler proectons and then proectng onto the ntersecton
7 SHQP STRATEGY FOR THE BAP 7 of the halfspaces by QP) can enhance the convergence of the method of alternatng proectons for the set ntersecton problem. In ths secton, we present our extenson of Dykstra s algorthm n Algorthm 3.1 and how t can ncorporate the SHQP strategy. In order to extend the proof of the Boyle-Dykstra theorem to establsh the prmal convergence of our extended Dykstra s algorthm, we need Theorem 3.4(2). The proof of Theorem 3.4(2) llustrates why lnes 8 and 12 of Algorthm 3.1 were desgned as such. The other parts of the Boyle-Dykstra theorem follow wth lttle modfcatons, so we defer the rest of the convergence proof to Appendx A. We now present our extended Dykstra s algorthm. Algorthm 3.1. (Extended Dykstra s algorthm) Consder the BAP (1.1). Let y (0) X be the startng dual varables for each component {1,..., m}. We also ntroduce a varable y (k) m+1 X, wth startng value y(0) m+1 beng 0, n our calculatons. Let Hm+1 0 = X. Set x (0) m+1 = d m+1 01 For k = 1, 2, x (k) 0 = x (k 1) m+1 03 For = 1, 2,..., m 04 z (k) 05 x (k) 06 y (k) := x (k) 1 + y(k 1) := P C (z (k) ) x (k) =1 y(0). := z (k) 07 End for 08 Let Cm+1 k X be such that C Ck m+1 Hk 1 09 z (k) m+1 := x(k) m + y (k 1) m+1 10 x (k) m+1 = P Cm+1 k (z(k) m+1 ) m y (k) m+1 = z(k) m+1 x(k) m+1 12 Let Hm+1 k be the halfspace wth normal y(k) m+1 passng through x(k) m+1,.e., 13 End for H k m+1 = {x : y (k) m+1, x x(k) m+1 0}. Remark 3.2. (Desgnng Cm+1 k ) In lne 8 of Algorthm 3.1, the set Ck m+1 can be chosen to be the ntersecton of Hm+1 k 1 and the halfspaces generated through earler proectons. The proecton P C k m+1 ( ) can then be calculated easly usng methods of quadratc programmng f the number of halfspaces defnng Cm+1 k s small. It s clear to see that Algorthm 3.1 reduces to the orgnal Dykstra s algorthm f we had kept Hm+1 k = Ck m+1 = X for all k {1, 2,... }. The choce of storng halfspaces for n lne 12 smplfes computatons nvolved. H k m+1 Remark 3.3. (Postonng sets of type Cm+1 k ) If the number m s large, then one can ntroduce more than ust one addtonal set of the type Cm+1 k at the end of all the orgnal sets n an mplementaton of Algorthm 3.1. For example, one can ntroduce the addtonal set after every fxed number of orgnal sets so that the quadratc programs formed wll have a manageable number of halfspaces. Theorem 3.4 below wll be crucal n provng that the terates {x (k) } of Algorthm 3.1 converges to the optmal prmal soluton. The proof of the Theorem 3.4 explans how the sets Cm+1 k and Hk m+1 were desgned n order to mantan the concluson n Theorem 3.4(2).
8 SHQP STRATEGY FOR THE BAP 8 Theorem 3.4. (Propertes of Algorthm 3.1) In Algorthm 3.1, defne the dual functon h k : X m+1 R at the kth teraton and h : X m+1 R by [ m ] h k (ỹ) = 1 2 d (y y m+1 ) 2 + δ (y, C ) + δ (y m+1, Hm+1 k ) [ =1 m ] h(ỹ) = 1 2 d (y y m+1 ) 2 + δ (y, C ) + δ (y m+1, C). =1 (3.1) Let ỹ (k) = (y (k) 1,..., y(k) m, y (k) m+1 ). The followng hold: (1) h k 1 (ỹ (k 1) ) h k (ỹ (k) ) + 1 m+1 2 =1 y(k) y (k 1) 2. (2) The sum m+1 =1 =1 y() y ( 1) 2 s fnte. Proof. We have the followng chan of nequaltes: = 1 2 d (y(k) y m (k) ) y (k 1) m δ (y (k 1) m+1, Hk 1 m+1 ) d (y(k) y m (k) ) y (k 1) m δ (y (k 1) m+1, Ck m+1 ) d (y(k) y m (k) ) y (k) m δ (y (k) m+1, Ck m+1) y(k) m+1 y(k 1) m+1 2 d (y(k) y m (k) ) y (k) m δ (y (k) m+1, Hk m+1 ) y(k) m+1 y(k 1) m+1 2. (3.2) The frst nequalty comes from the fact that Cm+1 k Hm+1 k 1, whch mples that δ (, Hm+1 k 1 ) δ (, Cm+1). k The second nequalty comes from the fact that y (k) m+1 s the mnmzer of the strongly convex functon wth modulus 1 defned by y 1 2 d (y(k) y m (k) ) y 2 + δ (y, Cm+1 k ). The fnal equaton follows readly from the defnton of Hm+1 k. We can apply the same prncple n (3.2) to show that for all {1,..., m}, we have d y(k) 1 y (k) 1 y(k 1) d y(k) 1 y (k) 1 y(k) +δ (y (k), C ) y(k) y (k 1) +1 y (k 1) m δ (y (k 1), C ) y (k 1) +1 y (k 1) m y (k 1) (3.3) Combnng (3.2) and (3.3) gves (1). From the fact that Hm+1 k C, we have δ (, Hm+1) k δ (, C), whch n turn mples that h k (y) h(y). Moreover, for each k, we make use of the observaton n Proposton 2.2(1) to get Hence Thus (2) follows. k m y () =1 =1 nf y X hk (y) = mn h(y). m+1 y X m+1 y ( 1) 2 h 0 (ỹ (0) ) h k (ỹ (k) ) h 0 (ỹ (0) ) h(ỹ (k) ) h 0 (ỹ (0) ) mn y h(y). The rest of the proof of the prmal convergence of Algorthm 3.1 s not too dfferent from the Boyle-Dykstra theorem, so we wll prove the convergence result n Appendx A.
9 SHQP STRATEGY FOR THE BAP 9 4. Convergence rate of alternatng mnmzaton and Dykstra s algorthm In ths secton, we frst recall the proof of the O(1/k) convergence rate of alternatng mnmzaton under the assumpton of strong convexty of subproblems and bounded level sets. Ths wll then gve us the convergence rate of the functon h( ) n the dual of Dykstra s algorthm. We also dscuss how ths analyss can be carred over to our extended Dykstra s algorthm. In Subsecton 4.3, we ntroduce another more natural way to ncorporate the SHQP heurstc nto Dykstra s algorthm and attans the nonasymptotc O(1/k) convergence rate when there s a dual mnmzer. But we note that we are unable to prove the global convergence to the prmal optmal soluton for ths new strategy General convergence rate result on alternatng mnmzaton. In ths subsecton, we recall that under certan condtons, alternatng mnmzaton has a nonasymptotc convergence rate of O(1/k). We need the followng result proved n [BT13] and [Bec15]. Lemma 4.1. (Sequence convergence rate) Let α > 0. Suppose the sequence of nonnegatve numbers {a k } k=0 s such that a k a k+1 + αa 2 k+1 for all k {1, 2,... }. (1) [BT13, Lemma 6.2] If furthermore, a α and a α a k 1.5 αk for all k {1, 2,... }. (2) [Bec15, Lemma 3.8] For any k 2, a k max { ( 1 2 In addton, for any ǫ > 0, f { k max ) (k 1)/2 a0, 4 α(k 1) 2 ln(2) [ln(a 0) + ln(1/ǫ)], 4 αǫ }. } + 1,, then then a n ǫ. The second formula refnes the frst by reducng the dependence of a k on the frst few terms of {a }. We now prove our general convergence rate result for alternatng mnmzaton. The followng result was dscussed n [CP15] and ts deas appeared n [BT13, Bec15]. Theorem 4.2. (O(1/k) Convergence rate of alternatng mnmzaton) Let f : X m R be a smooth convex functon, and g : X R be (not necessarly smooth) convex functons for {1,..., m}, Defne h : X m R by h(y 1, y 2,..., y m ) = f(y 1, y 2,..., y m ) + m g (y ). such that (1) The gradent f : X m X m s Lpschtz contnuous wth modulus L, and (2) There s a number µ > 0 such that for all {1,..., m} and fxed varables y 1, y 2,..., y 1, y +1,..., y m, the map =1 y f(y 1, y 2,..., y 1, y, y +1,..., y m ) s strongly convex wth modulus µ > 0.
10 SHQP STRATEGY FOR THE BAP 10 (3) A mnmzer ỹ = (y1, y 2,..., y m ) of h( ) exsts. Moreover, M defned by M = sup{ y (k) y : k 0} s fnte for all {1,..., m 1}. Suppose two successve terates ỹ (k 1) = (y (k 1) 1, y (k 1) 2,..., y m (k 1) ) and ỹ (k) defned smlarly are produced by alternatng mnmzaton descrbed n (2.3). Let M = max {1,...,m 1} M. Then h(ỹ (k 1) ) h(ỹ ) h(ỹ (k) ) h(ỹ ) + Applyng Lemma 4.1 to a k := h(ỹ (k) ) h(ỹ ) gves and µ 2(m 1) 3 M 2 L 2 [h(ỹ (k) ) h(ỹ )] 2. (4.1) h(ỹ (k) ) h(ỹ ) 1 k max{ 3(m 1)3 M 2 L 2 µ, h(ỹ (1) ) h(ỹ ), 2[h(ỹ (2) ) h(ỹ )]}, h(ỹ (k) ) h(ỹ ) max { ( 1 2 ) (k 1)/2 [h(ỹ (0) ) h(ỹ )], 8(m 1)3 M 2 L 2 µ(k 1) Proof. The proof of ths result follows smlar deas as those n [CP15], whch n turn appeared n [BT13, Bec15]. Snce we wll use elements of ths proof for the proof of Theorem 4.5, we now gve a self contaned proof. For each, let h : X R be defned by h (y) = h(y (k) 1, y(k) 2,..., y(k) 1, y, y(k 1) +1,..., y m (k 1) ). In other words, h ( ) s the th block of h : X m R. The mappng h ( ) has mnmzer y (k), and s strongly convex wth modulus µ from assumpton (2). Hence Hence h (y (k 1) ) h (y (k) ) + µ 2 y(k) y (k 1) 2. h(ỹ (k 1) ) h(ỹ ) h(ỹ (k) ) h(ỹ ) + m µ 2 y(k) =1 }. y (k 1) 2. (4.2) Next, we try to fnd a subgradent n h(ỹ (k) ) by lookng at the components h (ỹ). It s clear that 0 h m (y m (k) ). We then look at the th component of f ( ), whch we denote by f ( ). For each {1,..., m}, the optmalty condtons of each teraton of alternatng mnmzaton mples that Thus 0 f (y(k) 1, y(k) 2,..., y(k) f (ỹ (k) ) f (y (k) 1, y(k) 2,..., y(k) 1, y(k) 1, y(k), y (k 1) +1,..., y m (k 1) Choose a subgradent s h(ỹ (k) ), wth s X m such that We have s = f (ỹ(k) ) f (y(k) 1, y(k) 2,..., y(k) s f (ỹ(k) ) f (y(k) 1, y(k) 2,..., y(k) L m y (k) =+1 L m y (k) =2 y (k 1) y (k 1). ) + g (y (k) )., y (k 1) +1,..., y m (k 1) ) f (ỹ (k) ) + g (y (k) ). 1, y(k) 1, y(k), y (k 1) +1,..., y (k 1) m )., y (k 1) +1,..., y m (k 1) ) (4.3)
11 SHQP STRATEGY FOR THE BAP 11 The above dervaton also remnds us that s m = 0. Thus, makng use of condton (3), we have h(ỹ ) h(ỹ (k) ) + s, ỹ ỹ (k) h(ỹ (k) ) h(ỹ ) s, ỹ ỹ (k) m 1 s y y(k) =1 [ m L y (k) y (k 1) =2 [ m (m 1)ML Applyng (4.4) on (4.2) gves h(ỹ (k 1) ) h(ỹ ) h(ỹ (k) ) h(ỹ ) + m h(ỹ (k) ) h(ỹ ) + m y (k) =2 µ 2 y(k) =1 µ 2 y(k) =2 h(ỹ (k) ) h(ỹ ) + µ 2(m 1) ][ m 1 =1 y (k 1) [ m y y(k) ]. y (k 1) 2 y (k 1) 2 y (k) =2 ] y (k 1) h(ỹ (k) ) h(ỹ ) + 2(m 1) 3 M 2 L [h(ỹ (k) ) h(ỹ )] 2. 2 µ ] 2 (4.4) (4.5) Let a k = h(ỹ (k) ) h(ỹ ). Applyng Lemma 4.1 gves us our concluson. (For the µ frst formula, α = mn{ 2(m 1) 3 M 2 L, a 1, 0.75 a 2 }.) It s clear to see that condton (3) n Theorem 4.2 s satsfed when the level sets of h( ) are bounded. Condton (3) can be easly amended to havng all but one of the M for {1,..., m} beng fnte Convergence rate of extended Dykstra s algorthm. In Dykstra s algorthm, the functon f( ) n (1.1) s quadratc, and therefore ts gradent s lnear. Furthermore, each block f ( ) s strongly convex wth modulus 1. Thus condtons (1) and (2) of Theorem 4.2 are satsfed. We make some remarks condton (3) of Theorem 4.2. Remark 4.3. (Condton (3) of Theorem 4.2 for Dykstra s algorthm) As ponted out n [Han88], there may not exst a mnmzer ỹ of the dual problem (D ). Consder for example the problem of proectng onto the ntersecton of two crcles n R 2 ntersectng at only one pont. Furthermore, Gaffke and Mathar [GM89, Lemma 2] showed that for Dykstra s algorthm, f there s a λ > 2 such that x (k) m x 2 O(1/k λ ), then y = lm k y k exsts wth δ (y, C ) fnte, and ỹ = (y1,..., ym) mnmzng the functon h( ) of (2.1). Ths result can somewhat be seen as a converse of Theorem 4.2. Remark 4.4. (Fnteness of the M s) In our analyss of Dykstra s algorthm, suppose all but one of the M s n Theorem 4.2(3) are fnte for {1,..., m}. The Boyle- Dykstra theorem mples that the lmt lm [d y(k) 1 y m (k) ] = lm k k x(k) m exsts. Ths would mply that all the M s are fnte. We now provde the addtonal detals to show that Algorthm 3.1 (the extended Dykstra s algorthm) also converges at an O(1/k) rate.
12 SHQP STRATEGY FOR THE BAP 12 Theorem 4.5. (Convergence rate of extended Dykstra s algorthm) Consder Algorthm 3.1. Recall the defnton of h( ) n (2.1). Suppose the followng holds: (3 ) A mnmzer ỹ = (y1, y 2,..., y m) of h( ) exsts. Moreover, M defned by M = sup{ y (k) y : k 0} s fnte for all {1,..., m + 1}. (Compare ths to condton (3) of Theorem 4.2.) Recall the defnton of h k ( ) n (3.1). Then the sequence {h k (y (k) 1,..., y(k) m+1 )} k, converges to h(ỹ ) at a rate of O(1/k). Proof. We hghlght the dfferences ths proof has wth that of Theorem 4.2. Theorem 3.4(1) shows that h k 1 (ỹ (k 1) ) h k (ỹ (k) ) m+1 y (k) =1 y (k 1) 2, whch plays the role of (4.2). Next, f ỹ = (y 1,..., y m) s a mnmzer of h( ), then (y 1,..., y m, 0) s a mnmzer of hk ( ) for all k. Moreover, h k (y 1,..., y m, 0) = h(ỹ ). Next, we can prove an analogous result to (4.3) wth L = 1. The analogous result to (4.4) s [ m+1 ] h k (ỹ (k) ) h(ỹ ) mml y (k 1). (4.6) The analogous result to (4.5) s =2 y (k) h k 1 (ỹ (k 1) ) h(ỹ ) h k (ỹ (k) ) h(ỹ µ ) + 2m 3 M 2 L [h k (ỹ (k) ) h(ỹ )] 2. 2 (4.7) The concluson follows wth steps smlar to the proof of Theorem 4.2. An ndcator of whether an O(1/k) convergence rate s acheved would be whether condton (3) n Theorem 4.2 s satsfed. The next result gves suffcent condtons. Theorem 4.6. (Condton for bounded dual terates) Suppose X = R n, and consder the BAP (1.1). (1) Suppose at the prmal optmal soluton x = P C (d), we have m v = 0 and v N C (x ) for all {1,..., m} =1 (4.8) mples v = 0 for all {1,..., m}. Then the terates {ỹ (k) } of Dykstra s algorthm are bounded. Moreover, an accumulaton pont exsts, and s an optmal soluton for (D ), so condton (3) of Theorem 4.2 holds. (2) Suppose at the prmal optmal soluton x = P C (d), we have v = 0, v m+1 N C (x ) and v N C (x ) for all {1,..., m} m+1 =1 mples v = 0 for all {1,..., m + 1}. (4.9) Then the terates {ỹ (k) } of the extended Dykstra s algorthm are bounded. Moreover, an accumulaton pont exsts, and s a mnmzer of h : (R n ) m+1 R defned n (3.1), so condton (3 ) of Theorem 4.5 holds. (3) Suppose N C (x ) does not contan a lne for all {1,..., m}. In other words, the cones N C (x ) are ponted for all. Then (4.8) and (4.9) are equvalent.
13 SHQP STRATEGY FOR THE BAP 13 Proof. For (1), we prove the boundedness of the terates for Dykstra s algorthm. The other parts of the result are straghtforward. Seekng a contradcton, suppose the terates {ỹ (k) } are not bounded. Then m y (k) = d x (k) m 1 max y (k) =1 m =1 y (k) = 1 [d max y (k) x(k) m ]. (4.10) By the convergence of Dykstra s algorthm, lm k [d x (k) m ] exsts. Moreover, lm sup k max y (k) =, so by takng a subsequence f necessary (we do not relabel), the lmt of the RHS of (4.10) s zero. Let ŷ (k) y = (k) have m ŷ (k) =1 = 0. max y (k). We thus The sequence {(ŷ (k) 1,..., ŷ(k) m )} k has a convergent subsequence. Let an accumulaton pont be (ŷ1,..., ŷm). Note that ŷ (k) N C (x (k) ), so ŷ N C (x ). But not all the ŷ are zero. Ths gves us the contradcton to (4.8). We now show how to amend the proof of (1) to prove (2). For the extended Dykstra s algorthm, we can obtan the formula max y (k) m+1 1 max y (k) =1 y (k) = 1 max y (k) [d x(k) m+1 ], whch s smlar to (4.10). The sequence {(ŷ (k) 1,..., ŷ(k) m+1 )} k s defned smlarly by ŷ (k) y = (k), and has a convergent subsequence wth accumulaton pont (ŷ1,..., ŷ m ). For any c C, we have As we take lmts, we have ŷ (k) m+1, c x(k) m+1 0. ŷ m+1, c x 0, so ŷm+1 N C (x ). The same steps would mply that (4.9) s volated, hence a contradcton. Lastly, we prove (3). It s obvous that (4.9) mples (4.8) (ust take the partcular case when v m+1 = 0). We now prove that (4.8) mples (4.9). If (4.8) holds, then the formula for ntersecton of normal cones of convex sets (see [RW98, Theorem 6.42]) mples that N C (x ) = m N C (x ). =1 Suppose m+1 =1 v = 0, where v m+1 N C (x ) and v N C (x ) for all {1,..., m}. We can wrte v m+1 = m+1 =1 ṽ, where ṽ N C (x ) for all {1,..., m}. Then m =1 (v + ṽ ) = 0, and (v + ṽ ) N C (x ). Condton (4.8) would mply that v + ṽ = 0 for all {1,..., m}. Snce N C (x ) contans no lnes for all {1,..., m}, we have v = ṽ = 0 for all {1,..., m}. Ths mples that (4.9) holds. Remark 4.7. We make a few remarks on Theorems 4.6 and 4.2.
14 SHQP STRATEGY FOR THE BAP 14 (1) A smple example of a lne and a halfspace shows that (4.8) and (4.9) cannot be equvalent f the condtons n (3) were omtted. Even so, we can check that n ths smple example, the extended Dykstra s algorthm should perform better than the Dykstra s algorthm n general, even when (4.9) fals. See Fgure 4.1. (2) Even f condton (1) n Theorem 4.6 s not satsfed, condton (3) of Theorem 4.2 can hold. For example, consder the case of two (one dmensonal) lnes ntersectng only at the orgn n R 3. (3) The condton (4.8) s well known to be equvalent to the stablty of the sets {C } m =1 under perturbatons. See [Kru06] for example. Condton (4.8) s also mportant for establshng lnear convergence of the method of alternatng proectons for convex sets. See [BB96]. C 1 C 1 C 2 d C 2 d Fgure 4.1. In the dagram on the left, the lne shows the path Dykstra s algorthm takes. But for both the extended Dykstra s algorthm n Algorthm 3.1 (even f (4.9) s not satsfed) and Algorthm 4.8, we have convergence to P C (d) n a small number of steps. The dagram on the rght shows that Algorthms 3.1 and 4.8 are also advantageous for nonpolyhedral problems SHQP strategy for Dykstra s algorthm. We now show that n the case where a mnmzer exsts for h( ) as defned n (2.1), the SHQP strategy can be ncorporated nto Dykstra s algorthm. We present the followng addtonal step. Algorthm 4.8. (SHQP strategy for Dykstra s algorthm) Consder the orgnal warmstart Dykstra s algorthm (Algorthm 2.1). Between lnes 7 and 8, we can add as many copes of the followng code segment as needed. 01 Choose J {1,..., m} 02 Update y (k) 1,..., y(k) m by solvng the followng optmzaton problem (y (k) 1,..., y(k) m ) arg mn y 1,...,y m s.t. f(y y m ) + m δ (y, H ) (4.11) y = y (k) f / J. To llustrate the effectveness of the step n Algorthm 4.8, let us for now assume that J = {1,..., m}. Let y (k), 1,..., y m (k), X be the values of y (k) before lne 2 was performed n Algorthm 4.8, and let y (k),+ 1,..., y m (k),+ be the respectve values after lne 2 was performed. Note that n Dykstra s algorthm, lne 5 (x (k) = P C (z (k) )) s obtaned by proectng onto the set C, and ths proecton produces a supportng halfspace H at x (k) so that H C. Moreover, the halfspace H also satsfes =1 δ (y (k),, C ) = δ (y (k),, H ). (4.12)
15 SHQP STRATEGY FOR THE BAP 15 The ntersecton m =1 H would be a polyhedral outer approxmate of C = m =1 C. Snce H C, we have δ (, H ) δ (, C ). We therefore have y m (k), ) + m δ (y (k),, C ) f(y (k), =1 (4.12) = f(y (k), y m (k), ) + m δ (y (k),, H ) (4.11) f(y (k),+ = y m (k),+ ) + m δ (y (k),+, H ) f(y (k),+ = y m (k),+ ) + m δ (y (k),+, C ). Thus performng the step n lne 2 of Algorthm 4.8 mproves the dual obectve h( ). Note that the mnmzaton problem (4.11) s the dual of the problem of proectng a pont onto the polyhedron J H, whch can be solved effectvely by quadratc programmng f the number of halfspaces s small. If the number of halfspaces s large, then lne 1 of Algorthm 4.8 gves the flexblty of solvng a quadratc program of manageable sze nstead. In general, H can be chosen to be the ntersecton of halfspaces such that (4.12) s vald. If the boundary of C s smooth, then H approxmates C at x (k), and the algorthm reduces to sequental quadratc programmng. Ths gves a reason why the addtonal step n Algorthm 4.8 can be effectve n practce. The step explaned here gves a smlar knd of enhancement to what we saw earler for the extended Dykstra s algorthm. It s clear to see that the recurrence (4.1) s not affected by the addtonal step n Algorthm 4.8. Thus the convergence analyss gven n Subsecton 4.1 remans vald. But when h( ) does not have a mnmzer, we were not able to extend the Boyle-Dykstra Theorem (specfcally, Lemma A.4 below) for the proof of global convergence of the extenson of Dykstra s algorthm usng Algorthm 4.8. =1 5. Smultaneous Dykstra s algorthm Recall that Dykstra s algorthm reduces the best approxmaton problem to a seres of proectons. A varant of Dykstra s algorthm whch s more sutable for parallel computatons s the smultaneous Dykstra s algorthm proposed and studed n [IP91]. In ths secton, we gve some detals on dervng the smultaneous Dykstra s algorthm, and then show how the prncples descrbed n extendng Dykstra s algorthm can be appled for the smultaneous Dykstra s algorthm. Consder the BAP (1.1), where we want to fnd the proecton of d onto C = m =1 C. We now recall the product space formulaton of [Pe84]. Defne C X m and D X m by C := C 1 C m (5.1) and D := {(x,..., x) X m : x X}. Let λ 1,..., λ m be m postve numbers that sum to one, and let the nner product, Q n X m be defned by (u 1,..., u m ), (v 1,... v m ) Q := m λ u, v. =1
16 SHQP STRATEGY FOR THE BAP 16 The proecton of the pont (d,..., d) X m onto C D can easly be seen to be (P C (d),..., P C (d)). Dykstra s algorthm can be appled onto the product space formulaton. Ths gves the smultaneous Dykstra s algorthm proposed and studed n [IP91], whch we present below. Algorthm 5.1. [IP91](Smultaneous Dykstra s algorthm) Consder the BAP (1.1). Let y (0) X be the startng dual varables for each component {1,..., m}. Set x (0) = d m =1 λ y (0). 01 For k = 1, 2, For = 1, 2,..., m (Parallel proecton) 03 z (k) := x (k 1) + y (k 1) 04 x (k) = P C (z (k) ) 05 y (k) = z (k) x (k) 06 end for 07 x (k) = m =1 λ x (k) 08 end for We gve a bref explanaton of the smultaneous Dykstra s algorthm. Lnes 3 to 5 correspond to the proecton onto C. Lne 7 corresponds to proecton of (x (k) 1,..., x(k) m ) onto D,.e., (x (k),..., x (k) ) = P D (x (k) 1,..., x(k) m ). The advantage of the smultaneous Dykstra s algorthm s that lnes 3 to 5 can be performed n parallel. We now dscuss the convergence rate of the smultaneous Dykstra s algorthm. We saw n Secton 4 that the regularty condton (4.8) s a suffcent condton for O(1/k) convergence. We now show that ths regularty condton holds for the orgnal problem f and only f t holds for the product space formulaton. Proposton 5.2. (Equvalence of constrant qualfcaton) Let C be closed convex sets for {1,..., m}, and let C = m =1 C. Let C and D be as defned n (5.1). At a pont x C, the condtons and m v = 0 and v N C (x ) for all {1,..., m} =1 mples v = 0 for all {1,..., m} (5.2) (v 1,..., v m ) + (w 1,..., w m ) = 0, (v 1,..., v m ) N C (x,..., x ) (5.3) and (w 1,..., w m ) N D (x,..., x ) mples (v 1,..., v m ) = (w 1,..., w m ) = 0 are equvalent. Proof. Note that (v 1,..., v m ) N C (x,..., x ) f and only f v N C (x ) for all. Next, snce D s a lnear subspace, we have (w 1,..., w m ) N D (x,..., x ) f and only f (w 1,..., w m ) D. Proposton 5.3 gves the equvalent condton λ w = 0. So n other words, (v 1,..., v m ) + (w 1,..., w m ) = 0, (v 1,..., v m ) N C (x,..., x ) and (w 1,..., w m ) N D (x,..., x )
17 SHQP STRATEGY FOR THE BAP 17 s equvalent to v N C (x ) and w = v for all {1,..., m}, and m λ v = 0. Condtons (5.2) and (5.3) are now easly seen to be equvalent. As s well known n the study of Dykstra s algorthm, no correcton vectors for D are necessary snce D s an affne space. But we need to elaborate on the correcton vector to D before we show the dervaton of x (0). Let ths correcton vector be w (k) = (w (k) 1,..., w(k) m ). We have w (k) N D (x (k),..., x (k) ). But snce D s a lnear subspace, we have w (k) D. We have the followng easy result. Proposton 5.3. Let w = (w 1,..., w m ) be a vector n X m. Then w D f and only f λ w = 0. Proof. Ths follows easly from the followng chan: w D w, v = 0 for all v D λ w, v = 0 for all v X λ w = 0. Let ỹ (k) = (y (k) 1,..., y(k) m ). The default startng vector for the smultaneous Dykstra s algorthm n [IP91] s ỹ (0) = 0 X m, but we can warmstart Dykstra s algorthm as explaned n Subsecton 2.1. We now show that x (0) = d λ y (0) s ndeed the formula to warmstart the smultaneous Dykstra s algorthm. Proposton 5.4. (Formula for x (0) ) In Algorthm 5.1, for the startng dual vector ỹ (k) = (y (k) 1,..., y(k) m ) X m, the startng terate for x (0) s x (0) = d λ y (0). Proof. Let w (k) = (w (k) 1,..., w(k) m ) be the correcton vector correspondng to D. The terates (x (k),..., x (k) ) X m le n D for all k, and (d,..., d) D. From our study of Dykstra s algorthm earler, we have (w (k) 1,..., w(k) m ) = (d,..., d) (x(k),..., x (k) ) (y (k) 1,..., y(k) m ). Moreover, we have λ w (k) = 0 from Proposton 5.3, so λ (d x (k) y (k) ) = 0. Together wth the fact that λ = 1, we get the needed formula for x (0). We now look at how to mprove Algorthm 5.1. Lne 7 can be mproved by proectng (x (k) 1,..., x(k) m ) onto a set better than D. Recall that lne 4 produces supportng halfspaces of the set C. Consder the set Cm+1 k defned as the ntersecton of the supportng halfspaces produced n lne 4, and let C k X m be defned by C k = Cm+1 k Ck m+1 (m copes). We can add the set Ck to play the role of Cm+1 k n the extended Dykstra s algorthm (Algorthm 3.1) to enhance the algorthm A two-level Dykstra s algorthm. If we want to apply the SHQP strategy to enhance the smultaneous Dykstra s algorthm, then we mght want to cut up the problem nto smaller blocks so that the quadratc programs formed are defned by a manageable number of halfspaces. It s reasonable to assume that nformaton about the sets C communcate upwards from the leaves to the root of a tree (n the sense of graph theory). We llustrate wth an example wth m = 4 where we =1
18 SHQP STRATEGY FOR THE BAP 18 break down the sze of the quadratc programs to be at most 2. Let the sets D 1 and D 2 be defned by D 1 = {(x 1, x 2, x 3, x 4 ) X 4 : x 1 = x 2 } and D 2 = {(x 1, x 2, x 3, x 4 ) X 4 : x 3 = x 4 }. We present a two level Dykstra s algorthm. Algorthm 5.5. (Two level Dykstra s algorthm) Consder the BAP (1.1) where m = 4. Let y (0) X be the startng dual varables for each component {1,..., 4}. Set x (0) = d 4 =1 λ y (0). 01 For k = 1, 2, For {1, 2, 3, 4} 03 z (k) := x (k 1) + y (k 1) = P C (z (k) ) = z (k) x (k) 06 end for 07 x (k) (1,2) = λ1 λ 1+λ 2 x (k) 1 + λ2 λ 1+λ 2 x (k) 2 04 x (k) 05 y (k) 08 x (k) (3,4) = λ3 λ 3+λ 4 x (k) 3 + λ4 λ 3+λ 4 x (k) 4 09 x (k) = (λ 1 + λ 2 )x (k) (1,2) + (λ 3 + λ 4 )x (k) (3,4) 10 end for Lnes 2 to 6 descrbe the operaton nvolved n proectng onto C, whch s not dfferent from the smultaneous Dykstra s algorthm (Algorthm 5.1). Lne 7 descrbes the operaton n proectng onto D 1, lne 8 descrbes the operaton n proectng onto D 2, and lne 9 descrbes the operaton n proectng onto D. The agent that collects nformaton on x (k) 1 and x (k) 2 to obtan x (k) (1,2) can also collect the halfspaces generated by the proecton operaton used to obtan x (k) 1 and x (k) 2. We can make use of these halfspaces to form a superset of C that plays the role of Cm+1 k n the extended Dykstra s algorthm (Algorthm 3.1). In other words, the operatons n lnes 8 to 12 of Algorthm 3.1 can be nserted between lnes 7 and 8 of Algorthm 5.5. We can also nsert these same lnes between lnes 8 and 9 and between lnes 9 and 10 to enhance Algorthm 5.5. It s now easy to extend the prncples hghlghted here for problems nvolvng m > 4 sets and wth more than 2 levels. 6. Usng the APG for (D ) In ths secton, we depart from the dual alternatng mnmzaton strategy treated n the rest of the paper, and dscuss usng the accelerated proxmal gradent (APG) algorthm to solve (D ) n (2.1) n order to get a O(1/k 2 ) convergence rate. We remark that the APG can be augmented by the strategy descrbed n Subsecton 4.3. We recall the APG as presented n [Tse08, Secton 3], whch traces ts roots to Nesterov s second optmal method [Nes88]. We decde that t s best to adopt the notaton of [Tse08] even though t conflcts wth some of the notaton we have used n the rest of the paper.
19 SHQP STRATEGY FOR THE BAP 19 Algorthm 6.1. [Tse08, Algorthm 1] Consder the problem of mnmzng h(x) = f(x) + P (x), where f : X R s a smooth convex functon whose gradent f : X X s Lpschtz wth constant L, and P : X R s a (not necessarly smooth) convex functon. For each y X, defne l f ( ; y) : X R (a lnearzaton of h( ) at y) by l f (x; y) = f(y) + f(y), x y + P (x). Choose θ 0 (0, 1], x 0, z 0 dom(p ). k 0. Go to 1. (1) Choose a nonempty closed convex set X k X wth X k dom(p ). Let y k = (1 θ k )x k + θ k z k, (6.1) z k+1 = arg mn x Xk {l f (x; y k ) + θ kl 2 x z k 2 }, (6.2) ˆx k+1 = (1 θ k )x k + θ k z k+1. (6.3) Choose x k+1 such that h(x k+1 ) l f (ˆx k+1 ; y k ) + L 2 ˆx k+1 y k 2. (6.4) Choose θ k+1 (0, 1] satsfyng k k + 1, and go to 1. 1 θ k+1 θ 2 k+1 1. θ (6.5) 2 k The followng s the convergence result of Algorthm 6.1. We smplfy ther result by takng X k = X for all k. Theorem 6.2. [Tse08, Corollary 1(a)] Let {(x k, y k, z k, θ k, X k )} be generated by Algorthm 6.1 wth θ 0 = 1. Fx any ǫ > 0. Suppose θ k 2 k+2 (whch s the case when θ 0 = 1 and θ k+1 s determned from θ k by settng (6.5) to an equaton), and X k = X for all k. Then for any x dom(p ) wth h(x) nf(h) + ǫ, we have mn {h(x )} h(x) + ǫ whenever 4L k =0,1,...,k+1 ǫ x z 0 2. Even though the lne (6.4) s dfferent from that n [Tse08, (14)], t s easy to check that the nequalty [Tse08, (23)] remans vald wth ths change. Theorem 6.2 shows that the nfmum of {h(x k )} k produced by Algorthm 6.1 would converge to the nfmum of h( ). Furthermore, f a mnmzer of h( ) exsts, then the convergence rate of {h(x k ) nf(h)} k s of O(1/k 2 ). For the BAP (1.1) of proectng a pont onto the ntersecton of m sets, the functon h( ) was descrbed n (2.1). For ỹ X m, the mappng has Hessan ỹ d y 2 = f(y y m ) (6.6) I I I I I I..... I I I (.e., there are m 2 blocks n an m m block square matrx), and the gradent of the map n (6.6) s Lpschtz wth constant L = m. The step (6.2) can now be easly carred out usng Proposton 2.2(5) to obtan all m components of the mnmzer
20 SHQP STRATEGY FOR THE BAP 20 z +1. We can use the strategy descrbed n Subsecton 4.3 to get a better terate x k+1 satsfyng (6.4) than ˆx k Concluson In ths paper, we showed ways to ncorporate the SHQP heurstc to mprove Dykstra s algorthm. For the case when C are hyperplanes, the numercal experments n [Pan15a] shows the effectveness of the strateges explaned n ths paper. We defer further numercal experments to future work. Appendx A. Proof of convergence of Algorthm 3.1 In ths appendx, we present the proof of convergence of Algorthm 3.1, the extended Dykstra s algorthm. We already saw that f Hm+1 k = Ck m+1 = X for all k 0, then Algorthm 3.1 reduces to the orgnal Dykstra s algorthm. Apart from Theorem 3.4, our proof s mostly the same as the Boyle-Dykstra theorem [BD85] as presented n [Deu01b]. Note that the proof here also ncludes the warmstart case. Throughout ths secton, we follow the notaton of Algorthm 3.1. We need to follow the notaton n [Deu01b] and defne the sequences {e } = m and { x } =0 by e (m+1)(k 1)+ = y (k) (A.1) x (m+1)(k 1)+ = x (k). (A.2) The statement of Lemma A.6 makes the new notaton more natural. We denote [] to be the nteger n {1,..., m + 1} such that m + 1 dvdes []. Lemma A.1. In Algorthm 3.1, for each 1, such that [] {1,..., m}. Furthermore, f [] = m + 1, then δ (e, C [] y) = x y, e 0 for all y C []. (A.3) δ (e, C /(m+1) m+1 y) = x y, e 0 for all y C. (A.4) Proof. The proof of nequalty (A.3) s exactly the same as [Deu01b, Lemma 9.17], but our statement s now only vald for all n 1. We have x y, e = P C[] ( x 1 + e (m+1) ) y, x 1 + e (m+1) P K[] ( x 1 + e (m+1) ) 0, where the nequalty s an mmedate consequence from the propertes of proectons. The second nequalty n (A.4) s also clear. The equatons n both (A.3) and (A.4) are straghtforward from the defnton of δ (, ). Lemma A.2. In Algorthm 3.1, for each 0, d x = e m + e (m 1) + + e 1 + e. (A.5) Proof. Ths s easly seen from lnes 4 and 9 of Algorthm 3.1 and the formula for z (k) n (2.5). Lemma A.3. In Algorthm 3.1, { x } s a bounded sequence, and x 1 x 2 <. In partcular, =1 x 1 x 0 as. (A.6) (A.7)
21 SHQP STRATEGY FOR THE BAP 21 Proof. Formula (A.6) s ust a rephrasng of Theorem 3.4(2). Formula (A.7) follows easly. We now show the boundedness of { x }. For, let k = m+1. Defne v as v := 1 2 x P C (d) 2 + l= m Recall the defnton of h k ( ) n (3.1). We have v = 1 2 x P C (d) 2 + l= m e l, x l P C (d) e l, x l P C (d). (A.8) = 1 k(m+1) 2 x(k) k(m+1) P C(d) 2 + δ (y (k) l, C [l] P C (d)) +δ (y (k 1) m+1, Ck 1 m+1 P C(d)) + l=1 m l= k(m+1)+1 = h k 1 (y (k) 1, y(k) 2,..., y(k) k(m+1), y(k 1) k(m+1)+1,..., y(k 1) m+1 ) d, P C (d) P C(d) 2. δ (y (k 1) l, C [l] P C (d)) The proof of Theorem 3.4 shows that v s non-ncreasng.snce 0 Cm+1 k 1 P C(d) and 0 C [l] P C (d), we have v 1 2 x P C (d) 2 (ust lke n Proposton 2.2(3)), whch shows that { x } s a bounded sequence. Lemma A.4. In Algorthm 3.1, for any N, e x k 1 x k + k=1 max 1 l m+1 e l (m+1). (A.9) Proof. The proof s adusted from [Deu01b, Lemma 9.21]. We nduct on. It s clear to see that (A.9) holds for all { m,..., 0}. Suppose (A.9) holds for all r. Let M 1 = max 1 l m+1 e l (m+1). Then e +1 = x x +1 + e +1 (m+1) x x +1 + e +1 (m+1) +1 (m+1) x x +1 + x k 1 x k + M 1 +1 x k 1 x k + M 1, k=1 k=1 whch mples that (A.9) holds for r = + 1. Lemma A.5. In Algorthm 3.1, lm nf x k x, e k = 0. (A.10) k= m
22 SHQP STRATEGY FOR THE BAP 22 Proof. The proof needs to be adusted from [Deu01b, Lemma 9.22]. Let M 1 = max 1 l m+1 e l (m+1). Usng Schwarz s nequalty and Lemma A.4, we get k= m x k x, e k e k x k x k= m [( ) ] M 1 + k x 1 x x k x k= m =1 [( ) ( )] k x 1 x x l 1 x l + M 1 x k x k= m =1 l=k+1 k= m (m + 1) x 1 x x l 1 x l + M 1 x k x. =1 l= (m 1) } {{ } (1) k= m } {{ } (2) Term (2) converges to zero by Lemma A.3. Let a = x 1 x. To show our result, t suffces to show that [( ) ( )] lm nf a a l = 0 =1 l= (m 1) gven that =1 a2 s fnte. We refer the reader to the proof n [Deu01b, Lemma 9.22] for the proof of ths fact. Lemma A.6. In Algorthm 3.1, there exsts a subsequence { x } of { x } such that lm sup y x, d x 0 for each y C, and (A.11) lm k= m x k x, e k = 0. (A.12) Proof. The proof s almost exactly the same as [Deu01b, Lemma 9.23]. Lemma A.2, we have for all y C, m that y x, d x = y x, e m + e m e = y x, e k = k= m k= m y x k, e k + k= m By Lemma A.1, the frst sum s no more than 0. Hence y x, d x k= m x k x, e k. x k x, e k. Usng (A.13) By Lemma A.5, we deduce that there s a subsequence { } such that (A.12) holds. Note that the rght hand sde of (A.13) does not depend on y. In vew of (A.13), t follows that (A.11) also holds.
23 SHQP STRATEGY FOR THE BAP 23 Theorem A.7. (Warmstart Boyle-Dykstra Theorem) Consder Algorthm 3.1. Defne the sequence { x n } as n Step 2 of Algorthm 3.1 and (A.2). Then lm x P C (d) = 0. Proof. The proof of ths result s mostly the same as [Deu01b, Lemma 9.23]. By Lemma A.6, there exsts a subsequence { x } such that lm sup y x, d x 0 for each y C. (A.14) Snce { x } s bounded by Lemma A.3, t follows by [Deu01b, Theorem 9.12] (by passng to a further subsequence f necessary), that there s a y 0 X such that x w y0, (A.15) and lm x exsts. (A.16) By another property of Hlbert spaces ([Deu01b, Theorem 9.13]), y 0 lm nf x = lm x. (A.17) Snce [] takes on only m + 1 possbltes, an nfnte number of the s must be of the same value. If ths value s n {1,..., m}, say 0, then an nfnte number of the x s le n C 0. Snce C 0 s closed and convex, t s weakly closed by [Deu01b, Theorem 9.16], and hence y 0 C 0. By (A.7), x x 1 0. By a repeated applcaton of ths fact, we see that all the sequences { x +1}, { x +2},... converge weakly to y 0, and hence y 0 C for every. That s, y 0 C. For any y C, (A.17) and (A.14) mply that y y 0, d y 0 = y, d y, y 0 y 0, d + y 0 2 (A.18) lm [ y, d y, x x, d + x 2 ] = lm y x, d x 0. Hence y 0 = P C (d). Moreover, puttng y = y 0 n (A.18), we get equalty n the chan of nequaltes, and hence lm x 2 = y 0 2 (A.19) and lm y 0 x, d x = 0. By (A.15) and (A.19), t follows from [Deu01b, Theorem 9.10(2)] that x y 0 0. Hence x P C (d) = x y 0 0. (A.20)
MMA and GCMMA two methods for nonlinear optimization
MMA and GCMMA two methods for nonlnear optmzaton Krster Svanberg Optmzaton and Systems Theory, KTH, Stockholm, Sweden. krlle@math.kth.se Ths note descrbes the algorthms used n the author s 2007 mplementatons
More informationModule 9. Lecture 6. Duality in Assignment Problems
Module 9 1 Lecture 6 Dualty n Assgnment Problems In ths lecture we attempt to answer few other mportant questons posed n earler lecture for (AP) and see how some of them can be explaned through the concept
More informationprinceton univ. F 17 cos 521: Advanced Algorithm Design Lecture 7: LP Duality Lecturer: Matt Weinberg
prnceton unv. F 17 cos 521: Advanced Algorthm Desgn Lecture 7: LP Dualty Lecturer: Matt Wenberg Scrbe: LP Dualty s an extremely useful tool for analyzng structural propertes of lnear programs. Whle there
More informationSome modelling aspects for the Matlab implementation of MMA
Some modellng aspects for the Matlab mplementaton of MMA Krster Svanberg krlle@math.kth.se Optmzaton and Systems Theory Department of Mathematcs KTH, SE 10044 Stockholm September 2004 1. Consdered optmzaton
More informationLecture 10 Support Vector Machines II
Lecture 10 Support Vector Machnes II 22 February 2016 Taylor B. Arnold Yale Statstcs STAT 365/665 1/28 Notes: Problem 3 s posted and due ths upcomng Frday There was an early bug n the fake-test data; fxed
More informationFeature Selection: Part 1
CSE 546: Machne Learnng Lecture 5 Feature Selecton: Part 1 Instructor: Sham Kakade 1 Regresson n the hgh dmensonal settng How do we learn when the number of features d s greater than the sample sze n?
More informationKernel Methods and SVMs Extension
Kernel Methods and SVMs Extenson The purpose of ths document s to revew materal covered n Machne Learnng 1 Supervsed Learnng regardng support vector machnes (SVMs). Ths document also provdes a general
More informationAssortment Optimization under MNL
Assortment Optmzaton under MNL Haotan Song Aprl 30, 2017 1 Introducton The assortment optmzaton problem ams to fnd the revenue-maxmzng assortment of products to offer when the prces of products are fxed.
More informationCOS 521: Advanced Algorithms Game Theory and Linear Programming
COS 521: Advanced Algorthms Game Theory and Lnear Programmng Moses Charkar February 27, 2013 In these notes, we ntroduce some basc concepts n game theory and lnear programmng (LP). We show a connecton
More informationLecture 20: November 7
0-725/36-725: Convex Optmzaton Fall 205 Lecturer: Ryan Tbshran Lecture 20: November 7 Scrbes: Varsha Chnnaobreddy, Joon Sk Km, Lngyao Zhang Note: LaTeX template courtesy of UC Berkeley EECS dept. Dsclamer:
More informationMore metrics on cartesian products
More metrcs on cartesan products If (X, d ) are metrc spaces for 1 n, then n Secton II4 of the lecture notes we defned three metrcs on X whose underlyng topologes are the product topology The purpose of
More informationU.C. Berkeley CS294: Beyond Worst-Case Analysis Luca Trevisan September 5, 2017
U.C. Berkeley CS94: Beyond Worst-Case Analyss Handout 4s Luca Trevsan September 5, 07 Summary of Lecture 4 In whch we ntroduce semdefnte programmng and apply t to Max Cut. Semdefnte Programmng Recall that
More informationDifference Equations
Dfference Equatons c Jan Vrbk 1 Bascs Suppose a sequence of numbers, say a 0,a 1,a,a 3,... s defned by a certan general relatonshp between, say, three consecutve values of the sequence, e.g. a + +3a +1
More informationChapter 5. Solution of System of Linear Equations. Module No. 6. Solution of Inconsistent and Ill Conditioned Systems
Numercal Analyss by Dr. Anta Pal Assstant Professor Department of Mathematcs Natonal Insttute of Technology Durgapur Durgapur-713209 emal: anta.bue@gmal.com 1 . Chapter 5 Soluton of System of Lnear Equatons
More informationAffine transformations and convexity
Affne transformatons and convexty The purpose of ths document s to prove some basc propertes of affne transformatons nvolvng convex sets. Here are a few onlne references for background nformaton: http://math.ucr.edu/
More informationSolutions to exam in SF1811 Optimization, Jan 14, 2015
Solutons to exam n SF8 Optmzaton, Jan 4, 25 3 3 O------O -4 \ / \ / The network: \/ where all lnks go from left to rght. /\ / \ / \ 6 O------O -5 2 4.(a) Let x = ( x 3, x 4, x 23, x 24 ) T, where the varable
More informationGeneral viscosity iterative method for a sequence of quasi-nonexpansive mappings
Avalable onlne at www.tjnsa.com J. Nonlnear Sc. Appl. 9 (2016), 5672 5682 Research Artcle General vscosty teratve method for a sequence of quas-nonexpansve mappngs Cuje Zhang, Ynan Wang College of Scence,
More informationContents 1. Introduction 1 2. Algorithms 3 3. Strong convergence results 5 4. Performance of acceleration Conclusion 16 References 16
ACCELERATING THE ALTERNATING PROJECTION ALGORITHM FOR THE CASE OF AFFINE SUBSPACES USING SUPPORTING HYPERPLANES arxv:1406.4012v2 [math.oc] 16 Jul 2014 C.H. JEFFREY PANG Abstract. The von Neumann-Halpern
More informationLinear, affine, and convex sets and hulls In the sequel, unless otherwise specified, X will denote a real vector space.
Lnear, affne, and convex sets and hulls In the sequel, unless otherwse specfed, X wll denote a real vector space. Lnes and segments. Gven two ponts x, y X, we defne xy = {x + t(y x) : t R} = {(1 t)x +
More informationOn the Global Linear Convergence of the ADMM with Multi-Block Variables
On the Global Lnear Convergence of the ADMM wth Mult-Block Varables Tany Ln Shqan Ma Shuzhong Zhang May 31, 01 Abstract The alternatng drecton method of multplers ADMM has been wdely used for solvng structured
More information6.854J / J Advanced Algorithms Fall 2008
MIT OpenCourseWare http://ocw.mt.edu 6.854J / 18.415J Advanced Algorthms Fall 2008 For nformaton about ctng these materals or our Terms of Use, vst: http://ocw.mt.edu/terms. 18.415/6.854 Advanced Algorthms
More informationYong Joon Ryang. 1. Introduction Consider the multicommodity transportation problem with convex quadratic cost function. 1 2 (x x0 ) T Q(x x 0 )
Kangweon-Kyungk Math. Jour. 4 1996), No. 1, pp. 7 16 AN ITERATIVE ROW-ACTION METHOD FOR MULTICOMMODITY TRANSPORTATION PROBLEMS Yong Joon Ryang Abstract. The optmzaton problems wth quadratc constrants often
More informationSTAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 16
STAT 39: MATHEMATICAL COMPUTATIONS I FALL 218 LECTURE 16 1 why teratve methods f we have a lnear system Ax = b where A s very, very large but s ether sparse or structured (eg, banded, Toepltz, banded plus
More informationErrors for Linear Systems
Errors for Lnear Systems When we solve a lnear system Ax b we often do not know A and b exactly, but have only approxmatons  and ˆb avalable. Then the best thng we can do s to solve ˆx ˆb exactly whch
More informationCollege of Computer & Information Science Fall 2009 Northeastern University 20 October 2009
College of Computer & Informaton Scence Fall 2009 Northeastern Unversty 20 October 2009 CS7880: Algorthmc Power Tools Scrbe: Jan Wen and Laura Poplawsk Lecture Outlne: Prmal-dual schema Network Desgn:
More informationSupplement: Proofs and Technical Details for The Solution Path of the Generalized Lasso
Supplement: Proofs and Techncal Detals for The Soluton Path of the Generalzed Lasso Ryan J. Tbshran Jonathan Taylor In ths document we gve supplementary detals to the paper The Soluton Path of the Generalzed
More informationLectures - Week 4 Matrix norms, Conditioning, Vector Spaces, Linear Independence, Spanning sets and Basis, Null space and Range of a Matrix
Lectures - Week 4 Matrx norms, Condtonng, Vector Spaces, Lnear Independence, Spannng sets and Bass, Null space and Range of a Matrx Matrx Norms Now we turn to assocatng a number to each matrx. We could
More informationConvergence rates of proximal gradient methods via the convex conjugate
Convergence rates of proxmal gradent methods va the convex conjugate Davd H Gutman Javer F Peña January 8, 018 Abstract We gve a novel proof of the O(1/ and O(1/ convergence rates of the proxmal gradent
More informationNUMERICAL DIFFERENTIATION
NUMERICAL DIFFERENTIATION 1 Introducton Dfferentaton s a method to compute the rate at whch a dependent output y changes wth respect to the change n the ndependent nput x. Ths rate of change s called the
More informationDeriving the X-Z Identity from Auxiliary Space Method
Dervng the X-Z Identty from Auxlary Space Method Long Chen Department of Mathematcs, Unversty of Calforna at Irvne, Irvne, CA 92697 chenlong@math.uc.edu 1 Iteratve Methods In ths paper we dscuss teratve
More informationLecture 12: Discrete Laplacian
Lecture 12: Dscrete Laplacan Scrbe: Tanye Lu Our goal s to come up wth a dscrete verson of Laplacan operator for trangulated surfaces, so that we can use t n practce to solve related problems We are mostly
More informationGeneralized Linear Methods
Generalzed Lnear Methods 1 Introducton In the Ensemble Methods the general dea s that usng a combnaton of several weak learner one could make a better learner. More formally, assume that we have a set
More informationLecture 20: Lift and Project, SDP Duality. Today we will study the Lift and Project method. Then we will prove the SDP duality theorem.
prnceton u. sp 02 cos 598B: algorthms and complexty Lecture 20: Lft and Project, SDP Dualty Lecturer: Sanjeev Arora Scrbe:Yury Makarychev Today we wll study the Lft and Project method. Then we wll prove
More informationLecture Notes on Linear Regression
Lecture Notes on Lnear Regresson Feng L fl@sdueducn Shandong Unversty, Chna Lnear Regresson Problem In regresson problem, we am at predct a contnuous target value gven an nput feature vector We assume
More informationProblem Set 9 Solutions
Desgn and Analyss of Algorthms May 4, 2015 Massachusetts Insttute of Technology 6.046J/18.410J Profs. Erk Demane, Srn Devadas, and Nancy Lynch Problem Set 9 Solutons Problem Set 9 Solutons Ths problem
More informationAPPENDIX A Some Linear Algebra
APPENDIX A Some Lnear Algebra The collecton of m, n matrces A.1 Matrces a 1,1,..., a 1,n A = a m,1,..., a m,n wth real elements a,j s denoted by R m,n. If n = 1 then A s called a column vector. Smlarly,
More informationLinear Approximation with Regularization and Moving Least Squares
Lnear Approxmaton wth Regularzaton and Movng Least Squares Igor Grešovn May 007 Revson 4.6 (Revson : March 004). 5 4 3 0.5 3 3.5 4 Contents: Lnear Fttng...4. Weghted Least Squares n Functon Approxmaton...
More informationNumerical Heat and Mass Transfer
Master degree n Mechancal Engneerng Numercal Heat and Mass Transfer 06-Fnte-Dfference Method (One-dmensonal, steady state heat conducton) Fausto Arpno f.arpno@uncas.t Introducton Why we use models and
More informationModule 3 LOSSY IMAGE COMPRESSION SYSTEMS. Version 2 ECE IIT, Kharagpur
Module 3 LOSSY IMAGE COMPRESSION SYSTEMS Verson ECE IIT, Kharagpur Lesson 6 Theory of Quantzaton Verson ECE IIT, Kharagpur Instructonal Objectves At the end of ths lesson, the students should be able to:
More informationSalmon: Lectures on partial differential equations. Consider the general linear, second-order PDE in the form. ,x 2
Salmon: Lectures on partal dfferental equatons 5. Classfcaton of second-order equatons There are general methods for classfyng hgher-order partal dfferental equatons. One s very general (applyng even to
More informationSELECTED SOLUTIONS, SECTION (Weak duality) Prove that the primal and dual values p and d defined by equations (4.3.2) and (4.3.3) satisfy p d.
SELECTED SOLUTIONS, SECTION 4.3 1. Weak dualty Prove that the prmal and dual values p and d defned by equatons 4.3. and 4.3.3 satsfy p d. We consder an optmzaton problem of the form The Lagrangan for ths
More informationCSci 6974 and ECSE 6966 Math. Tech. for Vision, Graphics and Robotics Lecture 21, April 17, 2006 Estimating A Plane Homography
CSc 6974 and ECSE 6966 Math. Tech. for Vson, Graphcs and Robotcs Lecture 21, Aprl 17, 2006 Estmatng A Plane Homography Overvew We contnue wth a dscusson of the major ssues, usng estmaton of plane projectve
More informationExercise Solutions to Real Analysis
xercse Solutons to Real Analyss Note: References refer to H. L. Royden, Real Analyss xersze 1. Gven any set A any ɛ > 0, there s an open set O such that A O m O m A + ɛ. Soluton 1. If m A =, then there
More informationarxiv: v1 [math.oc] 27 Jun 2015
arxv:506.08246v [math.oc] 27 Jun 205 NONCONVEX SET INTERSECTION PROBLEMS: FROM PROJECTION METHODS TO THE NEWTON METHOD FOR SUPER-REGULAR SETS C.H. JEFFREY PANG Abstract. The problem of fndng a pont n the
More informationVector Norms. Chapter 7 Iterative Techniques in Matrix Algebra. Cauchy-Bunyakovsky-Schwarz Inequality for Sums. Distances. Convergence.
Vector Norms Chapter 7 Iteratve Technques n Matrx Algebra Per-Olof Persson persson@berkeley.edu Department of Mathematcs Unversty of Calforna, Berkeley Math 128B Numercal Analyss Defnton A vector norm
More informationThe Order Relation and Trace Inequalities for. Hermitian Operators
Internatonal Mathematcal Forum, Vol 3, 08, no, 507-57 HIKARI Ltd, wwwm-hkarcom https://doorg/0988/mf088055 The Order Relaton and Trace Inequaltes for Hermtan Operators Y Huang School of Informaton Scence
More informationVARIATION OF CONSTANT SUM CONSTRAINT FOR INTEGER MODEL WITH NON UNIFORM VARIABLES
VARIATION OF CONSTANT SUM CONSTRAINT FOR INTEGER MODEL WITH NON UNIFORM VARIABLES BÂRZĂ, Slvu Faculty of Mathematcs-Informatcs Spru Haret Unversty barza_slvu@yahoo.com Abstract Ths paper wants to contnue
More informationResearch Article. Almost Sure Convergence of Random Projected Proximal and Subgradient Algorithms for Distributed Nonsmooth Convex Optimization
To appear n Optmzaton Vol. 00, No. 00, Month 20XX, 1 27 Research Artcle Almost Sure Convergence of Random Projected Proxmal and Subgradent Algorthms for Dstrbuted Nonsmooth Convex Optmzaton Hdea Idua a
More informationAdditional Codes using Finite Difference Method. 1 HJB Equation for Consumption-Saving Problem Without Uncertainty
Addtonal Codes usng Fnte Dfference Method Benamn Moll 1 HJB Equaton for Consumpton-Savng Problem Wthout Uncertanty Before consderng the case wth stochastc ncome n http://www.prnceton.edu/~moll/ HACTproect/HACT_Numercal_Appendx.pdf,
More informationREAL ANALYSIS I HOMEWORK 1
REAL ANALYSIS I HOMEWORK CİHAN BAHRAN The questons are from Tao s text. Exercse 0.0.. If (x α ) α A s a collecton of numbers x α [0, + ] such that x α
More informationTHE CHINESE REMAINDER THEOREM. We should thank the Chinese for their wonderful remainder theorem. Glenn Stevens
THE CHINESE REMAINDER THEOREM KEITH CONRAD We should thank the Chnese for ther wonderful remander theorem. Glenn Stevens 1. Introducton The Chnese remander theorem says we can unquely solve any par of
More information3.1 Expectation of Functions of Several Random Variables. )' be a k-dimensional discrete or continuous random vector, with joint PMF p (, E X E X1 E X
Statstcs 1: Probablty Theory II 37 3 EPECTATION OF SEVERAL RANDOM VARIABLES As n Probablty Theory I, the nterest n most stuatons les not on the actual dstrbuton of a random vector, but rather on a number
More informationSection 8.3 Polar Form of Complex Numbers
80 Chapter 8 Secton 8 Polar Form of Complex Numbers From prevous classes, you may have encountered magnary numbers the square roots of negatve numbers and, more generally, complex numbers whch are the
More informationCase A. P k = Ni ( 2L i k 1 ) + (# big cells) 10d 2 P k.
THE CELLULAR METHOD In ths lecture, we ntroduce the cellular method as an approach to ncdence geometry theorems lke the Szemeréd-Trotter theorem. The method was ntroduced n the paper Combnatoral complexty
More informationThe Second Anti-Mathima on Game Theory
The Second Ant-Mathma on Game Theory Ath. Kehagas December 1 2006 1 Introducton In ths note we wll examne the noton of game equlbrum for three types of games 1. 2-player 2-acton zero-sum games 2. 2-player
More informationSolutions HW #2. minimize. Ax = b. Give the dual problem, and make the implicit equality constraints explicit. Solution.
Solutons HW #2 Dual of general LP. Fnd the dual functon of the LP mnmze subject to c T x Gx h Ax = b. Gve the dual problem, and make the mplct equalty constrants explct. Soluton. 1. The Lagrangan s L(x,
More informationThe Minimum Universal Cost Flow in an Infeasible Flow Network
Journal of Scences, Islamc Republc of Iran 17(2): 175-180 (2006) Unversty of Tehran, ISSN 1016-1104 http://jscencesutacr The Mnmum Unversal Cost Flow n an Infeasble Flow Network H Saleh Fathabad * M Bagheran
More informationPerfect Competition and the Nash Bargaining Solution
Perfect Competton and the Nash Barganng Soluton Renhard John Department of Economcs Unversty of Bonn Adenauerallee 24-42 53113 Bonn, Germany emal: rohn@un-bonn.de May 2005 Abstract For a lnear exchange
More informationFoundations of Arithmetic
Foundatons of Arthmetc Notaton We shall denote the sum and product of numbers n the usual notaton as a 2 + a 2 + a 3 + + a = a, a 1 a 2 a 3 a = a The notaton a b means a dvdes b,.e. ac = b where c s an
More informationInner Product. Euclidean Space. Orthonormal Basis. Orthogonal
Inner Product Defnton 1 () A Eucldean space s a fnte-dmensonal vector space over the reals R, wth an nner product,. Defnton 2 (Inner Product) An nner product, on a real vector space X s a symmetrc, blnear,
More informationSome Comments on Accelerating Convergence of Iterative Sequences Using Direct Inversion of the Iterative Subspace (DIIS)
Some Comments on Acceleratng Convergence of Iteratve Sequences Usng Drect Inverson of the Iteratve Subspace (DIIS) C. Davd Sherrll School of Chemstry and Bochemstry Georga Insttute of Technology May 1998
More informationThe Geometry of Logit and Probit
The Geometry of Logt and Probt Ths short note s meant as a supplement to Chapters and 3 of Spatal Models of Parlamentary Votng and the notaton and reference to fgures n the text below s to those two chapters.
More informationA CHARACTERIZATION OF ADDITIVE DERIVATIONS ON VON NEUMANN ALGEBRAS
Journal of Mathematcal Scences: Advances and Applcatons Volume 25, 2014, Pages 1-12 A CHARACTERIZATION OF ADDITIVE DERIVATIONS ON VON NEUMANN ALGEBRAS JIA JI, WEN ZHANG and XIAOFEI QI Department of Mathematcs
More information1 Convex Optimization
Convex Optmzaton We wll consder convex optmzaton problems. Namely, mnmzaton problems where the objectve s convex (we assume no constrants for now). Such problems often arse n machne learnng. For example,
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 12 10/21/2013. Martingale Concentration Inequalities and Applications
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.65/15.070J Fall 013 Lecture 1 10/1/013 Martngale Concentraton Inequaltes and Applcatons Content. 1. Exponental concentraton for martngales wth bounded ncrements.
More informationAppendix B. Criterion of Riemann-Stieltjes Integrability
Appendx B. Crteron of Remann-Steltes Integrablty Ths note s complementary to [R, Ch. 6] and [T, Sec. 3.5]. The man result of ths note s Theorem B.3, whch provdes the necessary and suffcent condtons for
More informationWeek 5: Neural Networks
Week 5: Neural Networks Instructor: Sergey Levne Neural Networks Summary In the prevous lecture, we saw how we can construct neural networks by extendng logstc regresson. Neural networks consst of multple
More information10-801: Advanced Optimization and Randomized Methods Lecture 2: Convex functions (Jan 15, 2014)
0-80: Advanced Optmzaton and Randomzed Methods Lecture : Convex functons (Jan 5, 04) Lecturer: Suvrt Sra Addr: Carnege Mellon Unversty, Sprng 04 Scrbes: Avnava Dubey, Ahmed Hefny Dsclamer: These notes
More informationMAT 578 Functional Analysis
MAT 578 Functonal Analyss John Qugg Fall 2008 Locally convex spaces revsed September 6, 2008 Ths secton establshes the fundamental propertes of locally convex spaces. Acknowledgment: although I wrote these
More informationOn the correction of the h-index for career length
1 On the correcton of the h-ndex for career length by L. Egghe Unverstet Hasselt (UHasselt), Campus Depenbeek, Agoralaan, B-3590 Depenbeek, Belgum 1 and Unverstet Antwerpen (UA), IBW, Stadscampus, Venusstraat
More informationOn a direct solver for linear least squares problems
ISSN 2066-6594 Ann. Acad. Rom. Sc. Ser. Math. Appl. Vol. 8, No. 2/2016 On a drect solver for lnear least squares problems Constantn Popa Abstract The Null Space (NS) algorthm s a drect solver for lnear
More informationSingular Value Decomposition: Theory and Applications
Sngular Value Decomposton: Theory and Applcatons Danel Khashab Sprng 2015 Last Update: March 2, 2015 1 Introducton A = UDV where columns of U and V are orthonormal and matrx D s dagonal wth postve real
More informationThe Multiple Classical Linear Regression Model (CLRM): Specification and Assumptions. 1. Introduction
ECONOMICS 5* -- NOTE (Summary) ECON 5* -- NOTE The Multple Classcal Lnear Regresson Model (CLRM): Specfcaton and Assumptons. Introducton CLRM stands for the Classcal Lnear Regresson Model. The CLRM s also
More informationLecture 4. Instructor: Haipeng Luo
Lecture 4 Instructor: Hapeng Luo In the followng lectures, we focus on the expert problem and study more adaptve algorthms. Although Hedge s proven to be worst-case optmal, one may wonder how well t would
More informationSL n (F ) Equals its Own Derived Group
Internatonal Journal of Algebra, Vol. 2, 2008, no. 12, 585-594 SL n (F ) Equals ts Own Derved Group Jorge Macel BMCC-The Cty Unversty of New York, CUNY 199 Chambers street, New York, NY 10007, USA macel@cms.nyu.edu
More information2E Pattern Recognition Solutions to Introduction to Pattern Recognition, Chapter 2: Bayesian pattern classification
E395 - Pattern Recognton Solutons to Introducton to Pattern Recognton, Chapter : Bayesan pattern classfcaton Preface Ths document s a soluton manual for selected exercses from Introducton to Pattern Recognton
More informationInexact Newton Methods for Inverse Eigenvalue Problems
Inexact Newton Methods for Inverse Egenvalue Problems Zheng-jan Ba Abstract In ths paper, we survey some of the latest development n usng nexact Newton-lke methods for solvng nverse egenvalue problems.
More informationThe optimal delay of the second test is therefore approximately 210 hours earlier than =2.
THE IEC 61508 FORMULAS 223 The optmal delay of the second test s therefore approxmately 210 hours earler than =2. 8.4 The IEC 61508 Formulas IEC 61508-6 provdes approxmaton formulas for the PF for smple
More informationarxiv: v1 [math.co] 1 Mar 2014
Unon-ntersectng set systems Gyula O.H. Katona and Dánel T. Nagy March 4, 014 arxv:1403.0088v1 [math.co] 1 Mar 014 Abstract Three ntersecton theorems are proved. Frst, we determne the sze of the largest
More informationSpectral Graph Theory and its Applications September 16, Lecture 5
Spectral Graph Theory and ts Applcatons September 16, 2004 Lecturer: Danel A. Spelman Lecture 5 5.1 Introducton In ths lecture, we wll prove the followng theorem: Theorem 5.1.1. Let G be a planar graph
More informationA 2D Bounded Linear Program (H,c) 2D Linear Programming
A 2D Bounded Lnear Program (H,c) h 3 v h 8 h 5 c h 4 h h 6 h 7 h 2 2D Lnear Programmng C s a polygonal regon, the ntersecton of n halfplanes. (H, c) s nfeasble, as C s empty. Feasble regon C s unbounded
More informationComplete subgraphs in multipartite graphs
Complete subgraphs n multpartte graphs FLORIAN PFENDER Unverstät Rostock, Insttut für Mathematk D-18057 Rostock, Germany Floran.Pfender@un-rostock.de Abstract Turán s Theorem states that every graph G
More informationStanford University CS359G: Graph Partitioning and Expanders Handout 4 Luca Trevisan January 13, 2011
Stanford Unversty CS359G: Graph Parttonng and Expanders Handout 4 Luca Trevsan January 3, 0 Lecture 4 In whch we prove the dffcult drecton of Cheeger s nequalty. As n the past lectures, consder an undrected
More informationEEE 241: Linear Systems
EEE : Lnear Systems Summary #: Backpropagaton BACKPROPAGATION The perceptron rule as well as the Wdrow Hoff learnng were desgned to tran sngle layer networks. They suffer from the same dsadvantage: they
More informationAssortment Optimization under the Paired Combinatorial Logit Model
Assortment Optmzaton under the Pared Combnatoral Logt Model Heng Zhang, Paat Rusmevchentong Marshall School of Busness, Unversty of Southern Calforna, Los Angeles, CA 90089 hengz@usc.edu, rusmevc@marshall.usc.edu
More informationU.C. Berkeley CS294: Spectral Methods and Expanders Handout 8 Luca Trevisan February 17, 2016
U.C. Berkeley CS94: Spectral Methods and Expanders Handout 8 Luca Trevsan February 7, 06 Lecture 8: Spectral Algorthms Wrap-up In whch we talk about even more generalzatons of Cheeger s nequaltes, and
More informationREGULAR POSITIVE TERNARY QUADRATIC FORMS. 1. Introduction
REGULAR POSITIVE TERNARY QUADRATIC FORMS BYEONG-KWEON OH Abstract. A postve defnte quadratc form f s sad to be regular f t globally represents all ntegers that are represented by the genus of f. In 997
More informationAn Interactive Optimisation Tool for Allocation Problems
An Interactve Optmsaton ool for Allocaton Problems Fredr Bonäs, Joam Westerlund and apo Westerlund Process Desgn Laboratory, Faculty of echnology, Åbo Aadem Unversty, uru 20500, Fnland hs paper presents
More informationEdge Isoperimetric Inequalities
November 7, 2005 Ross M. Rchardson Edge Isopermetrc Inequaltes 1 Four Questons Recall that n the last lecture we looked at the problem of sopermetrc nequaltes n the hypercube, Q n. Our noton of boundary
More information1 Matrix representations of canonical matrices
1 Matrx representatons of canoncal matrces 2-d rotaton around the orgn: ( ) cos θ sn θ R 0 = sn θ cos θ 3-d rotaton around the x-axs: R x = 1 0 0 0 cos θ sn θ 0 sn θ cos θ 3-d rotaton around the y-axs:
More informationCHAPTER 5 NUMERICAL EVALUATION OF DYNAMIC RESPONSE
CHAPTER 5 NUMERICAL EVALUATION OF DYNAMIC RESPONSE Analytcal soluton s usually not possble when exctaton vares arbtrarly wth tme or f the system s nonlnear. Such problems can be solved by numercal tmesteppng
More informationReport on Image warping
Report on Image warpng Xuan Ne, Dec. 20, 2004 Ths document summarzed the algorthms of our mage warpng soluton for further study, and there s a detaled descrpton about the mplementaton of these algorthms.
More informationMaximizing the number of nonnegative subsets
Maxmzng the number of nonnegatve subsets Noga Alon Hao Huang December 1, 213 Abstract Gven a set of n real numbers, f the sum of elements of every subset of sze larger than k s negatve, what s the maxmum
More informationThe Finite Element Method: A Short Introduction
Te Fnte Element Metod: A Sort ntroducton Wat s FEM? Te Fnte Element Metod (FEM) ntroduced by engneers n late 50 s and 60 s s a numercal tecnque for solvng problems wc are descrbed by Ordnary Dfferental
More informationn α j x j = 0 j=1 has a nontrivial solution. Here A is the n k matrix whose jth column is the vector for all t j=0
MODULE 2 Topcs: Lnear ndependence, bass and dmenson We have seen that f n a set of vectors one vector s a lnear combnaton of the remanng vectors n the set then the span of the set s unchanged f that vector
More informationSome basic inequalities. Definition. Let V be a vector space over the complex numbers. An inner product is given by a function, V V C
Some basc nequaltes Defnton. Let V be a vector space over the complex numbers. An nner product s gven by a functon, V V C (x, y) x, y satsfyng the followng propertes (for all x V, y V and c C) (1) x +
More informationPerron Vectors of an Irreducible Nonnegative Interval Matrix
Perron Vectors of an Irreducble Nonnegatve Interval Matrx Jr Rohn August 4 2005 Abstract As s well known an rreducble nonnegatve matrx possesses a unquely determned Perron vector. As the man result of
More informationRandomized block proximal damped Newton method for composite self-concordant minimization
Randomzed block proxmal damped Newton method for composte self-concordant mnmzaton Zhaosong Lu June 30, 2016 Revsed: March 28, 2017 Abstract In ths paper we consder the composte self-concordant CSC mnmzaton
More informationGame Theory. Lecture Notes By Y. Narahari. Department of Computer Science and Automation Indian Institute of Science Bangalore, India February 2008
Game Theory Lecture Notes By Y. Narahar Department of Computer Scence and Automaton Indan Insttute of Scence Bangalore, Inda February 2008 Chapter 10: Two Person Zero Sum Games Note: Ths s a only a draft
More information1 GSW Iterative Techniques for y = Ax
1 for y = A I m gong to cheat here. here are a lot of teratve technques that can be used to solve the general case of a set of smultaneous equatons (wrtten n the matr form as y = A), but ths chapter sn
More information