AN INCREMENTAL QUASI-NEWTON METHOD WITH A LOCAL SUPERLINEAR CONVERGENCE RATE. Aryan Mokhtari Mark Eisen Alejandro Ribeiro

Transcription

1 AN INCREMENTAL QUASI-NEWTON METHOD WITH A LOCAL SUPERLINEAR CONVERGENCE RATE Arya Mokhar Mark Ese Alejadro Rbero Deparme of Elecrcal ad Sysems Egeerg, Uversy of Pesylvaa ABSTRACT We prese a cremeal Broyde-Flecher-Goldfarb-Shao (BFGS) mehod as a quas-newo algorhm wh a cyclcally erave updae scheme for solvg large-scale opmzao problems. The proposed cremeal quas-newo () algorhm reduces compuaoal cos relave o radoal quas-newo mehods by resrcg he updae o a sgle fuco per erao ad relave o cremeal secod-order mehods by removg he eed o compue he verse of he Hessa. A local superlear covergece rae s esablshed ad a srog mproveme s show over frs order mehods umercally for a se of commo large-scale opmzao problems. Idex Terms Sochasc opmzao, quas-newo, cremeal mehod, superlear covergece. INTRODUCTION We sudy a large scale opmzao problem where we seek o mmze a objecve fuco ha s a aggregao of may compoe objecve fucos. To be more precse, cosder a varable x R p ad a fuco f whch s defed as he average of srogly covex fucos labelled f : R p R for =,...,. Our goal s o fd he opmal po x as he soluo o he problem x = argm f(x) := argm x R p x R p f (x), () a compuaoally effce maer eve whe s large. Problems of hs form arse mache learg [ 4], corol problems [5 7], ad wreless commucao [8 0]. Much of he heory ha has bee developed o solve () s ceered o he use of erave desce mehods. For large scale mache learg ad opmzao segs, ca be very large ad he full grade s ofe oo cosly o compue a each erao. As a alerave, sochasc mehods seek o fd a soluo o () whle compug oly a subse of he oal grades a each erao, hus sgfcaly reducg he compuaoal burde. The smples verso of a sochasc algorhm s sochasc grade desce (SGD), whch a each me sep draws a radom dex ad performs a desce usg he grade of oly a sgle fuco [2]. More sophscaed sochasc frs order mehods use varace-reduced grades (e.g. SVRG []) or averagg grades (e.g. [2, 3], [4]) o reduce compuao cos whle reag a lear covergece rae. Movg beyod frs order formao, here have bee sochasc quas-newo mehods o approxmae Hessa formao [5 9]. All of hese sochasc quas-newo mehods reduce compuaoal cos of quas-newo mehods by updag oly a radomly chose sgle or small subse of grades a each erao. However, hey are o able o recover he superlear covergece rae of quas-newo mehods [20 22]. Aleravely o sochasc mehods are cremeal mehods, whch deermscally erae hrough all compoe fucos a cyclc fash- Suppored by NSF CAREER CCF ad ONR N o, aga oly compug a sgle grade or Hessa per erao. Icremeal mehods have bee used wh boh aggregaed grades [23, 24] ad secod-order Hessa formao [25, 26]. The cremeal Newo mehod (NIM) [26] s he oly cremeal mehod show o have a superlear covergece rae; however, he Hessa fuco s o always avalable or compuaoally feasble. Moreover, he mplemeao of NIM requres compuao of he cremeal aggregaed Hessa verse whch has he compuaoal complexy of he order O(p 3 ). We propose a cremeal vara of BFGS whch acheves a greaer local covergece rae ha ha of frs order cremeal or sochasc mehods whle reducg he compuaoal cos per erao o O(p 2 ) by usg a cremeal aggregaed approxmao of he Hessa. We beg wh he paper by roducg he well-kow BFGS mehod ad s updae for solvg () (Seco 2). Whle cremeal mehods have bee used frs order mehods, hey acheve oly a lear covergece rae. We roduce a cremeal quas-newo () mehod ha requres oly compug a sgle grade ad Hessa approxmao per erao (Seco 3). The proposed mehod uses a approxmao of secod-order formao ha requres less cos ha compug he Hessa verse drecly whle maag s superor aalycal ad umercal performace. The local superlear covergece of s esablshed (Seco 4) ad performace advaages relave o frs order sochasc ad cremeal mehods are evaluaed umercally (Seco 5). Proofs for resuls preseed are foud [27]. 2. BFGS QUASI-NEWTON METHOD Cosder he problem () for a relavely large. I a coveoal opmzao seg, ca be solved usg a desce mehod, such as grade desce or Newo s mehod, whch eavley updaes a varable x for = 0,,... usg he geeral recursve expresso x + = x + η d, (2) where η s a scalar sep sze ad d s he desce dreco a me defed as eher d := f(x ) or d := ( 2 f(x )) f(x ) for grade desce ad Newo s mehod, respecvely. I boh cases, eraos x coverge o he opmal soluo x. Grade desce, however, usg oly he frs-order formao coaed he grade coverges a a lear rae whle Newo s mehod, usg secod-order formao he Hessa, coverges a a sgfcaly faser quadrac rae [28]. I he case ha he Hessa formao requred Newo s mehod s eher uavalable or oo cosly o evaluae, quas-newo mehods have bee developed o approxmae secod-order formao o mprove upo he covergece rae of frs order mehods [20]. Quasewo mehods perform a updae usg a desce dreco of he form d := (B ) f(x ), where B s a approxmao of he Hessa 2 f(x ). There are dffere approaches o approxmae he Hessa, wh he mos popular beg he mehod of Broyde-Flecher-Goldfarb- Shao (BFGS) [20 22]. I BFGS, wo auxlary varables are defed o capure dfferece varables ad grades of successve eraos,.e. s := x + x, y := f(x + ) f(x ). (3)

2 The, gve he varable varao s ad grade varao y, he Hessa approxmao s updaed usg he recursve equao B + = B + y y T y T s B s s T B s T B s. (4) The BFGS mehod s popular o oly for s srog umercal performace relave o he grade desce mehod, bu also because s show o exhb a superlear covergece rae [20], hereby provdg a heorecal guaraee of superor performace. I fac, ca be show ha, for he BFGS updae he equao (B 2 f(x ))s = 0 (5) s kow as he Des-Moré codo, whch s boh ecessary ad suffce for superlear covergece [22], s sasfed. Ths resul soldfes quas-newo mehods as a srog alerave o frs order mehods whe exac secod-order formao s uavalable. 3. INCREMENTAL QUASI-NEWTON METHOD We roduce a ovel cremeal quas-newo () algorhm, whch he updaed grade formao of oly a sgle fuco f s updaed a each erao. Cosder ha each compoe Hessa 2 f (x ) s approxmaed wh B ad defe copes of he varable x, oaed as z, z 2,..., z, each correspodg o a dffere fuco f. I parcular, z shows he vecor x a he las me before sep ha fuco f s chose o updae desce dreco. We defe he updae of Hessa approxmao B usg he radoal BFGS updae (3)-(4), bu sead usg he -h copes z ad z + place of x ad x +. We redefe he varable ad grade dffereces assocaed wh he fuco f as s := z + z y := f (z + ) f (z ), (6) Noe ha f a sep + ad τ, where τ 0, he fuco f s chose for updag he grade formao, he we have z + = x + ad z = x τ. I parcular, f we use a cyclc scheme for choosg he fucos we have τ =,.e., z = x +. Ths observao shows ha he varable ad grade varao defed (6) ca be wre as s := x + x τ ad y := f (x + ) f (x τ ). Therefore, hese defos are dffere from he oes for classc BFGS (3)-(4). Lkewse, he Hessa approxmao of he s dffere from BFGS usg s ad y sead of s ad y. Thus, he Hessa approxmao B correspodg o he fuco f s updaed as B + = B + y y T y T s B s T B s B s s T. (7) To derve he full varable updae, we use a approach smlar o ha used he Newo-ype Icremeal Mehod (NIM) [26]. Cosder he secod-order approxmao of he fuco f (x) ceered aroud z as f (x) f (z ) + f (z ) T (x z ) + 2 (x z ) T 2 f (z )(x z ). (8) As radoal quas-newo mehods, we replace he -h Hessa 2 f (z ) by B. Usg he approxmao marces place of Hessas, he objecve fuco f(x) = (/) f(x) ca be approxmaed wh f(x) f (z ) + f (z ) T (x z ) + 2 (x z ) T B (x z ). (9) Cosderg he approxmao (9), we approxmae he opmal argume of f(x) by he opmal argume of he fuco (9). Thus, we defe ˆx + := argm{(/) f(z )+(/) f(z ) T (x z ) + (/2) (x z ) T B (x z )} as he auxalary varable a erao +. Solvg hs quadrac programmg yelds he followg closed-form updae for he varable ˆx + ˆx + = ( B ) [ B z ] f (z ). (0) As radoal desce mehods, we roduce a sep sze 0 η ad defe he varable x + correspodg o sep + as he weghed average of he prevous erae x ad he evaluaed auxlary varable ˆx +,.e., x + = η ˆx + + ( η )x. () From here, remas o show how he dvdual varable copes z ad Hessa approxmaos B are updaed. We use a cyclc updae scheme o remove he eed o compue all grades ad Hessa approxmaos a each erao. Therefore, f we defe as he dex of he fuco chose a sep, s updaed by cyclcally erag hrough all dces order,.e., = {, 2,...,,, 2,...}. A me, we updae he varable copes as z + = x +, z + = z for all. (2) Thus oly a sgle varable z + s chaged a me whle all ohers are kep he same. Noe ha he varable dffereces (6) wll be ull uless = ad hus oly oe approxmao B wll chage a each me. To see ha hs updag scheme requres evaluao of oly a sgle grade ad Hessa approxmao marx per erao, cosder wrg he updae (7) as ˆx + = ( B ) ( u g ), (3) where we defe as he aggregae Hessa approxmao B := he aggregae Hessa-varable produc u := B z, ad he aggregae grade g := f(z ). The, gve ha a sep oly a sgle dex s updaed, we ca evaluae hese varables for sep + as B, B + = B + ( B + B ), (4) u + = u + ( B + z + B z ), (5) g + = g + ( f (z + ) f (z ) ). (6) Thus, oly a sgle B + ad f (z + ) eed o be compued a each erao, sgfcaly reducg he compuao burde of he algorhm relave o radoal quas-newo mehods. Alhough he updae of he marx B (7) reduces compuaoal complexy of he mehod, he updae (3) requres compuao of he verse marx ( B ) whch has a expesve compuaoal complexy of he order O(p 3 ). Ths exra compuao cos ca be avoded usg he fac ha he updae (7) ca be smplfed as B + = B + y y T y T s B s s T B. (7) s T B s To derve he expresso (7) we have subsued he dfferece B + B by s rak wo expresso (7). By applyg he Sherma- Morrso formula wce o he updae (7) we ca drecly compue he verse marx ( B + ) as ( B + ) = U + U (B s )(B s ) T U s T B s (B s ) T U (B s ), (8)

3 Algorhm Icremeal Quas-Newo () mehod Requre: x 0,{ f (x 0 )}, {B 0 }, η : Se z 0 = = z 0 = x 0 2: Se ( B 0 ) = ( B0 ), u 0 = B0 x 0, g 0 f(x0 ) = 3: for = 0,, 2,... do 4: Se = ( mod ) + 5: Compue ˆx + = ( B ) ( u g ) [cf. (3)] 6: Updae x + = η ˆx + + ( η )x [cf. ()] 7: Compue s +, y + [cf. (6)], ad B + [cf. (7)] 8: Se z + = x +, ad z + = z for [cf. (2)] 9: Updae u + [cf. (5)], g + [cf. (6)], ad ( B + ) [cf. (8)] 0: ed for where he marx U ca be evaluaed as U = ( B ) ( B ) y yt ( B ) yt s + yt ( B. (9) ) y Noe ha he compuaos (8) ad (9) have compuaoal complexy of he order O(p 2 ) whch s sgfcaly lower ha he O(p 3 ) cos of compug he verse drecly. The complee algorhm s ouled Algorhm. Begg wh al varable x 0 ad grade ad Hessa esmaes f (x 0 ) ad B 0 for all, each varable copy z 0 s se o x 0 Sep ad al values are se for u 0, g 0 ad ( B 0 ) Sep 2. For all, Sep 4 he dex of he ex fuco o updae s seleced cyclcally. The, ˆx + ad he ex varable x + are compued usg (3) ad () Seps 5 ad 6, are updaed Sep 7. I Sep 8, he ew varable z + s updaed as x +, keepg oher z + he same. Fally, he BFGS varables are used Sep 9 o updae u +, g +, ad ( B + ). We proceed o aalyze he local covergece properes of he mehod. respecvely. The ew BFGS varables s +, y +, ad B + 4. CONVERGENCE ANALYSIS We make wo assumpos our aalyss of, boh of whch are sadard for secod-order opmzao mehods. The frs assumpo esablshes boh srog covexy ad Lpschz couous grades for each compoe fuco f. Assumpo There exs posve cosas 0 < µ L such ha, for all {,..., } ad x, x R p, µ x x f (x) f ( x) L x x. (20) Thus, each fuco f s srogly covex wh parameer µ ad has Lpschz couous grades wh parameer L. We oe ha whle some mache learg applcaos, he loss fucos used are o srogly covex, e.g. logsc regresso, hey ca ypcally be made srogly covex by addg a regularzao erm o he objecve fuco. Our secod assumpo s ha he compoe Hessas are also Lpschz couous. Ths assumpo s commoly made o prove quadrac covergece of Newo s mehod [28] ad superlear covergece of quas-newo mehods [20 22]. Assumpo 2 There exss posve cosa 0 < L such ha, for all, 2 f (x) 2 f (y) L x y. (2) We proceed our aalyss of he covergece properes of, frs by esablshg a local lear covergece rae, he demosrag some properes of he Hessa approxmaos ad fally by showg ha a mproved superlear covergece rae of he sequece of resduals ca be obaed. We cosder he sepsze η = our aalyss sce we focus o local covergece of he proposed mehod. To sar he aalyss, frs he followg lemma we show ha he resdual x + x s bouded above by lear ad quadrac erms of he average error of he las eraes. Lemma Cosder he proposed Icremeal Quas-Newo mehod () (6)-(8). If he codos Assumpo hold, he he sequece of eraes geeraed by sasfes he equaly x + x (22) LΓ z x 2 + Γ ( B 2 f (x ) ) ( z x ), where Γ := ((/) B ). Lemma shows ha he resdual x + x s upper bouded by a sum of quadrac ad lear erms of he las resduals. Ths ca eveually lead o a superlear covergece rae by esablshg he lear erm coverges o zero a a fas rae, leavg us wh a upper boud of quadrac erms oly. Frs, however, we esablsh a local lear covergece rae he proceedg heorem. Theorem Cosder he proposed Icremeal Quas-Newo mehod () (6)-(8) ad assume ha Assumpo holds. The, for each r (0, ), here are posve cosas ɛ(r) ad δ(r) such ha for x 0 x ɛ(r) ad B 0 2 f (x ) δ(r), for all =,...,, he sequece of eraes geeraed by sasfes x x r +[ ] + x 0 x, (23) where [ ] + refers o he floor fuco. Moreover, he orms B ad (B ) are uformly bouded. From he resul Theorem ha he orms B ad (B ) are uformly bouded, we oba ha Γ s uformly bouded. Moreover, he resul (23) shows ha he sequece of eraes x coverges learly o he opmal argume. As a resul of hs lear covergece we oba ha he sequece of x + x s summable for all,.e., =0 x+ x <. The summably codo s ecessary o show ha he eraes of BFGS mehod sasfy he he Des-Moré codo (5). Lkewse, he followg proposo, we use he codo =0 x+ x < o show ha a smlar resul holds for he eraes of he mehod. Proposo Cosder he proposed Icremeal Quas-Newo mehod () (6)-(8). Suppose ha he codos Theorem are vald ad Assumpos ad 2 hold. The, for all =,..., we have (B 2 f (x ))(z + z ) z + z = 0. (24) Proposo s a ecessary sep owards a superlear covergece resul because, smlarly o he radoal BFGS aalyss, shows ha our Hessa approxmaos do coverge o he Hessa over me alog he dreco pog owards he opmal po, hus gvg us a mehod ha s locally close o Newo s mehod. I he followg lemma, we use he resul () o show ha he oquadrac erms (B 2 f (x ))(z x ) (22) coverge o zero faser ha z x. Lemma 2 Cosder he proposed Icremeal Quas-Newo mehod () (6)-(8). Suppose ha he codos Theorem are vald ad Assumpos ad 2 hold. The, he followg holds for all =,..., (B 2 f (x ))(z x ) z = 0. (25) x

4 Normalzed Error Normalzed Error Norm of Grade Number of E,ecve Passes Number of E,ecve Passes Number of E,ecve Passes Fg. : Covergece resuls of proposed mehod comparso o,, ad. I (a) he lef mage, we prese a sample covergece pah of he ormalzed error o he quadrac program wh a small codo umber. I (b) he ceer mage, we show he covergece pah for he quadrac program wh a large codo umber. I (c) he rgh mage, we prese a sample covergece pah for he logsc regresso problem. I all cases, provdes sgfca mproveme over frs order mehods, wh he dfferece creasg for larger codo umber. The resul Lemma 2 ca hus be used cojuco wh Lemma o show ha he resdual x + x s bouded by a sum of quadrac erms of prevous resduals ad a erm ha coverges o zero a a fas rae. Ths resul leads us o he ma resul, amely he local superlear covergece of he sequece of resduals, saed he followg heorem. Theorem 2 Cosder he proposed Icremeal Quas-Newo mehod () (6)-(8). Suppose ha he codos Theorem are vald ad Assumpos ad 2 hold. The, he sequece of resduals x x coverges o 0 a a superlear rae, x + x = 0. (26) x x The resul Theorem 2 shows ha he sequece of eraes x geeraed by he mehod coverges superlealy a local eghborhood of he opmal argume x. 5. NUMERICAL RESULTS We frs provde umercal expermes for he applcao of solvg he leas square problem, or equvalely a quadrac program, comparso agas sochasc ad cremeal frs order mehods, amely [2], [4], ad [23]. Cosder he problem x := argm x R p 2 xt A x + b T x, (27) where A R p p s a posve defe marx ad b R p s a radom vecor for all. We selec he marces A := dag{a } radomly for boh small (.e. 0 2 ) ad large (.e. 0 4 ) codo umbers whle b s chose uformly ad radomly from he box [0, 0 3 ] p. We se he varable dmeso p = 0 ad umber of fucos = 000. As we are focusg o local covergece, we use a cosa sep sze of η = for he proposed mehod whle choosg he larges sep sze allowable by he oher mehods o coverge. I he lef mage of Fgure we prese a represeave smulao of he covergece pah of he ormalzed error x x / x 0 x for he quadrac program wh small codo umber. Sep szes of η = 5 0 5, η = 0 4 ad η = 0 6 were used for,, ad, respecvely. These sepszes are ued o compare he bes performace of hese mehods wh. The proposed mehod reaches a error of 0 0 afer 0 passes hrough he daa whle acheves he same error of 0 5 afer 30 passes ( ad do o reach 0 5 afer 40 passes). I he ceer mage, we repea he same smulao bu wh larger codo umber ( usg η = whle ohers rema he same). Observe ha whle he performace of does o degrade wh larger codo umber, he frs order mehods all suffer large degradao.,, ad reach afer 40 passes a ormalzed error of , , ad 9.6 0, respecvely. I ca be see ha sgfcaly ouperforms he frs order mehod for boh codo umber szes, wh he ouperformace creasg for larger codo umber. Ths s expeced as frs order mehods ofe do o perform well for ll codoed problems. 5.. Logsc regresso We evaluae he performace of o a logsc regresso problem of praccal eres. A logsc regresso lears a lear classfer x ha ca predc he label of a daa po v {, } gve a feaure vecor u R p. I parcular, we use ad he frs order mehods o classfy wo dgs from he MNIST hadwre dg daabase [29]. We evaluae for a se of rag samples he probably of a label v = gve a mage vecor u as P (v = u) = /( + exp( u T x)). The classfer x s chose o he vecor ha maxmzes ha maxmzes he log lkelhood over all samples. Gve mages u wh assocaed labels v, he opmzao problem ca be wre as x := argm x R p λ 2 x 2 + log[ + exp( v u T x)], (28) where he frs erm s a regularzao erm paramerzed by λ 0. For our smulaos we selec from he MNIST daase = 000 mages wh dmeso p = 784 labelled as oe of he dgs 0 or 8. The regularzao parameer s fxed o be λ = / ad sep sze of η = 0.0 was used for,, ad. The covergece pahs of he orm of he grade for all mehods us show he rgh mage Fgure. Observe ha aga ouperforms he oher sochasc ad cremeal mehods. Afer 60 passes hrough he daa, reaches a grade magude of , whle he sroges performg frs order mehod () reaches oly a magude of Addoally, observe ha whle he frs order mehods beg o level ou afer 60 passes, he mehod coues o desced. Ths demosraes he effecveess of o a praccal mache learg problem wh real world daa. 6. CONCLUSIONS We preseed a cremeal quas-newo BFGS mehod o solve a large scale opmzao problem, whch a aggregae cos fuco s mmzed whle compug oly a sgle grade ad Hessa approxmao per erao. The cremeal quas-newo () mehod eravely updaes approxmaos o he Hessa verse o acheve a performace comparable o ha of cremeal Newo mehods. Aalycal resuls were esablshed o show a local superlear covergece rae ad superor umercal performace over frs order sochasc ad erave mehods for wo commo mache learg problems.

5 7. REFERENCES [] L. Boou ad Y. L. Cu, O-le learg for very large daases, Appled Sochasc Models Busess ad Idusry, vol. 2. pp. 37-5, [2] L. Boou, Large-scale mache learg wh sochasc grade desce, I Proceedgs of COMPSTAT 200, pp , Physca-Verlag HD, 200. [3] S. Shalev-Shwarz ad N. Srebro, SVM opmzao: verse depedece o rag se sze, I Proceedgs of he 25h eraoal coferece o Mache learg. pp , ACM [4] V. Cevher, S. Becker, ad M. Schmd, Covex opmzao for bg daa: Scalable, radomzed, ad parallel algorhms for bg daa aalycs, IEEE Sgal Processg Magaze, vol. 3, o. 5, pp , 204. [5] F. Bullo, J. Corés, ad S. Marez, Dsrbued corol of roboc eworks: a mahemacal approach o moo coordao algorhms. Prceo Uversy Press, [6] Y. Cao, W. Yu, W. Re, ad G. Che, A overvew of rece progress he sudy of dsrbued mul-age coordao, IEEE Trasacos o Idusral Iformacs, vol. 9, pp , 203. [7] C. G. Lopes ad A. H. Sayed, Dffuso leas-mea squares over adapve eworks: Formulao ad performace aalyss, Sgal Processg, IEEE Trasacos o, vol. 56, o. 7, pp , [8] I. D. Schzas, A. Rbero, ad G. B. Gaaks, Cosesus ad hoc WSNs wh osy lks-par : Dsrbued esmao of deermsc sgals, Sgal Processg, IEEE Trasacos o, vol. 56, o., pp , [9] A. Rbero, Ergodc sochasc opmzao algorhms for wreless commucao ad eworkg, Sgal Processg, IEEE Trasacos o, vol. 58, o. 2, pp , 200. [0], Opmal resource allocao wreless commucao ad eworkg, EURASIP Joural o Wreless Commucaos ad Neworkg, vol. 202, o., pp. 9, 202. [] R. Johso ad T. Zhag, Accelerag sochasc grade desce usg predcve varace reduco, Advaces Neural Iformao Processg Sysems, 203, pp [2] N. L. Roux, M. Schmd, ad F. R. Bach, A sochasc grade mehod wh a expoeal covergece rae for fe rag ses, Advaces Neural Iformao Processg Sysems, 202, pp [3] M. Schmd, N. Le Roux, ad F. Bach, Mmzg fe sums wh he sochasc average grade, Mahemacal Programmg, pp. 30, 206. [Ole]. Avalable: hp://dx.do.org/0.007/ s [4] A. Defazo, F. Bach, ad S. Lacose-Jule, : A fas cremeal grade mehod wh suppor for o-srogly covex compose objecves, Advaces Neural Iformao Processg Sysems, 204, pp [5] N. N. Schraudolph, J. Yu, S. Güer e al., A sochasc quas- Newo mehod for ole covex opmzao. AISTATS, vol. 7, 2007, pp [6] A. Mokhar ad A. Rbero, RES: Regularzed sochasc BFGS algorhm, Sgal Processg, IEEE Trasacos o, vol. 62, o. 23, pp , 204. [7], Global covergece of ole ed memory BFGS, Joural of Mache Learg Research, vol. 6, pp , 205. [8] P. Morz, R. Nshhara, ad M. I. Jorda, A learly-coverge sochasc L-BFGS algorhm, arxv prepr arxv: , 205. [9] R. M. Gower, D. Goldfarb, ad P. Rchárk, Sochasc block BFGS: Squeezg more curvaure ou of daa, arxv prepr arxv: , 206. [20] C. G. Broyde, J. E. D. Jr., Wag, ad J. J. More, O he local ad superlear covergece of quas-newo mehods, IMA J. Appl. Mah, vol. 2, o. 3, pp , Jue 973. [2] M. J. D. Powell, Some global covergece properes of a varable merc algorhm for mmzao whou exac le search, 2d ed. Lodo, UK: Academc Press, 97. [22] J. J. E. Des ad J. J. More, A characerzao of super lear covergece ad s applcao o quas-newo mehods, Mahemacs of compuao, vol. 28, o. 26, pp , 974. [23] D. Bla, A. O. Hero, ad H. Gauchma, A coverge cremeal grade mehod wh a cosa sep sze, SIAM Joural o Opmzao, vol. 8, o., pp. 29 5, [24] M. Gurbuzbalaba, A. Ozdaglar, ad P. Parrlo, O he covergece rae of cremeal aggregaed grade algorhms, arxv prepr arxv: , 205. [25] M. Gürbüzbalaba, A. Ozdaglar, ad P. Parrlo, A globally coverge cremeal Newo mehod, Mahemacal Programmg, vol. 5, o., pp , 205. [26] A. Rodomaov ad D. Kropoov, A superlearly-coverge proxmal ewo-ype mehod for he opmzao of fe sums, Proceedgs of The 33rd Ieraoal Coferece o Mache Learg, 206, pp [27] A. Mokhar, M. Ese, ad A. Rbero, A cremeal quas- Newo mehod wh a local superlear covergece rae, 206, avalable a hp:// maese/wk/cr-bfgsdraf.pdf. [28] S. Boyd ad L. Vadeberghe, Covex opmzao. Cambrdge uversy press, [29] Y. LeCu, C. Cores, ad C. J. Burges, MNIST hadwre dg daabase, AT&T Labs [Ole]. Avalable: hp://ya. lecu. com/exdb/ms, 200.