D 2 : Decentralized Training over Decentralized Data

Transcription

1 D : Deceralzed rag over Deceralzed Daa Hal ag Xagru La Mg Ya 3 Ce Zhag 4 J Lu 5 Absrac Whle rag a mache learg model usg mulple workers, each of whch collecs daa from s ow daa source, would be useful whe he daa colleced from dffere workers are uque ad dffere. Irocally, rece aalyss of deceralzed parallel sochasc grade desce D-PSGD reles o he assumpo ha he daa hosed o dffere workers are o oo dffere. I hs paper, we ask he queso: Ca we desg a deceralzed parallel sochasc grade desce algorhm ha s less sesve o he daa varace across workers? I hs paper, we prese D, a ovel deceralzed parallel sochasc grade desce algorhm desged for large daa varace amog workers mprecsely, deceralzed daa. he core of D s a varace reduco exeso of D-PSGD. I mproves he σ covergece rae from O + ζ 3 o /3 σ O where ζ deoes he varace amog daa o dffere workers. As a resul, D s robus o daa varace amog workers. We emprcally evaluaed D o mage classfcao asks, where each worker has access o oly he daa of a lmed se of labels, ad fd ha D sgfcaly ouperforms D-PSGD.. Iroduco rag mache learg models a deceralzed way has araced esve eress recely La e al., 07a; * Equal corbuo Deparme of Compuao Scece, Uversy of Rocheser Deparme of Compuaoal Mahemacs, Scece ad Egeerg, Mchga Sae Uversy 3 Deparme of Mahemacs, Mchga Sae Uversy 4 Deparme of Compuer Scece, EH Zurch 5 ece AI Lab. Correspodece o: Hal ag <hag4@ur.rocheser.edu>, Xagru La <xagru@yadex.com>, Mg Ya <yam@mah.msu.edu>, Ce Zhag <ce.zhag@f.ehz.ch>, J Lu <j.lu.uwsc@gmal.com>. Proceedgs of he 35 h Ieraoal Coferece o Mache Learg, Sockholm, Swede, PMLR 80, 08. Copyrgh 08 by he auhors. Yua e al., 06; Col e al., 06. I he deceralzed seg, here s a se of workers, each of whch collecs daa from dffere daa sources. Isead of sedg all daa o a ceralzed place, hese workers oly commucae wh her eghbors. he goal s o ge a model ha s he same as f all daa are colleced a ceralzed place. Deceralzed learg algorhms are mpora scearos where he ceralzed commucao s expesve or mpossble, or he uderlyg commucao ework has hgh laecy. For deceralzed learg o provde beefs, each user should provde daa ha s somehow uque,.e., he varace of daa colleced from dffere workers are large. However, may rece heorecal resuls La e al., 07a;b; Nedc & Ozdaglar, 009; Yua e al., 06 assume a bouded daa varace across workers whe daa hosed o dffere workers are very dffere, hese approaches coverge slowly, boh emprcally ad heorecally. I hs paper, we am a brgg hs dscrepacy bewee he curre heorecal udersadg ad he requremes from some praccal scearos. I hs paper, we prese D, a ovel deceralzed learg algorhm desged o be robus uder hgh daa varace. D s bul upo deceralzed parallel sochasc grade desce D-PSGD, bu beefs from a addoal varace reduco compoe. I D, each worker sores he sochasc grade ad s local model he prevous erae ad learly combes hem wh he curre sochasc grade ad local model. I resuls a mproved covergece rae over D-PSGD by elmag he daa varao amog workers. I parcular, he covergece rae s mproved from σ O + ζ 3 o O σ /3 where ζ s he daa varao amog all workers, σ s he daa varace wh each worker, s he umber of workers, ad s he umber of eraos. We emprcally show D ca sgfcaly ouperform D-PSGD by rag a mage classfcao model where each worker has access o oly he daa of a lmed se of labels. hroughou hs paper, we cosder he followg deceralzed opmzao: m x R N fx:= {}}{ E ξ D F x; ξ, =:f x

2 Deceralzed rag over Deceralzed Daa where s he umber of workers ad D s he local daa dsrbuo for worker. All workers are coeced hrough a coeced graph. Each worker ca oly exchage formao wh s eghbors. Defos ad oao hroughou hs paper, we use followg oao ad defos: F deoes he Frobeus orm of marces. deoes he l orm for vecors ad he specral orm for marces. f deoes he grade of a fuco f. f deoes he opmal soluo of. λ deoes he h larges egevalue of a marx. x deoes he local model of worker. F x ; ξ deoes a local sochasc grade of worker. = [,,, ] R deoes he all-oe vecor. I order o orgaze he algorhm more clearly, here we defe he cocaeao of all local varables, sochasc grades, ad her averages respecvely: X :=[x,..., x ] R N, X :=X = x, GX; ξ :=[ F x ; ξ,..., F x ; ξ ] R N, GX, ξ :=GX, ξ = fx := f X, fx := f x, F x ; ξ, where ξ s he colleco of radomly sampled daa from all workers. Orgazao hs paper s orgazed as follows: Seco revews relaed work abou he proposed approach; Seco 3 roduces he sae-of-he-ar deceralzed sochasc grade desce mehod ad s covergece rae; Seco 4 roduces he proposed algorhm ad s uo why mproves he sae-of-he-ar approach; Seco 5 provdes he heorecal guaraee; ad Seco 6 valdaes he proposed approaches va emprcal sudy; ad Seco 7 cocludes hs paper.. Relaed work I hs seco, we revew he sochasc grade desce algorhm ad s deceralzed varas, deceralzed algorhms, ad prevous varace reduco echologes. Sochasc grade desce SGD he SGD approahces Ghadm & La, 03; Moules & Bach, 0; Nemrovsk e al., 009 s que powerful for solvg largescale mache learg problems. I acheves a covergece rae of O /. As a mplemeao of SGD, he Ceralzed Parallel Sochasc Grade Desce C-PSGD, has bee wdely used parallel compuao. I C-PSGD, a ceral worker, whose job s o perform he varable updaes, s coeced o may leaf workers ha are used o compue sochasc grades parallel. C-PSGD has bee appled o may deep learg frameworks, such as CNK Sede & Agarwal, 06, MXNe Che e al., 05, ad esorflow Abad e al., 06. he covergece rae of C- PSGD s O, whch shows ha ca acheve lear speedup wh regards o he umber of leaf workers. Deceralzed algorhms Ceralzed algorhms requre a ceral server o commucae wh all oher workers Suresh e al., 07. I coras, deceralzed algorhms work o ay coeced ework ad oly rely o he formao exchage bewee eghbor workers Kashyap e al., 007; Lavae & Murray, 0; Nedc e al., 009. Deceralzed algorhms are especally useful uder a ework wh lmed badwdh or hgh laecy. I s more favorable whe daa prvacy s sesve. hese advaages have led o successful applcaos. he deceralzed approach for mul-ask reforceme learg was suded Omdshafe e al. 07; Mhamd e al. 07. I Col e al. 06, a dual based deceralzed algorhm was proposed o solve he parwse fuco opmzao. Sh e al. 04 ad Mokhar & Rbero 05 aalyzed he deceralzed verso of he ADMM opmzao algorhm. A formao heorec approach was used o aalyze deceralzao Dobbe e al. 07. he deceralzed verso of sub-grade desce was suded Nedc & Ozdaglar 009; Yua e al. 06. Is O/ covergece requres a dmshg sepsze or a cosa sepsze ha depeds o he oal umber of eraos. hs pheomeo happes because of he varace bewee he daa dffere workers, whch we call ouer varace o dffereae from he varace SGD. Recely, here are several deermsc deceralzed opmzao algorhms ha allows a cosa sepsze. For example, EXRA Sh e al., 05a s he frs modfcao of deceralzed grade desce ha coverges uder a cosa sepsze. Laer hs algorhm s exeded for problems wh he sum of smooh ad osmooh fucos a each ode Sh e al., 05b.

3 Deceralzed rag over Deceralzed Daa he algorhm DIGg s proposed Nedć e al. 07, where wo exchages are eeded each erao. However, her sepszes deped o boh he Lpschz cosa of he dffereable fuco ad he ework srucure. NIDS s he frs algorhm ha has a cosa ework depede sepsze L e al., 07. hs algorhm was smulaeously proposed by Yua e al. 07 for he smooh case oly usg a dffere approach. Deceralzed parallel sochasc grade desce D- PSGD he D-PSGD algorhm Nedc & Ozdaglar, 009; Ram e al., 00a;b requres each worker o compue a sochasc grade ad exchage s local model wh eghbors. I Duch e al. 0, a dual averagg based mehod s proposed for solvg he cosraed deceralzed SGD opmzao. I Yua e al. 06, he covergece rae for D-PSGD was aalyzed whe he grade s assumed o be bouded. I La e al. 07, a deceralzed prmal-dual ype mehod was proposed wh a compuaoal complexy of O /ɛ for geeral covex objecves. La e al. 07a proved ha D-PSGD ca adms lear speedup wh respec o he umber of workers wh a smlar covergece rae as C-PSGD. Varace reduco echology here have bee may mehods developed for reducg he varace SGD, cludg SVRG Johso & Zhag, 03, SAGA Defazo e al., 04, SAG Schmd e al., 07, MISO Maral, 05, ad msgd Koečỳ e al., 06. However, mos of hese echologes are desged for ceralzed approaches. he DSA algorhm Mokhar & Rbero, 06 appled he varace reduco smlar o SAGA o srogly covex deceralzed opmzao problems ad proved a lear covergece rae. However, he speedup propery s uclear ad a able of all sochasc grades eed o be sored. 3. Prelmary: deceralzed sochasc grade desce he deceralzed sochasc grade desce La e al., 07a; Zhag e al., 07; Shahrampour & Jadbabae, 07 allows each worker say worker maag s ow local varable x. Durg each erao say, erao, each worker performs he followg seps:. Query s eghbors local varables.. ake weghed average wh s local varable ad s eghbors local varables: x = W + j x j, j= where W j s he, j eleme of he marx W. W j = 0 meas worker ad worker j are o coeced. 3. Perform oe sochasc grade desce sep x + = x + γ F x ; ξ, where ξ represes he daa sampled worker a he erao followg he dsrbuo D. From a global po of vew, he updae rule of D-PSGD ca be vewed as X + = X W γgx ; ξ. I adms he followg rae show heorem. heorem Covergece rae of D-PSGD La e al., 07a. Uder cera assumpos, he oupu of D-PSGD adms he followg equaly γl =0 f0 f γ E fx + D E f X =0 + γl σ + γ L σ λd + 9γ L ς λ D, where ρ reflecs he propery of he ework, D ad D are defed o be D := D := 9γ L ρ, D, 8γ ρ L ad σ ad ς measure he varao wh each worker ad amog all workers respecvely E ξ D F x; ξ f x σ,, x, f x fx ζ,, x. 3 Choosg he opmal seplegh γ = L+σ K + 3 ζ 3 3 we have he followg covergece rae: E fx σ O + 3 ζ = he proposed D algorhm ca mprove he covergece rae by removg he depedece o he global boud of ouer varace ζ.

4 Deceralzed rag over Deceralzed Daa Algorhm he D algorhm : Ipu: Ial po x 0 = 0, sep legh γ > 0, cofuso marx W, ad he oal umber of eraos. : for = 0,,,..., do from he local daa of he h worker. 4: Compue a local sochasc grade based o ξ ad 3: Radomly sample ξ curre varable x 5: f =0 he 6: x + 7: else 8: x + = x = x : F x ; ξ. γ F x ; ξ, x γ F x ; ξ + γ F x ; ξ. 9: ed f 0: Each worker seds x o s eghbors ad akes + he weghed average x + = j= W j x j, + where x j s from he worker j. + : ed for : Oupu: x 4. he D algorhm I D algorhm, each worker say, worker repeas he followg updag rule say, a erao :. Compue a local sochasc grade F x samplg ξ from dsrbuo D ;. Updae he local model x γ F x ; ξ + + γ F x ; ξ ; ξ by x x usg he local models ad sochasc grades boh he h erao ad he h erao. 3. Whe he sychrozao barrer s me, exchage wh eghbors: x + x + = j= W j x j. + From a global po of vew, he updae rule of D ca be vewed as: X + = X X γgx ; ξ + γgx ; ξ W. he complee algorhm s summarzed Algorhm. D esseally rus he sochasc grade desce sep. o udersad he uo of D, le us cosder he mea value X, whch s updaed jus lke he sadard sochasc grade desce: X + = X X γgx ; ξ + γgx ; ξ W, X + =X X γgx ; ξ + γgx ; ξ, or equvalely X + X =X X γgx ; ξ + γgx ; ξ, =X X 0 γ GX ; ξ GX ; ξ k= = γgx ; ξ. X = X 0 γgx 0 ; ξ 0. 4 Why D mproves he D-PSGD? We may oce ha D- PSGD also esseally updaes he form of sochasc grade desce 4. he why D ca mprove D-PSGD? Assume ha X has acheved he opmum X := x wh all local models equal o he opmum x o. he for D-PSGD, he ex updae wll be X + = X γgx ; ξ. I shows ha he covergece whe we approach a soluo s affeced by E[ GX ; ξ F ], whch s bouded by Oσ + ζ, as we ca see from he followg: E[ GX ; ξ F ] =E F x ; ξ + f x + f x E F x ; ξ + f x + f x fx σ + ζ. Nex we apply a smlar aalyss for D by assumg ha boh X ad X have reached he opmal soluo X. he ex updae for D wll be: X + = X γgx ; ξ γgx ; ξ W. I shows ha for D, he covergece whe we approach a soluo reles o he magude of E[ GX ; ξ GX ; ξ F ], whch s bouded by: Oσ,

5 Deceralzed rag over Deceralzed Daa whch ca be see from: E[ GX ; ξ GX ; ξ F =E F x ; ξ f x E σ. F x ; ξ f x 5. heorecal guaraee hs seco provdes he heorecal guaraee for he proposed D algorhm. We frs gve he assumpos requred below. Assumpo. hroughou hs paper, we make he followg commoly used assumpos:. Lpschza grade: All fuco f s are wh L-Lpschza grades.. Bouded varace: Assume bouded varace of sochasc grade wh each worker E ξ D F x; ξ f x σ,, x. 3. Symmerc cofuso marx: he cofuso marx W s symmerc ad sasfes W =. 4. Specral gap: Le he egevalues of W R be λ λ λ. Deoe by for shor λ := max λ = λ. {,,} We assume λ < ad λ > Ialzao: W.l.o.g., assume all local varables are alzed by zero, ha s, X 0 = 0. Exsg deceralzed cosesus algorhms Sh e al., 05b; L e al., 07 use a modfcao of he doubly sochasc marx such ha λ > 0,.e., choose W = W + I/ where W s a doubly sochasc marx. Recely, L & Ya 07 show ha λ > /3 s opmal he covergece of EXRA. However, he opmal λ for NIDS L e al., 07 s ukow. I hs paper, we proved ha 3 s he fmum of λ, ad whe reduces o deermsc case, hs codo s weaker ha ha L e al., 07. hs s mpora, because we acually ca use a W ha performs beer. Gve Assumpo, we have followg covergece guaraee for D : heorem Covergece of Algorhm. Choose he seplegh γ Algorhm o be a cosa sasfyg 4C γ L > 0. Uder Assumpo, we have he followg covergece rae for Algorhm : A f0 + E fx + A E fx f0 f γ + L C γ σ where ζ 0 := = + L γ σ + 6L C γ ζ 0 + 6L C γ 4 L σ + 6L C γ σ, f 0 f0, v :=λ λ λ, { } C := max v, λ, { λ C := max v, λ }, λ λ := 4C γ L, A := 6L C γ, A := Lγ 6L C γ 4 L. By appropraely specfyg he sep legh γ, we reach he followg corollary: Corollary 3. Choose he sep legh γ Algorhm o be γ = 8 C, where C ad C are defed L+6 C L+σ heorem. Uder Assumpo, he followg covergece rae holds E fx σ + + ζ0 + σ =0 + σ + σ, where ζ 0 s defed heorem ad we rea f0 f, L, λ, ad λ as cosas. Noe ha we ca oba eve beer cosas by choosg dffere parameers ad applyg gher equales, however, he ma resul of hs corollary s o show he order of he covergece. We hghlgh a few key observaos from our heorecal resuls he followg. ghess of he covergece rae Seg σ = 0 ad ζ 0 = 0, whch reduces he VR-SGD o a ormal GD 5

6 Deceralzed rag over Deceralzed Daa algorhm, we shall see ha he covergece rae becomes O, whch s exacly he rae of GD. Lear speedup Sce he leadg erm of he covergece rae s O, whch s cosse wh he covergece rae of C-PSGD, hs dcaes ha we would acheve a lear speed up wh respec o he umber of odes. Cosse wh NIDS I NIDS L & Ya, 07, he ζ 0 erm depeds o ζ 0 he covergece rae s O. Whle he correspodg erm D ζ s O 0 +σ, whch dcaes whe our algorhm s cosse wh NIDS because NIDS σ s cosdered o be 0. Superory over D-PSGD Whe compared o D-PSGD, he covergece rae of D oly depeds o ζ 0, ad he correspodg decayg rae s ζ0. Whereas D-PSGD La e al., 07a, we eed o assume a upper boud for he global varace bewee dffere odes daase, ad s fluece ca be compared o σ, he er varace of each ode self. hs meas we ca always acheve a much beer covergece rae ha D-PSGD. 6. Expermes We evaluae he effecveess of D by comparg wh boh ceralzed ad deceralzed SGD algorhms. 6.. Experme Segs We coduc expermes wo segs.. RANSFERLEARNING: We es he case ha each worker has access o a local pre-raed eural ework as feaure exracor, ad we wa o ra a logsc regresso model amog all hese workers. I our experme, we selec he frs 6 classes of ImageNe ad use IcepoV4 as he feaure exracor o exrac 048 feaures for each mage. We coduc daa augmeao ad geerae a blurblack verso for each mage. I oal hs daase coas mages.. LENE: We es he case ha all workers collaboravely ra a eural ework model. We ra a LeNe o he CIFAR0 daase. I oal hs daase coas 50,000 mages of sze 3 3. Oe cavea of rag more rece eural eworks s ha moder archecures ofe have a bach ormalzao layer, whch herely assumes ha he daa dsrbuo s uform across dffere baches, whch s o he case ha we are eresed. I prcple, we could also flow he bach formao hrough he ework a deceralzed way; however, we leave hs as fuure work. By defaul, each worker oly has exclusve access o a subse of classes. For RANSFERLEARNING, we use 6 workers ad each worker has access o oe class; for LENE, we use 5 workers ad each worker has access o wo classes. For comparso, we also cosder a case whe he daases s frs shuffled ad he uformly paroed amog all he workers, we call hs he shuffled case, ad he defaul oe he ushuffled case. We use a rg opology for boh expermes. Parameer ug. For RANSFERLEARNING, we use cosa learg raes ad ue from {0.0, 0.05, 0.05, 0.075, 0.}. For LENE, we use cosa learg rae 0.05 whch s ued from {0.5, 0., 0.05, 0.0} for ceralzed algorhms ad bach sze 8 o each worker. Mercs. I hs paper, we maly focus o he covergece rae of dffere algorhms sead of he wall clock speed. hs s because he mplemeao of D s a mor chage over he sadard D-PSGD algorhm, ad hus hey has almos he same speed o fsh oe epoch of rag, ad boh are o slower ha he ceralzed algorhm. Whe he ework has hgh laecy, f a deceralzed algorhm D or D-PSGD coverges wh a smlar speed as he ceralzed algorhm, ca be up o oe order of magude faser La e al., 07a. However, he covergece rae depedg o he ouer varace s dffere for boh algorhms. 6.. Ushuffled Case varao across workers s maxmzed. Fgure shows he resul. I he ushuffled case, we see ha D-PSGD coverges slower ha he ceralzed case. hs s cosse wh he orgal D-PSGD paper La e al., 07a. O he oher had, D coverges much faser ha D-PSGD, ad acheves almos he same loss as he ceralzed algorhm. For he LeNe case, each worker oly has access o daa of assged wo labels, whch meas he daa varao s very large. he D-PSGD does o coverge wh he gve learg rae Shuffled Case As a say check, Fgure shows he resul of hree dffere algorhms o he shuffled daa. I hs case, he daa varao amog workers s small expecao, hey are draw from he same dsrbuo. We see ha, all sraeges have smlar covergece rae. hs valdae ha D s more effecve for larger daa varao bewee dffere workers. We ca ue he learg rae 50x smaller for D-PSGD o coverge hs case, bu dog so wll make D-PSGD suck a he sarg po for que a log me.

7 Deceralzed rag over Deceralzed Daa Loss Deceralzed D Ceralzed # Epochs a RANSFERLEARNING Loss D Deceralzed Ceralzed # Epochs b LENE Fgure. Covergece of Dffere Dsrbued rag Algorhms Ushuffled Case..5.5 Deceralzed Loss 0.5 Loss D Ceralzed 0 Deceralzed D Ceralzed # Epochs # Epochs a RANSFERLEARNING b LENE Fgure. Covergece of Dffere Dsrbued rag Algorhms Shuffled Case. 7. Cocluso I hs paper, we propose a deceralzed algorhm, amely, D algorhm. D algorhm egraes he D-PSGD algorhm wh he varace reduco echology, by whch we mproves he covergece rae of D-PSGD. he varace reduco echology used hs paper s dffere from he commoly used oes such as SVRG ad SAGA, ha are desged for ceralzed approaches. Expermes valdae he advaage of D over D-PSGD D coverges wh a rae ha s smlar o ceralzed SGD whle D-PSGD does o coverge o a soluo wh a smlar qualy whe he daa varace s large. Whle beg robus o large daa varace amog workers, he same performace beef of D-PSGD over he ceralzed sraegy sll holds for D. Ackowledgemes hs projec s suppored par by NSF CCF7853, NEC fellowshp, IBM faculy award, NSF DMS-6798, Swss NSF NRP , IBM Zurch, Mercedes- Bez Research & Developme Norh Amerca, Oracle

8 Deceralzed rag over Deceralzed Daa Labs, Swsscom, Zurch Isurace, ad Chese Scholarshp Coucl. Refereces Abad, M., Barham, P., Che, J., Che, Z., Davs, A., Dea, J., Dev, M., Ghemawa, S., Irvg, G., Isard, M., e al. esorflow: A sysem for large-scale mache learg. I OSDI, volume 6, pp , 06. Che,., L, M., L, Y., L, M., Wag, N., Wag, M., Xao,., Xu, B., Zhag, C., ad Zhag, Z. Mxe: A flexble ad effce mache learg lbrary for heerogeeous dsrbued sysems. arxv prepr arxv:5.074, 05. Col, I., Belle, A., Salmo, J., ad Clémeço, S. Gossp dual averagg for deceralzed opmzao of parwse fucos. I Ieraoal Coferece o Mache Learg, pp , 06. Defazo, A., Bach, F., ad Lacose-Jule, S. Saga: A fas cremeal grade mehod wh suppor for osrogly covex compose objecves. I Advaces eural formao processg sysems, pp , 04. Dobbe, R., Frdovch-Kel, D., ad oml, C. Fully deceralzed polces for mul-age sysems: A formao heorec approach. I Advaces Neural Iformao Processg Sysems, pp , 07. Duch, J. C., Agarwal, A., ad Wawrgh, M. J. Dual averagg for dsrbued opmzao: Covergece aalyss ad ework scalg. IEEE rasacos o Auomac corol, 573:59 606, 0. Ghadm, S. ad La, G. Sochasc frs- ad zeroh-order mehods for ocovex sochasc programmg. SIAM Joural o Opmzao, 34:34 368, 03. do: 0.37/ Johso, R. ad Zhag,. Accelerag sochasc grade desce usg predcve varace reduco. I Advaces eural formao processg sysems, pp , 03. Kashyap, A., Başar,., ad Srka, R. Quazed cosesus. Auomaca, 437:9 03, 007. Koečỳ, J., Lu, J., Rchárk, P., ad akáč, M. M-bach sem-sochasc grade desce he proxmal seg. IEEE Joural of Seleced opcs Sgal Processg, 0 :4 55, 06. La, G., Lee, S., ad Zhou, Y. Commucao-effce algorhms for deceralzed ad sochasc opmzao Lavae, J. ad Murray, R. M. Quazed cosesus by meas of gossp algorhm. IEEE rasacos o Auomac Corol, 57:9 3, 0. L, Z. ad Ya, M. A prmal-dual algorhm wh opmal sepszes ad s applcao deceralzed cosesus opmzao. arxv prepr arxv: , 07. L, Z., Sh, W., ad Ya, M. A deceralzed proxmalgrade mehod wh ework depede sep-szes ad separaed covergece raes. arxv prepr arxv: , 07. La, X., Zhag, C., Zhag, H., Hseh, C.-J., Zhag, W., ad Lu, J. Ca deceralzed algorhms ouperform ceralzed algorhms? a case sudy for deceralzed parallel sochasc grade desce a. La, X., Zhag, W., Zhag, C., ad Lu, J. Asychroous deceralzed parallel sochasc grade desce. arxv prepr arxv: , 07b. Maral, J. Icremeal majorzao-mmzao opmzao wh applcao o large-scale mache learg. SIAM Joural o Opmzao, 5:89 855, 05. Mhamd, E., Mahd, E., Hedrkx, H., Guerraou, R., ad Maurer, A. D. O. Dyamc safe errupbly for deceralzed mul-age reforceme learg. echcal repor, EPFL, 07. Mokhar, A. ad Rbero, A. Deceralzed double sochasc averagg grade. I Sgals, Sysems ad Compuers, 05 49h Aslomar Coferece o, pp IEEE, 05. Mokhar, A. ad Rbero, A. Dsa: Deceralzed double sochasc averagg grade algorhm. Joural of Mache Learg Research, 76: 35, 06. Moules, E. ad Bach, F. R. No-asympoc aalyss of sochasc approxmao algorhms for mache learg. I Shawe-aylor, J., Zemel, R. S., Barle, P. L., Perera, F., ad Weberger, K. Q. eds., Advaces Neural Iformao Processg Sysems 4, pp Curra Assocaes, Ic., 0. Nedc, A. ad Ozdaglar, A. Dsrbued subgrade mehods for mul-age opmzao. IEEE rasacos o Auomac Corol, 54:48 6, 009. Nedc, A., Olshevsky, A., Ozdaglar, A., ad sskls, J. N. O dsrbued averagg algorhms ad quazao effecs. IEEE rasacos o Auomac Corol, 54: , 009. Nedć, A., Olshevsky, A., ad Rabba, M. G. Nework opology ad commucao-compuao radeoffs deceralzed opmzao. arxv prepr arxv: , 07.

9 Deceralzed rag over Deceralzed Daa Nemrovsk, A., Judsky, A., La, G., ad Shapro, A. Robus sochasc approxmao approach o sochasc programmg. SIAM Joural o Opmzao, 94: , 009. do: 0.37/ Omdshafe, S., Pazs, J., Amao, C., How, J. P., ad Va, J. Deep deceralzed mul-ask mul-age rl uder paral observably. arxv prepr arxv: , 07. Ram, S. S., Nedć, A., ad Veeravall, V. V. Asychroous gossp algorhm for sochasc opmzao: Cosa sepsze aalyss. I Rece Advaces Opmzao ad s Applcaos Egeerg, pp Sprger, 00a. Yua, K., Lg, Q., ad Y, W. O he covergece of deceralzed grade desce. SIAM Joural o Opmzao, 63: , 06. do: 0.37/ Yua, K., Yg, B., Zhao, X., ad Sayed, A. H. Exac dffuso for dsrbued opmzao ad learg par : Algorhm developme. arxv prepr arxv:70.05, 07. Zhag, W., Zhao, P., Zhu, W., Ho, S. C., ad Zhag,. Projeco-free dsrbued ole learg eworks. I Ieraoal Coferece o Mache Learg, pp , 07. Ram, S. S., Nedć, A., ad Veeravall, V. V. Dsrbued sochasc subgrade projeco algorhms for covex opmzao. Joural of opmzao heory ad applcaos, 473:56 545, 00b. Schmd, M., Le Roux, N., ad Bach, F. Mmzg fe sums wh he sochasc average grade. Mahemacal Programmg, 6-:83, 07. Sede, F. ad Agarwal, A. Ck: Mcrosof s ope-source deep-learg oolk. I Proceedgs of he Nd ACM SIGKDD Ieraoal Coferece o Kowledge Dscovery ad Daa Mg, KDD 6, pp , New York, NY, USA, 06. ACM. ISBN do: 0.45/ Shahrampour, S. ad Jadbabae, A. Dsrbued ole opmzao dyamc evromes usg mrror desce. IEEE rasacos o Auomac Corol, 07. Sh, W., Lg, Q., Yua, K., Wu, G., ad Y, W. O he lear covergece of he admm deceralzed cosesus opmzao. IEEE ras. Sgal Processg, 67: , 04. Sh, W., Lg, Q., Wu, G., ad Y, W. Exra: A exac frsorder algorhm for deceralzed cosesus opmzao. SIAM Joural o Opmzao, 5: , 05a. Sh, W., Lg, Q., Wu, G., ad Y, W. A proxmal grade algorhm for deceralzed compose opmzao. IEEE rasacos o Sgal Processg, 63: , 05b. Suresh, A.., Yu, F. X., Kumar, S., ad McMaha, H. B. Dsrbued mea esmao wh lmed commucao. I Precup, D. ad eh, Y. W. eds., Proceedgs of he 34h Ieraoal Coferece o Mache Learg, volume 70 of Proceedgs of Mache Learg Research, pp , Ieraoal Coveo Cere, Sydey, Ausrala, 06 Aug 07. PMLR.

10 Deceralzed rag over Deceralzed Daa Supplemeal Maerals hs suppleme maeral cludes he proofs for heorem. Because he cofuso marx W s symmerc, ca be decomposed as W = P ΛP, where P = v, v,, v s a orhogoal marx,.e., P P = P P = I, ad Λ = dag{λ,..., λ } s a dagoal marx wh dagoal eres beg he egevalues of W he ocreasg order. he applyg he decomposo o he erao from W ad W o W + gves X + = X W X W γgx ; ξ W + γgx ; ξ W X + =X P ΛP X P ΛP γgx ; ξ P ΛP + γgx ; ξ P ΛP. Deoe Y = X P, HX ; ξ = GX ; ξ P, ad use y respecvely. he or he colums of Y ad HX ; ξ, ad h o dcae he -h colum of Y ad HX ; ξ, Y + =Y Λ Y Λ γhx ; ξ Λ + γhx ; ξ Λ, 6 y + =λ y y γh + γh. 7 From he properes of W Assumpo ad he decomposo, we have λ = ad v =,,,. herefore y = X. For all oher egevalues 3 < λ <, he equao 7 shows ha all y explas how he cofuso marx works. Lemma 4. Gve wo o-egave sequeces {a } = ad {b } = ha sasfyg wh ρ [0,, we have Proof. S k := D k := a = = a = = = = S k := D k := a = would decay o zero, whch ρ s b s, 8 a = b s ρ, a ρ = ρ s b s = ρ s b s = r= ρ where he las equaly holds because of 9. = =s ρ r b r = r= ρ s r b s + b r ρ s b s b s. ρ s b s = k s ρ b s =0 = r= = ρ = r= ρ s r b s b r ρ s r b s b s ρ. 9 b s 0

11 Deceralzed rag over Deceralzed Daa Lemma 5. For ay marx X R N, we have X v X v = X F = X P e = X P F = X F where e R wh he -h compoe beg ad all ohers beg 0. Proof. From he defo of he Frobeus orm for a marx, we have X v = X P F = r X P P X = r X X = X F. Sce X v 0, so X v X v = = X F. I he same way, we have X P e = X P = X F F. he resul s proved. Lemma 6. Gve ρ 3, 0 0,, for ay wo sequece {a } =0 ad {b } =0 ha sasfy a 0 =b 0 = 0, a =b, a + =ρa a + b b,, we have u + v + a + = a u v + u s+ v s+ β s, 0, u v where β s = b s b s, u = ρ + ρ ρ, v = ρ ρ ρ. More specfcally, f 0 < ρ <, we have a + s θ = a ρ / s [ + θ] + β s ρ s/ s [ + sθ], 0 where β s =b s b s, θ = arccos ρ. Proof. Whe = 0, he resuls s easy o verfy. Nex we cosder he case. Sce a + = ρa ρa + β,

12 Deceralzed rag over Deceralzed Daa We ca fd u = ρ + ρ ρ, v = ρ ρ ρ, such ha a + ua =a ua v + β. Noe ha u ad v are complex umbers whe 0 < ρ <. ha s wh θ = arccos ρ. Recursvely applyg gves Dvg boh sdes by u +, we oba u = ρe θ, v = ρe θ, a + ua =a ua v + β = a ua v + β v + β =a ua 0 v + β s v s a + u + = a u + u + = a + u u = a u + =a v + β s v s. due o a 0 = 0 a v + β s v s a v + β s v s + u a + v + u a k v k + k= he we mulply boh sdes by u + ad have a + =a u + u a k v k + k= =a u + β s v k s β s v k s v k + u u k= v k =a u + u u k=0 k=s v + =a u u v + u u u + v + =a + u v k= β s v s v u β s v s v u β s v s v k u k due o s v u u s+ v s+ β s. u v Whe ρ 0,, sce u = ρe θ ad v = ρe θ, we have k= s+ v u β s v s a s b k = k=s a s b k he resul s proved. / s [ + θ] a + = a ρ s θ + s/ s [ s + θ] β s ρ. s θ

13 Lemma 7. Uder Assumpo, we have 4C γ L Deceralzed rag over Deceralzed Daa =0 X x C X F + C γ σ + 6C γ 4 L σ + 6C γ 4 L fx, where γ, L, σ, θ, C ad C are defed heorem. Proof. o esmae he dfferece of he local models ad he global mea model, we have X x = X e X = X X F = X P P X v v F 0, 0, 0,, 0 0,, 0,, 0 = X P 0, 0,,, , 0, 0,, F = y, where y s he -h colum of X P. Noe ha we have, from 7, where β = γh = y + = λ y y γh + γh = λ y y + λ β, + γh. For all y ha correspodg o 3 < λ < 0, Lemma 6 shows y u + + =y v + + λ u v β s where u = λ + λ λ ad v = λ λ λ. herefore, we have For u+ v + u v y +, we have Usg 3, we oba u + y u + v + u v + λ u s+ β s = v s+, u v u s+ v s+ u v v + u u v v u v v u v v due o u < v. y + Summg from = 0 o = gves y + =0 = y = y v + λ y β s v s. v + λ =0 = β s. 3 v s.

14 Deoe a = have β s where v = λ λ λ. Deceralzed rag over Deceralzed Daa v s, whch has he same srucure as he sequece Lemma 4. herefore, whe λ < 0, we y = y v + λ v = y v + λ v For all y ha sasfes 0 λ <, from 7 ad Lemma 6, we have where β s he = γh s y + y + s θ = y λ/ s [ + θ ] + λ + γh s ad θ = arccos λ. s θ y y Summg from = 0 o gves y + =0 s θ = = β β, 4 β s λ s/ s [ + sθ ], λ s [ + θ ] + λ β s s [ + sθ ] λ s/ λ + λ y = s θ β s λ s/, y =0 λ + λ = β s λ s/ From Lemma 4, he s θ = λ gves { Deoe C = max β s λ s/ v, y = y = λ } has he same srucure as he sequece Lemma 4, so we have y s θ + λ λ λ β = y λ + λ λ λ = y λ + λ λ λ ad C = max y = =. β β. 5 { } λ v, λ. From 4 ad 5, we have λ λ y C + C β =. 6

15 Deceralzed rag over Deceralzed Daa We ex boud β = E = = β γ E h h =γ E G X ; ξ P e G X ; ξ P e γ = E G X ; ξ P e G X ; ξ P e =γ E G X ; ξ P G X ; ξ P F =γ E G X ; ξ G X ; ξ F due o Lemma 5 =γ E F x ; ξ F x ; ξ =γ F E =3γ E F + 3γ f 6γ σ + 3γ 6γ σ + 3γ x x x ; ξ f x F x ; ξ f x + f ; ξ f x f x E f L E x x + 3γ f x x =6γ σ + 3γ L E Y P e Y P e F x ; ξ f x x f x =6γ σ + 3γ L E Y P Y P F =6γ σ + 3γ L E Y Y F due o Lemma 5 =6γ σ + 3γ L E y y. 7 Combg 6 ad 7, we have = = y C Y F + C β = = C Y F + C 6γ σ + 3γ L = C Y F + C γ σ + 6C γ L E y E y = y y. 8

16 he ex sep s o boud E y y. Because Deceralzed rag over Deceralzed Daa y = X P e = X v = X = X, wha we eed o boud s E X + X. From 4, we have X + = X γg. herefore E X+ X = γ E G = γ E G fx + γ fx γ σ ad we have he follow boud for E y y Combg 8 ad 9 we ge 4C γ L = = = = y y E ogeher wh ad X 0 = 0, we have 4C γ L = : y + y + γ fx, γ σ + γ fx. 9 C Y F + C γ σ + 6C γ 4 L σ + 6C γ 4 L fx + 6C γ L E = = y y C Y F + C γ σ + 6C γ 4 L σ + 6C γ 4 L fx y + 6C γ L E + = = y C Y F + C γ σ + 6C γ 4 L σ + 6C γ 4 L fx y + 6C γ L E + = = y = = = due o y 0 = 0 C Y F + C γ σ + 6C γ 4 L σ + 6C γ 4 L fx + 4C γ L E = = y, C Y F + C γ σ + 6C γ 4 L σ + 6C γ 4 L fx. X x C Y F + C γ σ + 6C γ 4 L σ + 6C γ 4 L fx due o X F = Y F Acually, whe λ 3, we have v, he y ad C X F + C γ σ + 6C γ 4 L σ + 6C γ 4 L fx. = X x. he algorhm would fal o coverge hs suao, ad hs s why /3 s he fmum of λ. = = = =

17 Lemma 8. Followg he Assumpo, we have EfX + EfX γ E fx Proof. From 4, we have Deceralzed rag over Deceralzed Daa γ Lγ E fx + γ E fx fx + Lγ σ. X + = X γ GX ; ξ. From em of Assumpo, we kow ha f has a L-Lpschz couous grade. So, we have EfX + EfX + E fx, γ GX ; ξ + L E γ GX ; ξ =EfX + E fx, γ E ξ GX ; ξ + Lγ E GX ; ξ =EfX γ E fx, fx + Lγ E GX ; ξ fx + fx =EfX γ E fx, fx + Lγ E GX ; ξ fx + Lγ E fx + Lγ E E ξ GX ; ξ fx, fx =EfX γ E fx, fx + Lγ E GX ; ξ fx + Lγ E fx =EfX γ E fx, fx + Lγ E F x ; ξ f x + Lγ E fx =EfX γ E fx, fx + Lγ + E ξ F x E E F x ; ξ f x ; ξ f x, E ξ F x ; ξ f x EfX γ E fx, fx + Lγ σ + Lγ E fx + Lγ E fx =EfX γ E fx γ E fx + γ E fx fx + Lγ E fx + Lγ σ due o a, b = a + b a b =EfX γ E fx γ Lγ E fx + γ E fx fx + Lγ σ, 0 whch complees he proof. Proof o heorem Proof. We frs esmae he upper boud for E fx fx : E fx fx = E L f X f x E f X f x E X x.

18 Combg 0 Lemma 8 ad yelds Seg γ = γ, we oba Deceralzed rag over Deceralzed Daa γ E fx + E fx + Lγ E fx γ From Lemma 7, we have γ Lγ E fx EfX EfX + + γ E fx fx + Lγ EfX EfX + + L γ X x + Lγ σ. 4C γ L =0 σ EfX f EfX + f + L X x C X F + C γ σ + 6C γ 4 L σ + 6C γ 4 L fx, If γ s o oo large ha sasfes 4C γ L > 0, he deoe = 4C γ L, we would have =0 X x C X F + C γ σ Summarzg boh sdes of ad applyg 3 yelds I mples E fx + Lγ E fx =0 EfX 0 f γ f0 f γ =0 + 6L C γ 4 L + L =0 + L γ σ + L C fx. = = + 6C γ 4 L σ E X x + L γ σ X F + L C γ σ E fx + Lγ 6L C γ 4 L E fx f0 f γ = f0 f γ + L γ σ + L C + L γ σ + L C γ X F + L C γ σ G0; ξ 0 F + L C γ σ + 6C γ 4 L X x + Lγ σ. fx. 3 = + 6L C γ 4 L σ + 6L C γ 4 L σ However, G0; ξ 0 F ca be expaded as: G0, ξ 0 F = F 0, ξ f 0 + f 0 f0 + f0 + 6L C γ 4 L σ. 4 3σ + 3ζ f0, 5

19 Deceralzed rag over Deceralzed Daa where ζ 0 = f 0 f0 dcaes he dfferece bewee dffere workers daase a he sar po. Combg 4 ad 5, he we have he we have Deoe becomes 6L C γ f0 f γ E fx + Lγ 6L C γ 4 L E fx =0 f0 f γ + L γ σ + L C γ σ + 6L C γ 4 L σ + 6L C γ σ + 6L C γ ζ 0 + 6L C γ f0. f0 + = + L γ σ + L C γ σ E fx + A = 6L C γ Lγ 6L C γ 4 L E fx + 6L C γ 4 L σ + 6L C γ σ + 6L C γ ζ 0. A = Lγ 6L C γ 4 L, A f0 + E fx + A E fx f0 f γ I complees he proof. = + L γ σ + L C γ σ + 6L C γ 4 L σ + 6L C γ σ + 6L C γ ζ 0. Proof o Corollary 3 Proof. From he value of γ, we oba herefore C γ L 64, C γ L 36. = 4C γ L, A = 6L C γ, A = Lγ 6L C γ 4 L > 0, γ L + σ, γ 4 L 4 + σ 4.

20 Deceralzed rag over Deceralzed Daa he we ca remove he fx ad f0 o he lef had sde of 5 heorem, ad 5 becomes whch complees he proof. E fx 4f0 f L8 C + 6 C =0 + 4f0 f σ + Lσ + 48L C σ L + σ + 4L4 σ C L 4 + σ 4 + 4L C σ L + σ + 4L C ζ 0 L + σ,