arxiv: v1 [stat.ml] 21 Mar 2017

Transcription

1 Sochasc Prmal Dual Coordae Mehod wh No-Uform Samplg Based o Opmaly Volaos Asush Shbagak shbagak.a.mllab.@gmal.com Deparme of Scefc ad Egeerg Smulao, Nagoya Isue of Techology arxv: v [sa.ml] Mar 07 Ichro Takeuch akeuch.chro@ech.ac.jp Deparme of Compuer Scece/Research Isue for Iformao Scece, Nagoya Isue of Techology RIKEN Ceer for Advaced Iellgece Projec Ceer for Maerals Research by Iformao Iegrao, Naoal Isue for Maerals Scece Absrac We sudy prmal-dual ype sochasc opmzao algorhms wh o-uform samplg. Our ma heorecal corbuo hs paper s o prese a covergece aalyss of Sochasc Prmal Dual Coordae SPDC Mehod wh arbrary samplg. Based o hs heorecal framework, we propose Opmaly Volao-based Samplg SPDC, a o-uform samplg mehod based o Opmaly Volao. We also propose wo effce heursc varas of called ovssdpc ad ovssdpc. Through esve umercal expermes, we demosrae ha he proposed mehod ad s varas are faser ha oher sae-of-he-ar prmal-dual ype sochasc opmzao mehods. Iroduco I hs paper, we cosder lear classfcao ad regresso problems by solvg a regularzed emprcal rsk mmzao ERM problem wh he umber of saces ad he umber of feaures d he followg form m w P w := f x w λgw, [] where f : R R s a loss fuco, g : R d R s a pealy fuco, ad λ > 0 s regularzao parameer. The rag se s wre as {x, y } [] where x R d, y R for regresso, y {±} for classfcao, ad [] := {,..., }. We also deoe X R d o be he daa marx defed as X := [x,..., x ] ad s j-h colum s deoed as X :j. We assume ha f s covex ad /γ-smooh,.e., he grade s /γ-lpschz couous for all [], g s -srogly covex, ad x 0 for all []. As a workg example of a loss fuco ha has /γ-smoohess, we cosder smoohed hge loss defed as f z := 0 for y z > ad y z γ for y z < γ ad γ y z oherwse. As a example of -srogly covex pealy, we cosder elasc e pealy gw := w w. We oe ha all he heores ad algorhms dscussed hs paper ca be appled o ay oher combao of /γ-smooh loss ad -srogly covex pealy. Whe he daa s large, sochasc opmzao s ofe he mehod of choce for solvg a ERM problem. Amog several ypes of sochasc opmzao mehods, we focus hs paper o prmal-dual ype sochasc Correspodg auhor

2 opmzao mehods Shalev-Shwarz ad Zhag, 03; Csba e al., 0; Zhao ad Zhag, 0; Vasecher e al., 0; Qu e al., 0; Zhag ad Xao, 0; Zhu ad Sorkey, 0; We Yu e al., 0; Qu e al., 06. The dual problem of he emprcal rsk mmzao problem s wre as max α Dα:= f α λg X α λ, [] where f s he covex cojugae fuco of f, ad g s he covex cojugae fuco of g. For example, he covex cojugae of smoohed hge loss s wre as f z = γ z y z for y z [, 0] ad oherwse. The covex cojugae of elasc e pealy s g v = j [d] max{ v j, 0}. I prmal-dual ype opmzao mehods, he dual varables ad he prmal varables are aleravely updaed. Mos exsg prmal-dual ype sochasc opmzao mehods uses he uform samplg for selecg oe or more dual varables o be updaed each erao. Recely, several sudes are coduced for o-uform samplg Csba e al., 0; Zhao ad Zhag, 0; Vasecher e al., 0; Qu e al., 0; Zhag ad Xao, 0. Oe of such o-uform samplg approach s daa-drve samplg, where he samplg probably of he -h example depeds o s orm x. A possble lmao of daa-drve samplg s ha he samplg probables cao be chaged durg he opmzao process. Recely, a mehod called Quarz s proposed Qu e al., 0. Quarz s a SDCAlke prmal-dual ype sochasc opmzao mehod whose covergece s proved wh arbrary samplg. By usg arbrary samplg scheme, s possble o chage he samplg probables durg he opmzao process. Table summarzes a seleced ls of rece prmal-dual ype sochasc opmzao algorhms. Sochasc Dual Coordae Asce SDCA Shalev-Shwarz ad Zhag, 03 s oe of he mos well-kow o-acceleraed sochasc opmzao mehods wh uform samplg. I he work of Iprox-SDAC Zhao ad Zhag, 0, he auhors showed ha, he covergece rae ca be mproved by roducg he daa-drve samplg probably. As meoed above, Quarz Qu e al., 0 s a SDCA-lke mehod wh arbrary samplg. O he oher had, Sochasc Prmal Dual Coordae Mehod SPDC Zhag ad Xao, 0 s oe of he mos well-kow acceleraed sochasc opmzao mehod wh uform samplg. The auhors Zhag ad Xao, 0 also cosdered so-called Weghed SPDC whch uses he daa-drve samplg. A vara of SPDC called AdaSPDC was proposed Zhu ad Sorkey, 0. We oe ha AdaSPDC s a uform samplg-based mehod. The dfferece from he valla SPDC s ha he sep sze each erao depeds o he maxmum orm of he saces seleced he dual varable updae phase. Our corbuos As dcaed Table, oe of our ma heorecal corbuos hs paper s o aalyze he covergece of AdaSPDC whe saces are sampled based o a arbrary dsrbuo see 3. To he bes of our kowledge, here are o exsg sudes aalyzg he covergece of a acceleraed prmal-dual ype sochasc opmzao mehod wh arbrary samplg. Our secod corbuo s o propose parcular examples of samplg probables for AdaSPDC wh he heorecal framework meoed above see 4. Specfcally, as he frs such example, we propose o perodcally chage he samplg probables based o he progress of he opmzao ad demosrae s favorable emprcal performace whe he m-bach sze s small. We call hs mehod as Opmaly Volao-based Samplg SPDC mehod. As he secod example, we dscuss a approach suded AdaSDCA Csba e al., 0, where samplg probables are chaged a each erao based aga o he progress of he opmzao process. The hrd example s based o rece safe screeg sudes El Ghaou e al., 0; Ndaye e al., 0; Ogawa e al., 03; Shbagak e al., 06. Safe screeg allows us o defy a par of he prmal ad he dual varables ha ured ou o be zero a he opmal soluo. If we ca defy hese varables by safe screeg, we ca se he samplg probables of hose saces o zero. Smlar approach s employed a mehod called Affe-SDCA Vasecher e al., 0. Uforuaely, he above hree parcular examples of samplg probables have lmao geeral praccal use. Our hrd corbuo hs paper s o propose wo heursc varas of called ad.

3 Fgure : A llusrao of opmzao process whe solvg wh smoohed hge loss, elasc e pealy, ad λ = 0 4 dagyx / o w8a daase by usg SPDC wh he uform samplg. The lef par shows he orm x of 446 saces sored he creasg order of α x-axs. The ceer par shows how he dfferece bewee a dual varable ad s opmal value log 0 α α chages he hea map represeao. The rgh par shows he dual opmaly volao log 0 κ formally defed 4. Noe ha he dual opmaly volao s hghly correlaed wh he values he ceer par. I dcaes ha he dual opmaly volao ca be used for desgg a samplg probably for o-uform samplg. No Acc. Acceleraed Table : Seleced rece prmal-dual ype sochasc opmzao mehods. Uform Daa-drve Arbrary SDCA Quarz Shalev-Shwarz ad Zhag, 03 Zhao ad Zhag, 0 Qu e al., 0 SPDC, AdaSPDC Zhag ad Xao, 0, Zhu ad Sorkey, 0 Zhag ad Xao, 0 Ths paper Especally, he laer s useful whe he daa marx X s large boh he umber of saces ad he umber of feaures d. Whe we apply he heursc vara o he ERM problem wh smoohed hge loss ad elasc e pealy meag ha boh of he prmal ad he dual varables are sparse, he expermeal resuls dcae ha performs almos as well as oracle case where oe has he kowledge of he rue sparsy paers of he prmal ad he dual opmal soluo. Fgure llusraes he movao of hs sudy. From he ceer plo, we ca see ha abou dual varables coverged o he opmal values afer 00 or less eraos. Obvously, hose saces should o be sampled afer 00 eraos,.e., s desrable o se he samplg probables of hese saces o zero afer 00 eraos. The lef plo shows ha he dual opmaly volao are hghly correlaed wh he values he ceer plo. I meas ha, f we use he dual opmaly volao for seg he samplg probables, we mgh be able o assg hgh samplg probables o saces whose correspodg dual varables are sll away from he opmal values. O he oher had, he orm x he lef plo are o hghly correlaed wh he values he ceer plo. I suggess ha daa-drve samplg approaches based o x s o helpful. I he followg secos, all he proofs of heorems, lemmas, ad proposos are preseed Appedx A uless 3

4 oherwse saed. Relaed Works Sochasc opmzao algorhms wh o-uform samplg have bee suded he pas few years. Sochasc grade desce ype algorhms were suded Needell e al., 04; Rchárk ad Takáč, 0; Nu e al., 0; Palaappa ad Bach, 06, whle acceleraed coordae desce ype algorhms were suded Qu ad Rchárk, 06; Alle-Zhu e al., 06. I hs seco, we revew rece advaces prmal-dual ype sochasc opmzao mehods wh o-uform samplg. A prmal-dual ype sochasc opmzao algorhm aleravely eraes he prmal varable updae phase ad he dual varable updae phase. I mos of he exsg prmal-dual ype sochasc opmzao algorhms, all he prmal varables are updaed he prmal varable updae phase, whle oly a par of he dual varables are radomly seleced ad updaed he dual varable updae phase. Defo Proper samplg probably. Le us wre he samplg probably a dual varable updae phase as p := [p,..., p ], where p represes he probably ha he -h rag sace s sampled. We call p s proper f p > 0 for all []. Defo M-bach sze. Le us deoe a [, ] o be he m-bach,.e., he umber of dual varables sampled a each of he dual varable updae phase. As brefly dscussed, Table summarzes a seleced ls of rece prmal-dual ype sochasc opmzao algorhms. I hs seco, we revew some of hese relaed sudes more deal.. Daa-drve samplg We call a samplg probably p as daa-drve dsrbuo f p depeds oly o x. Zhao ad Zhag, 0 s a daa-drve samplg vara of SDCA. I hs algorhm, he case where f s he smoohed hgeloss ad he m-bach sze a =, a lower boud of E [ Dα Dα ] s maxmzed by seg he samplg probably as p x /λγ, ad he auhors showed ha he covergece rae of s faser ha valla SDCA. Weghed-SPDC Zhag ad Xao, 0 s a daa-drve samplg vara of SPDC, whose samplg probably s defed as p = x k [] x k. I dcaes ha saces whose orm x are greaer ha ohers ed o be sampled more ofe. The auhors showed ha, by usg he above daa-drve samplg probably, he erao complexy for obag a ε-accurae soluo ca be mproved o [] x from max [] x he case of valla SPDC.. Arbrary samplg Quarz Qu e al., 0 s a dual coordae asce ype algorhm such as SDCA. The auhors aalyzed he covergece propery of hs algorhm wh a proper arbrary samplg probably uder he codo ha a so-called Expeced Separable Over-approxmao ESO propery Rchárk ad Takáč, 0 s sasfed. I he case where f s /γsmooh, g s -srogly covex ad he m-bach sze a =, he auhor proposed a specfc samplg probably based o he orm x, whch resuls smlar ypes of samplg probably as. I 3, we wll dscuss more abou Quarz. 4

5 Algorhm : AdaSPDC wh o-uform samplg Ialze: a m-bach sze, w 0, α 0 al soluos, w 0 w 0, ᾱ 0 α 0 For =,,... o coverged do Geerae radom dex from p a mes wh replaceme deoe he se of radom dces as K Updae he dual ad he prmal coordaes: { } argmax β x, w f α = β p σ β α K β R α K ᾱ = α w = argm w R d ap α α [] { λgw w X ᾱ τ w = w θw w, } w w { where θ := max } τλ, max [] ap aσ γ, σ ad τ are he parameers ha would be deermed f he probably vecor p s deermed see Theorem 4 ad Theorem. 3 SPDC wh arbrary samplg I hs seco, we aalyze he covergece propery of AdaSDPC uder he suao ha dual varables are sampled from a proper arbrary samplg probably. Algorhm descrbes AdaSDPC wh o-uform samplg. Smlar covergece aalyss of valla SPDC Zhag ad Xao, 0 wh a proper samplg s preseed Appedx D. Lemma 3. Assume ha g s -srogly covex ad f s covex ad /γ-smooh for each []. Le p 0, /a] be a samplg probably, w ad α are he opmal soluo of ad, respecvely, ad defe as w w := //τ λ { γ α α } σ p a [] w w 4τ α α Xw w. The, f he parameers τ, σ, θ ad p Algorhm sasfy he followg equaly: τ x k ap k θ aσ k ap k 0, k K he, for 0, Algorhm acheves E [ ] θ. If we fx he samplg probably p for all 0, we ca derve he erao complexy of Algorhm for obag a ε-accurae soluo. Theorem 4. Suppose ha he assumpos of Lemma 3 hold. For ay proper dsrbuo p wh p 0, /a], [], f we se he parameers τ ad σ as τ = ar γ λ, σ = p λ x γ, where R := m p [] x, he Algorhm [ guaraees E w T w ] ε for T max [] ap x p a log 0 /τ λ γλ ε eraos. We ca mprove he erao complexy of Theorem 4 by resrcg samplg probables ad m-bach sze a as follows.

6 Doma facor of Necessary m-bach sze SPDC erao complexy for achevg opmal erao Lm o a Lm o p for obag complexy O τ, σ γλ log ε ε-approx soluo whe /γλ < /γλ < τ = R σ = R aγ λ λ aγ Zhag ad Xao, 0 τ = ar γ λ σ = p λ x Theorem 4 τ = ar σ = γ λ p x γ λ γ Uform a R aγλ p 0, a ] Uform max [] ap x ap γλ a p 0, a ] max [] Theorem Uform a R a γλ τ = σ = γ R λ λ x γ Theorem Uform - - R a γλ ap x ap γλ a - - Uaaable R γλ γλ Table : Ls of he doma facor of he erao complexy for obag a ε-approx soluo ad he ecessary m-bach sze for achevg he opmal erao complexy O γλ log/ε ha focus o he seg of τ ad σ, he lm o a, ad he lm o p. We defe R := max [] x. Theorem. Suppose ha he assumpos of Lemma 3 hold ad a. p 0, /a ], [], f we se he parameers τ ad σ as τ = ar [ p w m [] x, he Algorhm guaraees E T w ] eraos. For ay proper dsrbuo p wh γ λ, σ = p λ x γ ε for T max [] ap x p a γλ log, where R := 0 /τ λ ε I he remag par of hs seco, we wll dscuss relaos of Theorems 4 ad wh exsg works. Frs, whe we go back o uform samplg p = / whou m-bachg, he erao complexy Theorem s reduced o ha of valla SPDC ce as he followg remark. Remark 6. If p s uform ad a =, he he followg equaly holds max [] p x p λγ = λγ max [] x, where he rgh had sde s he doma facor of he erao complexy for obag ε-accurae prmal soluo valla SPDC. I meas ha Theorem s cosdered as a geeralzao of he resul Zhag ad Xao, 0. Nex, whe we cosder uform samplg p = / wh m-bachg,.e., a >, our resul Theorem mproves he aalyss of valla SPDC ce as he followg remark. Remark 7. If p s uform ad a >, we ca see ha Theorem mproves he erao complexy Corollary Zhag ad Xao, 0 from a max [] x aγλ o a max [] x a by resrcg m-bach sze a. γλ The prevous SPDC Zhag ad Xao, 0 suffers form poor m-bach effcecy ad whose he oal compuaoal cos for obag a ε-approxmae soluo creases by usg m-bachg as he auhors Qu e al., 0; Muraa ad Suzuk, 07 descrbed. However, Theorem shows ha he order of he oal compuaoal cos does o chage for a see Table 3. We oe ha he erao complexy Theorem acheves he opmal erao complexy of he frs order algorhm O γλ log ε whe m-bach sze a s ad /γλ, alhough he 6

7 Toal compuaoal cos for Necessary m-bach sze for achevg Algorhm wh he uform samplg obag ε-approx soluo opmal erao complexy O γλ log ε whe m-bach sze a > whe /γλ whe < /γλ < SPDC O d a λγ log ε Zhag ad Xao, 0 APCG O d a λγ log ε L e al., 04 Kayusha O d a λγ log ε Alle-Zhu, 06 DASVRDA O d λγ a λγ log ε λγ Muraa ad Suzuk, 07 SPDC O d λγ log ε for a Uaaable Theorem ad O d a λγ log ε for a λγ Table 3: Ls of he oal compuaoal cos for obag a ε-approx soluo ad he ecessary erao complexy for obag he opmal erao complexy O γλ log ε o each algorhm. erao complexy Zhag ad Xao, 0 eeds m-bach sze a =. We ca also mprove he resuls Zhag ad Xao, 0 for a see Appedx A ad B for more deal. Tables ad 3 compares he covergece properes of our aalyss wh exsg aalyss. See also Appedx B for more dealed aalyss ad expermes for SPDC wh m-bachg. Comparso wh We compare our aalyss wh he heorecal properes of. Remark 8. If max [] x k [] x k λγ holds ad we se p wh p x λγ ad a =, he he p mmzes he doma facor of he erao complexy max [] p makes he erao complexy oe-half besde see Appedx C for more deal. γσ Theorem ad Comparso wh Quarz We compare our aalyss wh he heorecal properes of Quarz. Remark 9. Theorem 4 shows ha he doma erm of he erao complexy depeds o / λγ. Compared wh he correspodg doma erm /λγ Quarz, our proposed mehod provdes he acceleraed covergece rae as dcaed Table. Remark 0. Whe he m-bach sze a >, he ESO propery Quarz s resrcve he sese ha s hard o fd proper samplg probably excep he uform oe. Therefore, he auhors Qu e al., 0 cosder o-uform samplg probably oly whe a =. O he oher had, as Theorem 4 dcaes, s possble o fd proper ouform samplg probables eve whe a > as log as p 0, /a], []. Acually, we prese a parcular example of such a proper samplg probably for a Examples of Samplg Probables I hs seco, we propose hree examples of samplg probables for SPDC wh he heorecal framework developed 3. 7

8 P w Dα w8a, m-bach sze a =, λ =e-λmax SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr #updaed dual coordaes/ w8a, m-bach sze a =, λ =e-3λmax SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr #updaed dual coordaes/ w8a, m-bach sze a =, λ =e-4λmax SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr. w8a, m-bach sze a =, λ =e-λmax SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr #updaed dual coordaes/ #updaed dual coordaes/ P w Dα real-sm, m-bach sze a =, λ =e-λmax SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr. real-sm, m-bach sze a =, λ =e-3λmax SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr #updaed dual coordaes/ #updaed dual coordaes/ real-sm, m-bach sze a =, λ =e-4λmax SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr #updaed dual coordaes/ real-sm, m-bach sze a =, λ =e-λmax SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr #updaed dual coordaes/ The llusrao of he covergece speed of each mehod o w8a o ormalzed ad real-sm or- Fgure : malzed. 4. Proper samplg based o opmaly volaos Here, we propose a proper samplg probably based o opmaly volaos a he curre soluo. The basc dea s o assg larger weghs o he saces whose correspodg dual varables are sll away from he opmal values. Le κ := α f x w, []. 3 Ths quay, whch we call dual opmaly volao, mples he volao of he opmaly codo of he curre prmal ad dual varables see Fgure. By usg κ, we se ρ x k [] [], p = ρ k x k p /a, ζ ζ ρ x k [] ρ k x k oherwse, where ρ := κ m k {l [] κ κ l 0} k, p := max [] ρ x / k [] ρ k x k, ζ := /a //p /. Here, s mpora o oe ha p sasfy he codos Theorem,.e., p 0, /a ] []. Ths s possble because we defe all he ρ o be ozero by addg a offse erm m k {l [] κ l 0} κ k. We calculae κ ad updae p every /a eraos. We frs eed o compue κ for all [], whch akes Od compuao. Furhermore, for o-uform samplg, we use Alas mehod Walker, 977 whch akes O for preprocessg me, ad O for samplg. Thus, he addoal cos for o-uform samplg s Od O per /a eraos, meag ha he order of he compuaoal me s kep same as uform samplg. 8

9 4. No proper samplg based o opmaly volao I he prevous seco, p s ozero eve he opmaly volao κ s zero. If we do so, he samplg probably p would o be proper aymore, ad he heorecal resuls dscussed 3 cao be used for provg he covergece. I a resrced case ha he loss fuco s he squared loss ad a =, he auhors Csba e al., 0 showed he covergece of AdaSDCA eve f p s exacly zero whe he opmaly volao κ s zero as log as κ s updaed a each erao. Ths s possble because κ = 0 meas α = α. We ca explo hs fac also for SPDC. For example, smoohed hge loss, he followg proposo holds. Proposo. A he -h erao of Algorhm, f f s smoohed hge loss ad γ p /σ 0 he α f x w = 0 α = α. By usg hs proposo ad compug κ a each erao, we ca prove he covergece eve f p = 0 for α f x w = 0. We ca also use he prmal opmaly volao defed as ψj := w j gj λ X :j α. 4 For example, elasc e pealy case he followg proposo holds. Proposo. A he -h erao of Algorhm, f g s elasc e pealy, λ/τ 0, ad sgwj = sgw j, he wj g j λ X :j ᾱ = 0 wj = wj. Ths proposo mples ha we ca prove covergece eve f we skp he updae of wj f w j g j λ X :j ᾱ = 0, λ/τ 0 ad sgwj = sgwj he prmal varable updae phase as log as ψ s updaed a each erao. 4.3 No proper samplg based o safe screeg Aoher approach for usg o-proper samplg probably wh he heorecal framework 3 s o roduce a dea safe sace screeg Ogawa e al., 03. Safe sace screeg allows us o defy a par of o-acve dual varables.e., α = 0 whou kowg he opmal soluo self. We ca prove covergece eve f p = 0 f he correspodg dual varable α s screeed-ou by safe sace screeg. We ca se p > 0 as a arbrary value for he oher. Affe-SDCA Vasecher e al., 0 explos hs dea,.e., chages p durg he opmzao process by seg p = 0 f he correspodg dual varable s defed as o-acve by safe sace screeg see Theorem 0 Vasecher e al., 0. We ca use he same dea for SPDC as saed he followg proposo. Proposo 3. Suppose ha he assumpos Lemma 3 hold, a = ad he al soluos sasfy 0 ε. Le I be he se of dces whch are defed as o-acve dual varables by safe sace screeg. If we se p = 0 for I ad arbrary value 0, / ] oherwse, τ = ar γ λ, σ = γ, where := I, R := p λ x m []\I p / x, p := m []\I p, he Algorhm guaraees, for T max []\I ap x p a γλ E [ T ] ε. There are also several sudes o safe feaure screeg whch allows us o defy a par of o-acve prmal varables.e., wj = 0 El Ghaou e al., 0; Ndaye e al., 0. We ca also prove he covergece eve f we skp he updae of wj f s defed as o-acve by safe feaure screeg. log ε ε, 9

10 4.4 Expermes Here, we demosrae he effecveess of he mehod proposed 4.. hrough umercal expermes o ERM wh smoohed hge loss ad elasc e pealy. We se γ = ad cosdered four dffere choces of he regularzao parameer λ {0, 0 3, 0 4, 0 }λ max, where λ max := dagyx /. Table 4 shows he daases used he expermes. Those daases were obaed form LIBSVM Daa Chag ad L, 0. We se a = because he compeg mehods wh o-uform samplg do o suppor m-bachg a >. Due o he space lmao, we oly provde he expermeal resuls o w8a ad real-sm. The expermeal resuls o he oher daases are preseed Appedx F. We mplemeed all he algorhms usg C ad Ege, ad aached our code as a supplemeary maeral. I wll be publshed o he web afer he paper s acceped. All he compuaos were performed o Iel Xeo CPU E-687W v3 3.0GHz, 6GB RAM. Table 4: Bechmark daases used he expermes. Daase ame Normalzed d #z/d w8a 4, jc-ra 3, a9a 3, real-sm 7,0 0, rcv-es 677,399 47, rcv-ra 0,4 47, #z dcaes he umber of o-zero elemes. Normalzed dcaes [], x =, meag ha daadrve samplg s reduced o he uform samplg. I Fgure, we compared he proposed mehod deoed as osvspdc wh e algorhms. Eve hough w8a s o ormalzed see Fgure, s o effecve. Smlar resuls were also repored by Vasecher e al., 0. The resuls show ha he proposed mehod s faser ha he oher mehods all cases. I parcular, s subsaally faser whe λ s small. Algorhm : pu : a m-bach sze, w 0, α 0 for =,,... o coverged do α updaeα [] w updaew Compue κ ad se p as ŵ 0, ˆα 0 w, α for u =,,... /a do Radomly pck up a subse dces K ˆα u updaeˆαu K ŵ u updaeŵ u f P λ ŵ u D λ ˆα u < P λ w D λ α he w, α ŵ u, ˆα u Algorhm 3: pu : a m-bach sze, w 0, α 0 for =,,... o coverged do α updaeα [] wj updaew j j [d] Compue κ, ψ ad se p as ŵ 0, ˆα 0 w, α for u =,,... /a do Radomly pck up a subse dces K ˆα u updaeˆαu K ŵj u updaeŵu j j {m [d] ψm 0} f P λ ŵ u D λ ˆα u < P λ w D λ α he w, α ŵ u, ˆα u 0

11 Fgure 3: A llusrao of opmzao process whe solvg wh smoohed hge loss, elasc e pealy, ad λ = 0 λ max o real-sm daase by usg SPDC wh he uform samplg he same maer as Fgure. We observe ha he prmal opmaly volao s also hghly correlaed wh he dfferece bewee a prmal varable ad s opmal value, suggesg ha he prmal opmaly volao ca be also used for selecvely updag he prmal varables. Effce heursc varas Uforuaely, each of he hree parcular examples of samplg probables dscussed 4 have lmao geeral praccal use. preseed 4. works well whe he m-bach sze a s small as demosraed he expermes 4.4. However, whe a s large, performace of s o sasfacory. The mehod descrbed 4. s mpraccal as meoed Csba e al., 0 abou AdaSDCA. Specfcally, hs mehod ad also AdaSDCA, oe mus compue κ ad ψ wh he compuaoal cos Od a each erao. Fally, he mehod based o safe screeg 4.3 s o so effecve especally he early sage of he opmzao because s oo coservave as we observed he expermes 4.4. I hs seco, order o overcome hese lmaos, we go beyod he heorecal framework developed 3, propose wo heursc varas of. Algorhms ad 3 descrbe hese wo heursc varas, each of whch s called ad, respecvely., we assume he separably of he pealy fuco.e., gw = j [d] g jw j. s parcularly desged for he case where he umber of saces s large, ad he dual opmal varables are sparse as s he case of hge loss or s varas. O he oher had, s parcularly desged for he case where boh he umber of saces ad he umber of feaures d are large, ad boh of he prmal ad he dual opmal varables are sparse. as s he case of hge loss or s varas sparsy ducg pealy such as elasc e. has ouer ad er loops. I he ouer loop, all he prmal ad he dual varable are updaed for guaraeeg he covergece, ad he opmaly volao κ s compued for seg he samplg probably as { / { [] κ p = 0} κ 0, 0 κ = 0, whch meas ha dual varables sasfyg he opmaly codos are ever updaed. I he er loop, all he prmal varables ad a par of he dual varables as specfed by he samplg probably are updaed. I he er loop, he umber of dual varables are cosdered o be { [] κ 0} sead of.

12 Fgure 4: The llusraos of he covergece speed of each mehod o real-sm.

13 , we also cosder updag oly a par of he prmal varables s er loop. To do hs, compues ψ he ouer loop, ad oly updaes he prmal varables whose ψj 0. Fgure 3 shows w j w j ad ψ j he process of he opmzao. Sce wj w j ad ψ j are hghly correlaed, ψ j ca be used as a crero for selecg he prmal varables o be updaed he er loop as s doe ovssdpc.. Expermes We demosrae he advaage of he proposed heursc varas hrough umercal expermes. The expermeal seups are he same as hose 4.4. We frs compare he compuaoal me of Algorhm, SPDC wh he uform samplg Zhag ad Xao, 0, SPDC wh he uform samplg Theorem, SPDC usg safe sace screeg, ad Oracle dual. Here, Oracle dual meas ha he sparsy paer of α has bee already kow, meag ha he opmzao s carred ou oly for hose who are acve. We cosdered wo dffere choces of m-bach sze a {4, 64}. Fgure 4 shows he resul o real-sm. are comparable s sll effecve he case of a = 4, alhough s less effecve ha whe a = 64 ad λ s small. Nex, we compared he compuaoal me of Algorhm 3, SPDC wh he uform samplg, SPDC usg smulaeous safe screeg Shbagak e al., 06, ad Oracle dual-prmal. Aga, Oracle dualprmal meas ha he sparsy paers of α ad wj have bee already kow, meag ha he opmzao s carred ou oly for hose who are acve. Fgure 4 shows he resul o real-sm, dcag ha s comparable o Oracle dual-prmal. 3

14 Ackowledgemes IT was parally suppored by JST CREST 6630, MEXT KAKENHI 6H0638, ad RIKEN Ceer for Advaced Iellgece Projec. Refereces Alle-Zhu, Z. 06. Kayusha: The frs drec accelerao of sochasc grade mehods. arxv prepr arxv: Alle-Zhu, Z., Qu, Z., Rchark, P., ad Yua, Y. 06. Eve faser acceleraed coordae desce usg o-uform samplg. Proceedgs of The 33rd Ieraoal Coferece o Mache Learg, pages 0 9. Chag, C.-C. ad L, C.-J. 0. Lbsvm: A lbrary for suppor vecor maches. ACM Trasacos o Iellge Sysems ad Techology TIST, 3:7. Csba, D., Qu, Z., ad Rchárk, P. 0. Sochasc dual coordae asce wh adapve probables. Proceedgs of The 3d Ieraoal Coferece o Mache Learg, pages 0 9. El Ghaou, L., Vallo, V., ad Rabba, T. 0. Safe feaure elmao for he lasso ad sparse supervsed learg problems. Pacfc Joural of Opmzao, 84: L, Q., Lu, Z., ad Xao, L. 04. A acceleraed proxmal coordae grade mehod. I Advaces Neural Iformao Processg Sysems 7, pages Muraa, T. ad Suzuk, T. 07. Doubly acceleraed sochasc varace reduced dual averagg mehod for regularzed emprcal rsk mmzao. arxv prepr arxv: Ndaye, E., Fercoq, O., Gramfor, A., ad Salmo, J. 0. Gap safe screeg rules for sparse mul-ask ad mul-class models. I Advaces Neural Iformao Processg Sysems, pages Needell, D., Ward, R., ad Srebro, N. 04. Sochasc grade desce, weghed samplg, ad he radomzed kaczmarz algorhm. I Advaces Neural Iformao Processg Sysems 7, pages Nu, J., Schmd, M. W., Laradj, I. H., Fredlader, M. P., ad Koepke, H. A. 0. Coordae desce coverges faser wh he gauss-souhwell rule ha radom seleco. Proceedgs of The 3rd Ieraoal Coferece o Mache Learg, pages Ogawa, K., Suzuk, Y., ad Takeuch, I. 03. Safe screeg of o-suppor vecors pahwse svm compuao. I Proceedgs of he 30h Ieraoal Coferece o Mache Learg, pages Palaappa, B. ad Bach, F. 06. Sochasc varace reduco mehods for saddle-po problems. I Advaces Neural Iformao Processg Sysems, pages Qu, Z. ad Rchárk, P. 06. Coordae desce wh arbrary samplg : Algorhms ad complexy. Opmzao Mehods ad Sofware, 3: Qu, Z., Rchárk, P., Takáč, M., ad Fercoq, O. 06. Sda: Sochasc dual ewo asce for emprcal rsk mmzao. I Proceedgs of he 33rd Ieraoal Coferece o Mache Learg, pages Qu, Z., Rchárk, P., ad Zhag, T. 0. Quarz: Radomzed dual coordae asce wh arbrary samplg. Advaces Neural Iformao Processg Sysems 8. 4

15 Rchárk, P. ad Takáč, M. 0. O opmal probables sochasc coordae desce mehods. Opmzao Leers, pages. Shalev-Shwarz, S. ad Zhag, T. 03. Sochasc dual coordae asce mehods for regularzed loss mmzao. Joural of Mache Learg Research, 4Feb: Shbagak, A., Karasuyama, M., Haao, K., ad Takeuch, I. 06. Smulaeous safe screeg of feaures ad samples doubly sparse modelg. I Proceedgs of he 33rd Ieraoal Coferece o Mache Learg, pages Vasecher, D., Lu, H., ad Zhag, T. 0. Local smoohess varace reduced opmzao. I Advaces Neural Iformao Processg Sysems 8, pages Walker, A. J A effce mehod for geerag dscree radom varables wh geeral dsrbuos. ACM Trasacos o Mahemacal Sofware TOMS, 33:3 6. We Yu, A., L, Q., ad Yag, T. 0. Doubly sochasc prmal-dual coordae mehod for emprcal rsk mmzao ad blear saddle-po problem. arxv prepr arxv: v. Zhag, Y. ad Xao, L. 0. Sochasc prmal-dual coordae mehod for regularzed emprcal rsk mmzao. Proceedgs of The 3d Ieraoal Coferece o Mache Learg, pages Zhao, P. ad Zhag, T. 0. Sochasc opmzao wh mporace samplg for regularzed loss mmzao. Proceedgs of The 3rd Ieraoal Coferece o Mache Learg. Zhu, Z. ad Sorkey, A. J. 0. Adapve sochasc prmal-dual coordae desce for separable saddle po problems. I Jo Europea Coferece o Mache Learg ad Kowledge Dscovery Daabases, pages Sprger.

16 A Proofs I hs Appedx, we prove Lemma 3, Theorem 4,,, ad Proposo,. A. Proof of Lemma 3 We exeded he proof Zhag ad Xao, 0. Lemma 3. Assume ha g s -srogly covex ad f s covex ad /γ-smooh for each []. Le p 0, /a] be a probably vecor, w ad α are he opmal soluo of ad, respecvely, ad defe as := τ λ w w { σ [] γ p α α a } w w 4τ α α Xw w. The, f he parameers τ, σ, θ ad p Algorhm sasfy he followg equaly: τ x k ap k θ aσ k ap k 0, k K 6 he, for 0, Algorhm acheves E [ ] θ. Proof. Le α as follows: { α := argm β x, w f β p } β α β R σ. From /γ-smoohess of f, he objecve fuco of he above opmzao problem s γ p σ -srogly covex. Therefore, we have he followg equaly: [] α x, w f α p σ α α α x, w f α p σ α α γ p α α. 7 σ Sce f s γ-srogly covex, we have By combg 7 ad 8, we have Sce p σ α α f α α x, w f α α x, w γ α α. 8 p σ γ α α p σ α α α α x, w w. E [ α α ] = ap α α ap α α E [ α α ] = ap α α E [ α ] = ap α ap α, 6

17 we have p α α γ E [ α σ aσ ap aσ E [ α α ] α ] ap aσ γ ap α α ap E [ α α ] α α x, w w. By summg =,..., ad dvdg by, we have { ap γ α α } { σ p a σ [] [] [ E where u := / [] α x, u := / [] α x. Le w as follows: { w := argm w R d u k K λgw u k K u u k K γ [ E α p α ] a } E [ k K α k α k ] aσ { } ] ap k α k αk x k, w w, 9 { } ap k α k αk x k, w τ w w }. From -srogly covexy of g, he objecve fuco of he above opmzao problem s τ λ-srogly covex. Therefore, we have { } λgw ap k α k αk x k, w w w τ λgw u k K Sce g s -srogly covex, we have { } ap k α k αk x k, w w w τ τ λ u, w λgw u, w λgw λ By combg 0 ad, ad akg expecao, we have w w τ τ λ E [ E [ w w ] E u u k K { ap k α k αk x k w w. 0 w w. [ w w ] τ }, w w ]. 7

18 By combg 9 ad, we have w w { τ [] τ λ E E ap γ α α σ p a [ w w ] { [] [ w w ] τ σ E u u { k K } θ α α Xw w γ [ E α α ] p a ap k α k α k x k } E [ k K α k αk ] aσ }, w w θw w } {{ } v. 3 By Leg δ := [ap,, ap ] R, ad be as he eleme-wse produc operaor, we have he followg equaly: v = α α Xw w θ α α Xw w δ α α Xw w θδ α α Xw w = α α Xw w α α Xw w θ α α Xw w δ α α Xw w θδ α α Xw w = α α Xw w θ α α Xw w { δk ap k α k αk x k, w w } θ { δk α k αk x k, w w }. 4 k K k K By usg Cauchy equaly wh τ, we have δ k ap k α k αk x k, w w k K k K δ k α k α k x k, w w From 3, 4,, ad 6, we have w w 4τ w w 4τ τ k K τ k K δk ap k α δk α w w { ap γ α α } w w θ τ σ p a 4τ [] [ τ λ E w w ] { γ [ E α α ] } σ p a [] [ ] E τ x k ap k θ aσ k ap k α k αk k K k k αk x k, αk x k. 6 θ α α Xw w w w 4τ α α Xw w. From he defo of θ, f he parameers τ, σ, θ ad p sasfy 6, he we have E [ ]. 8

19 A. Proof of Theorem 4 Lemma 4. Suppose ha he assumpos of Lemma 3 hold. If τ, σ, θ, ad p sasfes he followg equales: τσ ap k [], x k 4 x [], 7 /τ 4aσ he we have τ λ E [ w w ] { 4σ [] γ p [ E α α ] } θ 0. a Proof. By usg Cauchy equaly wh τ ad O he oher had, we have α α Xw w Form he defo θ ad 9, we have From 0, we have k [] x k /τ 4aσ, we have w w 4τ w w 4τ 6 a τσ x ap θ p τσ τσ ap 4 x α α X F /τ α α. 8 4aσ ap x ap θ. 9 ap x ap θ. 0 τ 4σ x ap = 6. By usg Lemma 3, 8, ad, we have τ λ E [ w w ] [] { 4σ γ p [ E α α ] } θ 0. a Theorem 4. Lemma 3 hold. For ay proper dsrbuo p wh p 0, /a], [], f we se he parameers τ ad σ as τ = ar γ λ, σ = p λ p, where R := m, x γ [] x he Algorhm guaraees T max [] ap x p a γλ log 0 /τ λ ε = E [ w T w ] ε. 9

20 Proof. From he lm o p ad he defo of τ, σ, ad R we have By usg, 3, ad Lemma 4, we have τ λ E O he oher had, we have τσ = ap R ap 4 x 4 x k [] x k /τ [ w w ] [] ap 4 x, max k [] x k 4σ x ap. 3 4aσ { 4σ γ p τλ = max x [] p a λγ, = x ap aγσ ap p a λγ. [ E α α ] } θ 0. a Therefore, he erao T whch archves θ T 0 /τ λ ε s as follows: T log 0/τ λ/ε max x logθ [] ap p a 0 /τ λ log. γλ ε A.3 Proof of Theorem Theorem. Suppose ha he assumpos of Lemma 3 hold ad a. For ay proper dsrbuo p wh p 0, /a ], [], f we se he parameers τ ad σ as τ = ar γ λ, σ = p λ p, where R := m, x γ [] x he Algorhm guaraees for T max [] ap x p a γλ log 0 /τ λ Proof. From he lm o p ad he defo of τ, σ, ad R we have ε = E [ w T w ] ε. By usg 4,, ad Lemma 4, we have τ λ E τσ = a p R ap 4 x 4 x, 4 k [] x k max k [] x k /τ 4σ x ap. 4aσ [ w w ] [] { 4σ γ p [ E α α ] } θ 0. a 0

21 O he oher had, we have τλ = max x [] p a λγ, = x ap aγσ ap p a λγ. Therefore, he erao T whch archves θ T 0 /τ λ ε s as follows: T log 0/τ λ/ε max x logθ [] ap p a 0 /τ λ log. γλ ε If p s uform, we have max x [] ap p a γλ = a max [] x a. λγ Therefore, Theorem mproves he doma erm of he complexy Corollary Zhag ad Xao, 0 for a. We cosder he seg of p proposed 4.: [], p = ρ x k [] ρ k x k ζ ζ ρ x k [] ρ k x k p /a oherwse, where p := max [] ρ x k [] ρ, ζ := /a /. k x k p / If p /a, he we have p 0, /a ] by he defo of p. I coras, f p > /a, he we have also p 0, /a ] because p ζ ζp = /a for all []. A.4 Proof of Theorem Theorem. Suppose ha he assumpos of Lemma 3 hold ad a Algorhm. For he uform probably vecor p, f we se he parameers τ ad σ as τ = γ R λ, σ = λ a x γ, where R := max [] x, he [ w Algorhm guaraees E T w ] ε for T max [] a R γλ log 0 /τ λ ε eraos. Proof. From he lm o p, lm o a, he defo of τ, σ, ad R we have By usg 6, 7, ad Lemma 4, we have τ λ E τσ = ap 4aR x 4 x, 6 k [] x k R. 7 /τ 4aσ R x 4aσ [ w w ] [] { 4σ γ p [ E α α ] } θ 0. a O he oher had, we have τλ = R, λγ = ap aγσ a x λγ

22 Therefore, he erao T whch archves θ T 0 /τ λ ε s as follows: T log 0/τ λ/ε logθ a R log λγ 0 /τ λ ε. A. Proof of Proposo Proposo. I a he erao of algorhm, f f s smoohed hge loss ad γ p /σ 0 he α f x w = 0 s a ecessary ad suffce codo for α = α. Proof. We frs prove α = α = α f x w = 0. Sce { βx α = argmax β R w f β p σ β α }, we have x w p σ α α f α. 8 From 8 ad he assumpo α = α ad he propery of subgrade, we have x w f α α f x w. Therefore, f assumpo holds, he α = α = α f x w = 0, because f s dffereable. Nex, we prove α f x w = 0 = α = α. Sce α = f x w, ad from he propery of subgrade, we have x w = f α. From 8, we have 0 f α f α p σ α α. From he dffereal of he cojugae of smoohed hge loss, we have γ p /σ α α = 0 Therefore, f he assumpos hold, he we have α = α. A.6 Proof of Proposo Proposo. I a he erao of algorhm, f g s elasc e λ /τ 0 ad sgwj = sgw j he wj g j λ X :j ᾱ = 0 s a ecessary ad suffce codo for wj = wj. Proof. We frs prove wj = wj = wj g j λ X :j ᾱ = 0. Sce, we have w j { = argm λg j v v X:j, ᾱ } v R τ v w j λ X :j ᾱ λτ w j wj g j wj. 9 From 9 ad he assumpo wj = wj ad he propery of subgrade, we have w j gj λ X :j ᾱ. Therefore, f he assumpos hold, he wj = wj = wj gj λ X :j ᾱ = 0, because gj s dffereable. Nex, we prove wj gj λ X :j ᾱ = 0 = wj = wj. Sce w j = gj λ X :j ᾱ, ad from he propery of subgrade, we have λ X :j ᾱ g j wj. From 9, we have 0 g jwj g j wj λτ w j wj. From he dffereal of elasc e pealy ad he assumpos, we have λ /τwj wj = 0 Therefore, f he assumpos hold, he we have wj = wj.

23 B SPDC v.s. Quarz m-bachg wh he uform samplg I hs Appedx, we compare our resuls wh SPDC Zhag ad Xao, 0 ad Quarz Qu e al., 0 usg he uform samplg m-bach seg. Table shows he doma facor of he erao complexy of SPDC ad Quarz whe m-bach sze a > ad x = for all []. Table 6 shows he doma facor of he erao complexy whe λγ = Θ/ or λγ = Θ or λγ = Θ. We ca see ha he erao complexy of Zhag ad Xao, 0 s decreased by oly / a mes whe λγ = Θ/ ad λγ = Θ, alhough he erao complexes of Theorem ad are decreased by /a mes. We demosrae he advaage of our resuls hrough umercal expermes. The expermeal seups are he same as hose 4.4. Table 7 shows he daases used he expermes. Fgure llusraes he resuls whe λγ = /, λγ =, ad λγ =. We ca see ha he our resuls are more effecve ha he prevous SPDC Zhag ad Xao, 0. Quarz Qu e al., 0 ca coverge o he arfcal daa whose r s exremely small, alhough Quarz coverge slowly o rcv-ra ad real-sm. Algorhm Quarz Qu e al., 0 SPDC Zhag ad Xao, 0 SPDC Theorem ad hs paper Doma facor of erao complexy a r a λγ a λγa { a max a λγ, λγ Table : The doma facor of he erao complexy of SPDC ad Quarz whe m-bach sze a > ad x = for all []. We defe r as r := Bĩ/ max [] x /a, where B := { d j= r j a / Xj} for all [], r j := [] X j 0 for all j [d], ad ĩ := argmax B. [] } Algorhm Doma facor of erao complexy λγ = Θ/ λγ = Θ λγ = Θ Quarz O 3/ a r O a r O a r Qu e al., 0 SPDC O /4 a O a O 3/4 a Zhag ad Xao, 0 a SPDC a O /4 /a O/a O/a Ths paper a O/a 3/4 O/a / O/a /4 Table 6: Comparso of he doma facor of he erao complexy of SPDC ad Quarz whe λγ = Θ/ or λγ = Θ or λγ = Θ. We oe r = whe #z/d =. C AdaSPDC wh daa-drve samplg I hs Appedx, we derve he erao complexy of AdaSPDC wh daa-drve samplg whe a =. The followg corollares ca derved by usg Theorem 4 ad. Corollary 6. Suppose ha he assumpos of Lemma 3 hold ad a = Algorhm. For he probably vecor p wh p = x γλ/ k [] x k γλ, [], f we se he parameers τ ad σ as 3

24 Daase ame r.99 a = 4.99 a = 6 30 a = 0 30 a = a = a = 00 Arfcal daa rcv-ra real-sm mj [d] rj maxj [d] rj mearj varrj #z/d Table 7: Bechmark daases used he expermes. We ca boud r as r maxj [d] rj. SPDC Our a = 4 0 SPDC Our a = 0 SPDC Our a = SPDC Zhag a = 0 SPDC Zhag a = 00 SPDC Our a = Ierao / real-sm, γλ = P w Dα Quarz a = Quarz a = 0 Quarz a = 00 SPDC Zhag a = SPDC Zhag a = 0 SPDC Zhag a = 00 SPDC Our a = 0 8 SPDC Our a = SPDC Zhag a = 00 0 Ierao / SPDC Zhag a = 0 SPDC Our a = 00 9 SPDC Our a = SPDC Our a = SPDC Our a = 0 0 Quarz a = Quarz a = 0 Quarz a = 00 SPDC Zhag a = 6 SPDC Our a = 6 0 SPDC Zhag a = SPDC Our a = 4 3 SPDC Zhag a = P w Dα SPDC Zhag a = 6 30 Ierao / 3 SPDC Zhag a = Quarz a = Quarz a = 4 Quarz a = 6 SPDC Zhag a = P w Dα rcv-ra, γλ = Quarz a = Quarz a = 0 Quarz a = 00 SPDC Zhag a = Ierao / 0 9 SPDC Our a = real-sm, γλ = 0 0 Quarz a = Quarz a = 0 Quarz a = 00 SPDC Zhag a = Arfcal daa, γλ = 0 8 SPDC Our a = SPDC Our a = 4 SPDC Our a = 00 0 Ierao / 0 4 SPDC Zhag a = 6 0 SPDC Our a = SPDC Zhag a = 4 SPDC Zhag a = 00 P w Dα P w Dα Quarz a = Quarz a = 4 Quarz a = 6 SPDC Zhag a = P w Dα rcv-ra, γλ = Arfcal daa, γλ = 9 0 SPDC Zhag a = 0 Ierao / Ierao / SPDC Zhag a = SPDC Zhag a = 0 Quarz a = Quarz a = 0 Quarz a = 00 SPDC Zhag a = 0 Quarz a = Quarz a = 0 Quarz a = 00 SPDC Zhag a = Ierao / SPDC Our a = 6 real-sm, γλ = / P w Dα SPDC Zhag a = SPDC Zhag a = 4 0 rcv-ra, γλ = / P w Dα P w Dα Quarz a = Quarz a = 4 Quarz a = 6 SPDC Zhag a = Arfcal daa, γλ = / Ierao / Fgure : The llusrao of he covergece speed of each mehod o each daase descrbed Table 7. 4

25 τ = R γ λ, σ = p λ x γ, where R := m [] k [] T x k γλ log eraos. 0 /τ λ ε p x, he Algorhm guaraees E [ w T w ] ε for Corollary 7. Suppose ha he assumpos of Lemma 3 ad max [] x k [] x k λγ hold ad a = Algorhm. For he probably vecor p wh p = x γλ/ k [] x k γλ, [], f we se he parameers τ ad σ as τ = R [ w guaraees E T w ] ε for T SPDC wh daa-drve samplg p γ p λ x λ, σ = k [] x k γλ log γ, where R := m [] eraos. 0 /τ λ ε Doma facor of erao complexy Lm o max [] x p x, he Algorhm Theorem Zhag ad Xao, 0 Corollary 6 hs paper Corollary 7 hs paper x R x λγ R λγ x λγ R λγ R λγ - R λγ - R λγ R λγ Table 8: Ls of he doma facor of he erao complexy of SPDC wh daa-drve samplg. We defe R as R := [] x. Table 8 shows he doma facor of he erao complexy of Zhag ad Xao, 0 ad our mehods. We demosrae he advaage of our resuls hrough umercal expermes. The expermeal seups are he same as hose 4.4. Fgure 6 shows he resuls. We ca see ha he our daa-drve samplg s more effecve ha Zhag ad Xao, 0. D valla SPDC wh arbrary samplg I hs appedx, we derve he erao complexy of valla SPDC wh arbrary samplg. We om he proofs because he followg corollares are almos same as Theorem 4 ad. Corollary 8. Suppose ha he assumpos of Lemma 3 hold. For ay proper dsrbuo p wh p 0, /a], [], f we se he parameers τ ad σ as τ = ar γ λ, σ = R λ γ, where R := m p [] x, he Algorhm 4 [ w guaraees E T w ] ε for T max [] ap Ra log 0 /τ λ γλ ε eraos. Corollary 9. Suppose ha he assumpos of Lemma 3 hold ad a. For ay proper dsrbuo p wh p γ λ, σ = R λ γ, where R := m [] era- 0, /a ], [], f we se he parameers τ ad σ as τ = ar [ w he Algorhm 4 guaraees E T w ] os. ε for T max [] ap Ra γλ log 0 /τ λ ε p x, E Doubly SPDC wh arbrary samplg Here, we aalyze Doubly SPDC DSPDC was proposed by We Yu e al., 0. For usg o-uform samplg ad he coveece of he proof, we made chages DSPDC as Algorhm. Here, we assume he separably of he pealy fuco.e., gw = j [d] g jw j.

26 Fgure 6: The llusrao of he covergece speed of each mehod o each daase. Algorhm 4: valla SPDC wh o-uform samplg Zhag ad Xao, 0 Ialze: a m-bach sze, w 0, α 0 al soluos, w 0 w 0, ᾱ 0 α 0 for =,,... o coverged do Geerae radom dex from p a mes wh replaceme deoe he se of radom dces as K Updae he dual ad he prmal coordaes: { argmax β x, w f α β p σ β α } K = β R α K ᾱ = α w = argm w R d ap α α [] { λgw w X ᾱ τ w = w θw w, w w } { where θ := max } τλ, max [] ap aσγ, σ, τ, θ are he parameers ha would be uquely deermed f he probably vecor p s deermed. 6

27 Algorhm : DSPDC wh o-uform samplg Ialze: a dual m-bach sze, b prmal m-bach sze, w 0, α 0 al soluos, w 0 w 0, ᾱ 0 α 0 Paro he dces of prmal varables o d mod b m-baches whose he umber of elemes are b ad d/b d mod b m-baches whose he umber of elemes are b for =,,... o coverged do Geerae a radom dex from p a mes deoe he se of radom dces as K Radomly pck up a subse M h follows he probably vecor q Updae prmal ad dual coordaes: { argmax β x, w f α β p σ β α } K = β R α K ᾱ = α ap k α α [] { argm λg wj j v v X:j, ᾱ } q M h d τ M = h v w j j M h v R wj j M h w = w θw w Theorem 0. Assume ha g s -srogly covex ad for each [], f s /γ-smooh ad covex. Le p 0, /a], q R d/b be probably vecors ad defe as follows: := { d τ M l λ } w M q l w M l { Ml σ γ α α } p a l d/b [] d w w 4τ M h α α Xw w. q Mh If he parameers τ, σ, θ ad p, q Algorhm sasfy he followg equales: d τ M l λ q M l / d q Ml τ M l λ q θ, Ml σ ap γ p θ / q θ, θ / Mh 4τ 4τq θ, Mh / σ γ p θ [], aσ ap k /q Mh θτλ k ap k 0 k K, 30 where Λ k := max M =b X k,m he for each 0, Algorhm acheves E [ ] θ. Proof. Le w j as follows: w j := argm v R { λg j v v X:j, ᾱ g } M h d τ M h v w j, j M h. Sce he objecve fuco of he above opmzao problem s λ g Mh d/τ M h -srogly covex from -srogly covexy of g j,, we have λg j w j w j λg j w j w j X:j, ᾱ q M h d τ M h w j wj 3 X:j, ᾱ q M h d λ τ M h w j w j q M h d w j wj. 3 τ M h 7

28 Sce g j s -srogly covex, we have λg j w j w j X :j, α λg j w j w j X :j, α λ w j w j. 33 By combg 3 ad 33, ad summg j M h boh sde, we have q Mh d w τ M h Mh wm h gj d τ M h λ E q Mh [ d w ] τ M h E M h wm h ᾱ α X :Mj E q Mh [ w M h w M h ] q M h w q Mh w M h Mh [ ] w M h q M h wm q h wm h Mh, because [ w E M h wm h [ w E M h wm h ] = q Mh w Mh wm h ] = q Mh wmh wj q M h w Mh wm h [ ] E w M h = q Mh w Mh q Mh wm h. By summg M l = M,..., M d/b, l d/b l d/b { d τ M l λ q M l q Ml d τ M l E [ w w [ w E ] } w Ml M l { d τ M l λ q Ml j d/b ] [ E ᾱ α X w w w w q Mh From 0 ad 34, we have { d τ M l λ q M l [ w E q ] } w Ml M l Ml j d/b { d τ M l λ q Ml d τ M l E [ w w [] { σ ap γ p { σ [ w ]} E M l wm l [] ] [ α aσ E α ] [ E ᾱ α X γ [ E α p [ w ]} E M l wm l ] α α } α ] a a } 34 q Mh w w θw w ]. Sce δ α α Xw w = k K τ k K δ α α Xw w τ k K ap k ap k α k αk X km h, w M h wm h Λ k ap k α k α k ap k Λ k α k α k ap k w w 4τ w w 4τ 8

29 , we have ᾱ α X w w θw w q Mh = q Mh α α Xw w q Mh δ α α Xw w θ α α Xw w θ δ α α Xw w α α Xw w θ q Mh α α Xw w w w 4τq Mh θ w w 4τ k K Therefore, we have { d τ M l λ q M l [ w E q ] } w Ml M l Ml l d/b [] θ w α α Xw w w θ 4τ { d τ M l λ [ w ]} E M q l wm l { Ml σ j d/b [] d α α Xw w q Mh τ M l E k K { aσ τλ k ap k /q Mh θ ap k τλk ap k /q Mh θ ap k α k α k. { σ ap γ p γ [ E α p [ w w ] 4τq Mh E [ α k αk ]}. α α } α ] a a } Hece, f he equales 30 sasfy, he we have E [ ]. F Oher expermes I hs appedx, we show he res of he expermeal resuls 4.4 ad.. 9

30 P w Dα P w Dα P w Dα P w Dα jc-ra, m-bach sze a =, λ =e-λmax jc-ra, m-bach sze a =, λ =e-3λmax jc-ra, m-bach sze a =, λ =e-4λmax jc-ra, m-bach sze a =, λ =e-λmax SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr #updaed dual coordaes/ SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr. 300 a9a, m-bach sze a =, λ =e-λmax 0 0 #updaed dual coordaes/ #updaed dual coordaes/ #updaed dual coordaes/ a9a, m-bach sze a =, λ =e-4λmax SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr. 0 0 SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr #updaed dual coordaes/ a9a, m-bach sze a =, λ =e-λmax SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr #updaed dual coordaes/ 00 SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr #updaed dual coordaes/ rcv-ra, m-bach sze a =, λ =e-λmax rcv-ra, m-bach sze a =, λ =e-3λmax rcv-ra, m-bach sze a =, λ =e-4λmax rcv-ra, m-bach sze a =, λ =e-λmax SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr. SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr. SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr. SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr a9a, m-bach sze a =, λ =e-3λmax SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr #updaed dual coordaes/ SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr. 0 0 #updaed dual coordaes/ #updaed dual coordaes/ #updaed dual coordaes/ #updaed dual coordaes/ rcv-es, m-bach sze a =, λ =e-λmax rcv-es, m-bach sze a =, λ =e-3λmax rcv-es, m-bach sze a =, λ =e-4λmax rcv-es, m-bach sze a =, λ =e-λmax SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr. SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr. SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr. SDCA uform SDCA u. Safe scr. Affe-SDCA Quarz uform SPDC uform SPDC u. Safe scr #updaed dual coordaes/ #updaed dual coordaes/ #updaed dual coordaes/ #updaed dual coordaes/ Fgure 7: The llusrao of he covergece speed of each mehod o jc-ra, a9a o ormalzed, rcv-ra ad rcv-es ormalzed he seg s he same as

31 Fgure 8: The llusrao of he covergece speed of each mehod o real-sm he seg s he same as. for a = {4, 6, 64}. 3

32 Fgure 9: The llusrao of he covergece speed of each mehod o rcv-ra ad rcv-es he seg s he same as. for a = {4, 6, 64}. 3

33 Fgure 0: The llusrao of he covergece speed of each mehod o rcv-ra he seg s he same as. for a = {4, 6, 64}. 33

34 Fgure : The llusrao of he covergece speed of each mehod o rcv-es he seg s he same as. for a = {4, 6, 64}. 34

35 G Opmaly Volaos Samplg v.s. Heursc Varas I hs appedx, we compare proposed 4. wh wo heursc varas of SDCA hrough umercal expermes. AdaSDCA Csba e al., 0 s heorecal mehods, however AdaSDCA eeds he calculao of κ a each erao ad suppors oly he squared loss. I order o overcome hese ssues, he auhors of Csba e al., 0 proposed AdaSDCA. AdaSDCA calculaes κ ad ses p wh p = κ x λγ a every eraos. I he er loop, AdaSDCA ses p = p /m, where m s he hyper-parameer. Emprcal SDCA, whch he auhors of Vasecher e al., 0 proposed, ses p as depeds o he α α. We cosder Quarz Qu e al., 0 wh he samplg proposed 4. whch s called ovsquarz because Quarz ca prove he covergece wh proper arbrary samplg he case of m-bach sze a =. The expermeal seups are he same as hose 4.4. We cosder he case of a = because wo heursc varas does o suppor m-bachg. Fgure shows he resuls. AdaSDCA s faser ha he ohers whe λ s large. s sable ad fas all cases, especally whe λ s small. Fgure : The llusrao of he covergece speed of each mehod o each daase. 3