Supplement of Limited-memory Common-directions Method for Distributed Optimization and its Application on Empirical Risk Minimization

Transcription

1 Suppement of Limited-memory Common-directions Method for Distributed Optimization and its Appication on Empirica Risk Minimization Ching-pei Lee Po-Wei Wang Weizhu Chen Chih-Jen Lin I Introduction In this document, we present more experimenta resuts, and detaied proofs for the theorems stated in the main paper II More Experiments We present more experimenta resuts that are not incuded in the main paper in this section We consider the same experiment environment, and the same probem being soved We present the resuts using different vaues of C to see the reative efficiency when the probems become more difficut or easier The resut of C = 0 3 is shown in Figure (I), and the resut of C = 000 is shown in Figure (II) For C = 0 3, the probems are easier to sove We observe that L-CommDir-BFGS is faster than existing methods on a data sets, and L-CommDir-Step outperforms state of the art on a data sets but ur, but is sti competitive on ur For C = 000, L-CommDir-BFGS is the fastest on most data sets, and the ony exception is KDD200-b, on which L-CommDir-BFGS is sighty sower than L-CommDir- Step, but faster than other methods On the other hand, L-CommDir-Step is sower than TRON on webspam but faster than existing methods on other data sets These resuts show that our method is highy efficient by using (29) or (20) to decide P k The other choice, L-CommDir-Grad, is obviousy inferior for most cases We aso modify from TRON to obtain a ine-search truncated Newton sover to compare with our method This ine-search truncated Newton method is denoted by NEW- TON in the resuts in Figures (III)-(V) Resuts show that NEWTON is consistenty the fastest on criteo for a choices of C, and outperforms L-CommDir-Step in the ater stage on ur for a C, and on epsion for C = 000, but L- CommDir-BFGS and L-CommDir-Step are faster on a other cases Therefore, in most cases, our method is the most efficient one III Proof of Theorem 3 The soution of (34) aso soves the foowing inear system (III) Q T H k Qt = Q T f(w k ), where Q [q,, q s ] If q j satisfies (36), then the righthand side of (III) is not a zero, hence t 0 and Qt 0 Therefore, from (III), we have (III2) p T k f(w k ) = (Qt) T H k Qt M 2 Qt 2 = M 2 p k 2 We then have from Assumption and (III2) that f(w k + θ k p k ) f(w k ) + θ k f(w k ) T p k + θk 2 2 p k 2 f(w k ) + θ k f(w k ) T p k ( θ k 2M 2 ) From (III2), f(w k ) T p k < 0 Therefore, when θ k 2M 2 c, (22) is satisfied Thus, we have that the backtracking ine search gets a step size θ k such that (III3) ( θ k min, 2β ( c ) ) M 2 Therefore, the backtracking ine search procedure takes at most min ( 0, og β (2β( c )(M 2 /)) ) steps IV Proof of Theorem 32 The j-th equation in the inear system (III) is (IV) p T k H k q j = f(w k ) T q j By (35), (36), and (IV), University of Wisconsin-Madison ching-pei@cswiscedu Carnegie Meon University poweiw@cscmuedu Microsoft wzchen@microsoftcom Nationa Taiwan University cjin@csientuedutw p k q j (p M k ) T H k q j = f(w k ) T q M j f(w k ) q j δ M

2 Therefore, (IV2) p k δ M f(w k ) Combining (III2) and (IV2), we can estabish the foowing resut pt k f(w k) p k f(w k ) M 2 p k 2 p k f(w k ) δm 2 M Now to show the convergence, we notice that a point w is a stationary point if and ony if f( w) = 0 By (22), (III2), and (IV2), we have f(w k+ ) f(w k ) + θ k c f(w k ) T p k f(w k ) M 2δ 2 θ k c 2 f(w k ) 2 From (III3), we can repace θ k in the above resut with some positive constant κ and get (IV3) f(w k+ ) f(w k ) M 2δ 2 κc 2 f(w k ) 2 Thus, summing (IV3) from w 0 to w k, we get (IV4) k j=0 M 2 δ 2 κc 2 f(w j ) 2 f(w 0 ) f(w k+ ) f(w 0 ) f, Note that in the second inequaity, we utiized the fact that β, c, σ/ [0, ] Therefore, the backtracking ine search procedure takes at most og β (β( c )(σ/)) steps, and we can take (V) κ = β( c )σ Now to show the inear convergence, the Poyak- Łojasiewicz condition (37) hods with σ from the strong convexity by [, Theorem 20] and noting that f(w ) = 0 Therefore, by (V), (IV5) becomes f(w k+ ) f ( 2βc ( c )σ 3 3 )(f(w k ) f ), and the required iterations to reach an ɛ-accurate soution is therefore k og (f (w 0) f ) + og ( ) ( ( ɛ )) og / 2βc( c)σ3 3 References [] Y E Nesterov Introductory Lectures on Convex Optimization: A Basic Course Kuwer Academic Pubishers, 2003 where f is the minima objective vaue of f Consequenty, min f(w j) 2 0 j k k + k f(w j ) 2 j=0 2 k + M 2 δ 2 (f(w 0 ) f ) κc and the resut foows Now consider that f satisfies (37) Deducting f from both sides of (IV3) and combining it with (37), we get (IV5) f(w k+ ) f ( 2σM 2δ 2 κc 2 )(f(w k ) f ), and inear convergence is proved Note that our assumptions give c > 0, and σm 2 /M 2 > 0 Therefore the coefficient in the right-hand side of (IV5) is smaer than V Proof of Coroary 3 We note that in Agorithm, because the gradient itsef is incuded in the directions, (36) is satisfied with δ = Moreover, (35) is satisfied by (M, M 2 ) = (, σ) from Assumption and the strong convexity Thus from (III3), θ k min (, 2β ( c ) σ ) β( c )σ

3 (a) criteo (b) kdd202 (c) ur (d) KDD200-b (e) epsion (f) webspam (g) news20 (h) rcvt Figure (I): Comparison of different agorithms with C = 0 3 We present training time vs reative difference to the optima objective vaue The horizonta ines mark the stopping condition of TRON in MPI-LIBLINEAR: k f (w)k i = ) min(#yi =,#y k f (0)k, with = 0 2 (defaut), 0 3, and 0 4 (a) criteo (b) kdd202 (c) ur (d) KDD200-b (e) epsion (f) webspam (g) news20 (h) rcvt Figure (II): Comparison of different agorithms with C = 000 We present training time vs reative difference to the optima objective vaue The horizonta ines mark the stopping condition of TRON in MPI-LIBLINEAR: k f (w)k i = ) min(#yi =,#y k f (0)k, with = 0 2 (defaut), 0 3, and 0 4

4 (a) criteo (b) kdd202 (c) ur (d) KDD200-b (e) epsion (f) webspam (g) news20 (h) rcvt Figure (III): Comparison with ine-search truncated Newton with C = 0 3 We present training time vs reative difference to the optima objective vaue The horizonta ines mark the stopping condition of TRON in MPI-LIBLINEAR: f(w) ɛ min(#yi=,#yi= ) f(0), with ɛ = 0 2 (defaut), 0 3, and 0 4 (a) criteo (b) kdd202 (c) ur (d) KDD200-b (e) epsion (f) webspam (g) news20 (h) rcvt Figure (IV): Comparison with ine-search truncated Newton with C = We present training time vs reative difference to the optima objective vaue The horizonta ines mark the stopping condition of TRON in MPI-LIBLINEAR: f(w) ɛ min(#yi=,#yi= ) f(0), with ɛ = 0 2 (defaut), 0 3, and 0 4

5 (a) criteo (b) kdd202 (c) ur (d) KDD200-b (e) epsion (f) webspam (g) news20 (h) rcvt Figure (V): Comparison with ine-search truncated Newton with C = 000 We present training time vs reative difference to the optima objective vaue The horizonta ines mark the stopping condition of TRON in MPI-LIBLINEAR: f(w) ɛ min(#yi=,#yi= ) f(0), with ɛ = 0 2 (defaut), 0 3, and 0 4