Efficient Methods for Large-Scale Optimization

Transcription

1 Efficient Methods for Large-Scale Optimization Aryan Mokhtari Department of Electrical and Systems Engineering University of Pennsylvania Ph.D. Proposal Advisor: Alejandro Ribeiro Committee: Ali Jadbabaie (chair), Asuman Ozdaglar, and Gesualdo Scutari November 21st, 2016 Aryan Mokhtari Efficient Methods for Large-Scale Optimization 1

2 Introduction Introduction Stochastic quasi-newton algorithms Proposed research: Incremental aggregated BFGS Distributed methods Proposed research: Quasi-Newton extension Proposed research: Stochastic decentralized methods Adaptive sample size algorithms Proposed research: Ada Newton Conclusions Aryan Mokhtari Efficient Methods for Large-Scale Optimization 2

3 Large-scale Optimization Expected risk minimization problem min w R p E θ [f (w, θ)] The distribution is unknown We have access to N independent realizations of θ We settle for solving the Empirical Risk Minimization (ERM) problem 1 min F (w) := min w Rp w R p N N f (w, θ i ) i=1 Large-scale optimization or machine learning: large N, large p N: number of observations (inputs) p: number of parameters in the model Applications: classification problem, wireless networks, sensor networks,... Aryan Mokhtari Efficient Methods for Large-Scale Optimization 3

4 Optimization methods Stochastic methods: a subset of samples is used at each iteration SGD is the most popular; however, it is slow because of Noise of stochasticity Variance reduction (SAG, SAGA, SVRG,...) Poor curvature approx. Stochastic QN (SGD-QN, RES, olbfgs,...) Decentralized methods: samples are distributed over multiple processors Primal methods: DGD, Acc. DGD, NN,... Dual methods: DDA, DADMM, DQM, EXTRA, ESOM,... Adaptive sample size methods: start with a subset of samples and increase the size of training set at each iteration Ada Newton The solutions are close when the number of samples are close Aryan Mokhtari Efficient Methods for Large-Scale Optimization 4

5 Stochastic quasi-newton algorithms Introduction Stochastic quasi-newton algorithms Proposed research: Incremental aggregated BFGS Distributed methods Proposed research: Quasi-Newton extension Proposed research: Stochastic decentralized methods Adaptive sample size algorithms Proposed research: Ada Newton Conclusions Aryan Mokhtari Efficient Methods for Large-Scale Optimization 5

6 Stochastic Gradient Descent Objective function gradients s(w) := F (w) = 1 N N f (w, θ i ) i=1 Gradient descent iteration w t+1 = w t ɛ t s(w t) Gradient evaluation is not computationally affordable Stochastic gradient Replace expectation by sample average ŝ(w, θ) = 1 L L f (w, θ l ) l=1 Defined aggregate sample random variable θ = [θ 1;...; θ L ] Stochastic gradient descent iteration w t+1 = w t ɛ t ŝ(w t, θ t) (Stochastic) gradient descent is (very) slow. Newton is impractical Aryan Mokhtari Efficient Methods for Large-Scale Optimization 6

7 BFGS quasi-newton method Correct curvature with Hessian approximation matrix B 1 t w t+1 = w t ɛ t B 1 t s(w t) Approximation B t close to H(w t) := 2 F (w t). Broyden, DFP, BFGS Variable variation: v t = w t+1 w t. Gradient variation: r t = s(w t+1) s(w t) Follow the colors Matrix B t+1 satisfies secant condition B t+1v t = r t. Underdetermined Aryan Mokhtari Efficient Methods for Large-Scale Optimization 7

8 BFGS quasi-newton method Resolve indeterminacy making B t+1 closest to previous approximation B t B t+1 = argmin tr(b 1 t Z) log det(b 1 t Z) n s. t. Z v t = r t, Z 0 Proximity measured in terms of differential entropy Solve to find Hessian approximation update B t+1 = B t + rtrt t v T Btvtvt T B t t r t v T t B tv t For positive definite B 1 0 and nonnegative variation v T t r t > 0 B t stays positive definite for all iterations t Aryan Mokhtari Efficient Methods for Large-Scale Optimization 8

9 Stochastic (online) BFGS SBFGS substitute all the gradients in BFGS by stochastic gradients Correct curvature with Hessian approximation matrix B 1 t w t+1 = w t ɛ t ˆB 1 t ŝ(w t, θ t) Variable variation v t = w t+1 w t Stochastic gradient variation ˆr t = ŝ(w t+1, θ t) ŝ(w t, θ t) Update the Hessian ˆB t+1 = ˆB t + ˆrtˆr t T ˆB T tv tv t ˆBt v t Tˆr t v T t ˆBtv t ˆB t can become arbitrarily close to be positive semidefinite Eigenvalues of ˆB 1 t are unbounded No guarantee for convergence Aryan Mokhtari Efficient Methods for Large-Scale Optimization 9

10 Regularized BFGS Modify BFGS to resolve the issue of unbounded eigenvalues Secant condition is fundamental. Proximity criteria somewhat arbitrary Restrict search to matrices Z whose smallest eigenvalue is at least δ B t+1 = argmin tr[b 1 t (Z δi)] log det[b 1 t (Z δi)] n s. t. Z v t = r t, Z 0 Solve to find regularized Hessian approximation update B t+1 = B t + rt r t T Btvtvt T B t v t T r t v T + δi t B tv t Modified gradient variation r t := r t δv t Replaced gradient variation r t by modified gradient variation r t := r t δv t Added regularization term δi Aryan Mokhtari Efficient Methods for Large-Scale Optimization 10

11 Regularized Stochastic BFGS (RES) Replace stochastic gradient in descent iteration and secant condition Matrix ˆB t+1 satisfies instantaneous secant condition and is closest to ˆB t ˆB t+1 = argmin tr[ˆb 1 t (Z δi)] log det[ˆb 1 t (Z δi)] n s. t. Z v t = ˆr t, Z 0 Solve in closed form to find Hessian approximation update (if ˆB δi) ˆB t+1 = ˆB t + rt r t T ˆB T tv tv t ˆBt v t T r t v T t ˆB tv t + δi Stochastic modified gradient variation r t = ˆr t δv t Replaced gradient variation r t by stochastic gradient variation ˆr t Stochastic BFGS descent w t+1 = w t ɛ t (ˆB 1 t +ΓI) ŝ(w t, θ t) Added ΓI to Hessian inverse approximation ˆB 1 t Aryan Mokhtari Efficient Methods for Large-Scale Optimization 11

12 RES Require Initial variable w 0. Hessian approximation ˆB 0 δi. for t = 0, 1, 2,... do 1: Collect L realization of random variable θ t = [θ t1,..., θ tl ] 2: Compute stochastic gradient ŝ(w t, θ t) = 1 L L l=1 f (wt, θ tl) 3: Update the variable w t+1 = w t ɛ t (ˆB 1 t + ΓI) ŝ(w t, θ t) 4: Compute variable variation v t = w t+1 w t 5: Compute stochastic gradient ŝ(w t+1, θ t) = 1 L L l=1 f (wt+1, θ tl) 6: Compute modified stochastic gradient variation r t r t = ˆr t δv t = ŝ(w t+1, θ t) ŝ(w t, θ t) δv t 7: Hessian approximation ˆB t+1 = ˆB t + rt r T t ˆB T tv tv t ˆB t + δi. v t T r t v T t ˆBtv t Aryan Mokhtari Efficient Methods for Large-Scale Optimization 12

13 Assumptions Eigenvalues of instantaneous Hessians H(w, θ) := 2 f (w, θ) bounded mi H(w, θ) MI Second moment of the norm of the stochastic gradient is bounded E [ ŝ(w t, θ t) 2] S 2 Regularization parameter δ is less than m δ < m Guarantee positiveness of inner product r T t v t > 0 Aryan Mokhtari Efficient Methods for Large-Scale Optimization 13

14 Almost sure convergence Theorem Consider the regularized stochastic BFGS algorithm as defined. If the stepsize sequence satisfies ɛ t = and ɛ 2 t <, the limit of the squared distance t=1 to optimality w t w 2 satisfies t=1 lim t wt w 2 = 0, w.p.1 Aryan Mokhtari Efficient Methods for Large-Scale Optimization 14

15 Convergence rate Theorem Consider the regularized stochastic BFGS algorithm as defined and let the step size sequence given by ɛ t = ɛ 0t 0/(t 0 + t) such that 2ɛ 0t 0Γ > 1. Then the difference between the expected objective value E[F (w t)] and the optimal objective F (w ) satisfies E[F (w t)] F (w ) O ( ) 1 t Aryan Mokhtari Efficient Methods for Large-Scale Optimization 15

16 Numerical results SVM problem N = 10 3 and p = 40 RES outperforms the other methods by exploiting curvature information Objective function value (F(wt)) SAA SGD S2GD SAG RES x 10 4 Number of processed feature vectors (Lt) RES has the fastest convergence time to achieve accuracy F (w t ) = 10 4 n N RES SGD SAG S2GD SAA ms 520 ms 350 ms 280 ms > 690 ms Aryan Mokhtari Efficient Methods for Large-Scale Optimization 16

17 Making RES more efficient B 1 t depends on B 1 t 1 and the curvature information pairs {vt 1, rt 1} depends on B 1 0 and all previous curvature information {v u, r u} t 1 u=0 The old curvature information pairs should be less related B 1 t LBFGS uses only the last τ curvature information pairs {v u, r u} t 1 u=t τ The computation complexity of implementing LBFGS is O(τp) O(p 2 ) Stochastic method Substitute gradients with stochastic gradients No regularization is required for the stochastic version Aryan Mokhtari Efficient Methods for Large-Scale Optimization 17

18 Bounded eigenvalues without regularization Lemma Consider the ol-bfgs algorithm as defined. Then, the trace and determinant of the approximate Hessian ˆB t are bounded as ) tr (ˆB t (p + τ)m, ) det (ˆB t m p+τ [(p + τ)m] τ Lemma Consider the ol-bfgs algorithm as defined. Then, the eigenvalues of the approximate Hessian ˆB t are bounded as m p+τ [(p + τ)m] p+τ 1 I =: ci ˆB t CI := (p + τ)m I. The eigenvalues of the approximate Hessian inverse ˆB 1 t are bounded Aryan Mokhtari Efficient Methods for Large-Scale Optimization 18

19 Large-scale Click-Through Rate (CTR) Soso search engine data set from Tencent Sun (2014) with searches We use N = feature vectors with dimension p = Description Components Max # Mean # Gender Age Ad URL 26, Depth Position Depth and Position Advertisement ID 641, Advertiser ID 14, Query ID 24,122, Keyword ID 1,188, Title ID 3,735, Description ID 2,934, User ID 22,023, Query Tokens 1,039, Ad Keyword Tokens 91, Ad Title Tokens 101, Ad Description Tokens 122, Cosine Similarities Depth (Numerical) Position (Numerical) # of Query Tokens # of Ad Keyword Tokens # of Ad Title Tokens # of Ad Description Tokens Total Counts 56,040, Aryan Mokhtari Efficient Methods for Large-Scale Optimization 19

20 Comparison in terms of convergence rate olbfgs and SGD processing up to feature vectors Relative Objective Function Value (F(w) F(w )) SGD olbfgs Prediction Error SGD olbfgs Processed Feature Vectors (Lt) x Processed Feature Vectors x 10 7 SGD achieves F (w t) F (w ) = 0.78 after processing feature vectors olbfgs achieves F (w t) F (w ) = 0.13 after processing feature vectors Prediction error of olbfgs is less than SGD after processing feature vectors olbfgs error = SGD error = 0.38 Aryan Mokhtari Efficient Methods for Large-Scale Optimization 20

21 Comparison in terms of convergence time olbfgs and SGD running for up to 9 hours Relative Objective Function Value (F(w) F(w )) SGD olbfgs Prediction Error SGD olbfgs Run Time (Hours) Run Time (Hours) olbfgs outperforms SGD despite larger per-iteration computational cost Aryan Mokhtari Efficient Methods for Large-Scale Optimization 21

22 Proposed research: Incremental aggregated BFGS Introduction Stochastic quasi-newton algorithms Proposed research: Incremental aggregated BFGS Distributed methods Proposed research: Quasi-Newton extension Proposed research: Stochastic decentralized methods Adaptive sample size algorithms Proposed research: Ada Newton Conclusions Aryan Mokhtari Efficient Methods for Large-Scale Optimization 22

23 Proposed research SQN methods improve the curvature approximation in SGD They outperform SGD in ill-conditioned problems However, they can t improve the convergence rate of SGD Convergence rate is sublinear Can t recover the superlinear rate of quasi-newton methods Reason? Noise of stochasticity does not vanish with constant stepsize Solution: Incremental aggregated algorithms (variance reduction methods) Track the information of all the functions f 1,..., f N in a memory Update the information of only one function f i at each iteration Use the most recent inf. of all functions f 1,..., f N to update the model Aryan Mokhtari Efficient Methods for Large-Scale Optimization 23

24 Incremental aggregated gradient method f t 1 f t it f t N f i t (wt+1 ) f t+1 1 f t+1 it f t+1 N Consider i t as the index of the function chosen at step t Step 1: Compute w t+1 = w t α N N f i i=1 Step2: Update the gradient corresponding to function f it Aryan Mokhtari Efficient Methods for Large-Scale Optimization 24

25 Incremental BFGS method (Part I) z t 1 z t it z t N B t 1 B t it B t N f t 1 f t it f t N w t+1 BFGS f i t (wt+1 ) z t+1 1 z t+1 it z t+1 N B t+1 1 B t+1 it B t+1 N f t+1 1 f t+1 it f t+1 N B i Approximates Hessian 2 f i z i copy of variable w corresponding to function f i Consider i t as the index of the function chosen at step t ( Step 1: Compute w t+1 = w t 1 N N i=1 B t i ) 1 ( ) 1 N f i N i=1 Step2: Update the information corresponding to function f it It doesn t work Could be as slow as the incremental gradient method Aryan Mokhtari Efficient Methods for Large-Scale Optimization 25

26 Incremental BFGS method (Part II) Approximate f i (w) by its (BFGS) quadratic approximation around z t i The global loss f (w) := (1/N) N i=1 f i(w) is therefore approximated by f (w) 1 N N i=1 [ f i (z t i ) + f i (z t i ) T (w z t i ) + 1 ] 2 (w zt i ) T B t i (w z t i ) Solving this quadratic programming yields the update ( w t+1 1 N = N i=1 B t i ) 1 [ 1 N B t i z t i 1 N N i=1 ] N f i (z t i ) i=1 Aryan Mokhtari Efficient Methods for Large-Scale Optimization 26

27 Numerical results Quadratic programming f (w) := (1/N) N i=1 wt A i w/2 + b T i w A i R p p is a diagonal positive definite matrix b i R p is a random vector from the box [0, 10 3 ] p N = 1000, p = 100, and condition number (10 2, 10 4 ) Relative error w t w / w 0 w of SAG, SAGA, IAG, and IQN Normalized Error SAG SAGA IAG IQN Normalized Error SAG SAGA IAG IQN Number of Effective Passes (a) small condition number Number of Effective Passes (b) large condition number Aryan Mokhtari Efficient Methods for Large-Scale Optimization 27

28 Distributed methods Introduction Stochastic quasi-newton algorithms Proposed research: Incremental aggregated BFGS Distributed methods Proposed research: Quasi-Newton extension Proposed research: Stochastic decentralized methods Adaptive sample size algorithms Proposed research: Ada Newton Conclusions Aryan Mokhtari Efficient Methods for Large-Scale Optimization 28

29 Distributed optimization Stochastic methods attempt to distribute samples (functions) over time The alternative approach is to distribute samples over processors Network with n nodes. Each node i has access to local function f i (x) n Collaborate to minimize global objective f (x) = f i (x) i=1 f 2 (x) f 5 (x) f 8 (x) f 1 (x) f 3 (x) f 6 (x) f 9 (x) f 4 (x) f 7 (x) f 10 (x) Recursive exchanges with neighbors j N i to aggregate global information Aryan Mokhtari Efficient Methods for Large-Scale Optimization 29

30 Methods for distributed optimization n Replicate common variable at each node f (x 1,..., x n) = f i (x i ) Enforce equality between neighbors x i = x j (thus between all nodes) i=1 f 2 (x) f 5 (x) f 8 (x) f 1 (x) f 3 (x) f 6 (x) f 9 (x) f 4 (x) f 7 (x) f 10 (x) Operate recursively to enforce equality asymptotically. Differ on how. Distributed gradient descent, recursive averaging, [Nedic, Ozdaglar 09] Distributed dual ascent, prices, [Rabbat et al 05] Distributed ADMM, prices, [Schizas et al 08] All are first order methods, thus, convergence times not always reasonable Aryan Mokhtari Efficient Methods for Large-Scale Optimization 30

31 Methods for distributed optimization n Replicate common variable at each node f (x 1,..., x n) = f i (x i ) Enforce equality between neighbors x i = x j (thus between all nodes) i=1 f 2 (x 2 ) f 5 (x 5 ) f 8 (x 8 ) f 1 (x 1 ) f 3 (x 3 ) f 6 (x 6 ) f 9 (x 9 ) f 4 (x 4 ) f 7 (x 7 ) f 10 (x 10 ) Operate recursively to enforce equality asymptotically. Differ on how. Distributed gradient descent, recursive averaging, [Nedic, Ozdaglar 09] Distributed dual ascent, prices, [Rabbat et al 05] Distributed ADMM, prices, [Schizas et al 08] All are first order methods, thus, convergence times not always reasonable Aryan Mokhtari Efficient Methods for Large-Scale Optimization 31

32 (Approximate) Network Newton (NN) Reinterpret distributed gradient descent (DGD) as a penalty method Newton step for objective + a penalty requires global coordination Approximate with local operations by truncating Taylor s series of Hessian inverse kth term of series is k-hop neighbor sparse NN-k aggregates information from k-hop neighborhood f 2 (x 2 ) f 5 (x 5 ) f 8 (x 8 ) f 1 (x 1 ) f 3 (x 3 ) f 6 (x 6 ) f 9 (x 9 ) f 4 (x 4 ) f 7 (x 7 ) f 10 (x 10 ) NN-k converges linearly always and exhibits a quadratic phase in a range Aryan Mokhtari Efficient Methods for Large-Scale Optimization 32

33 (Approximate) Network Newton (NN) Reinterpret distributed gradient descent (DGD) as a penalty method Newton step for objective + a penalty requires global coordination Approximate with local operations by truncating Taylor s series of Hessian inverse kth term of series is k-hop neighbor sparse NN-k aggregates information from k-hop neighborhood NN-1 f 2 (x 2 ) f 5 (x 5 ) f 8 (x 8 ) f 1 (x 1 ) f 3 (x 3 ) f 6 (x 6 ) f 9 (x 9 ) f 4 (x 4 ) f 7 (x 7 ) f 10 (x 10 ) NN-k converges linearly always and exhibits a quadratic phase in a range Aryan Mokhtari Efficient Methods for Large-Scale Optimization 33

34 (Approximate) Network Newton (NN) Reinterpret distributed gradient descent (DGD) as a penalty method Newton step for objective + a penalty requires global coordination Approximate with local operations by truncating Taylor s series of Hessian inverse kth term of series is k-hop neighbor sparse NN-k aggregates information from k-hop neighborhood NN-1 NN-2 f 2 (x 2 ) f 5 (x 5 ) f 8 (x 8 ) f 1 (x 1 ) f 3 (x 3 ) f 6 (x 6 ) f 9 (x 9 ) f 4 (x 4 ) f 7 (x 7 ) f 10 (x 10 ) NN-k converges linearly always and exhibits a quadratic phase in a range Aryan Mokhtari Efficient Methods for Large-Scale Optimization 34

35 (Approximate) Network Newton (NN) Reinterpret distributed gradient descent (DGD) as a penalty method Newton step for objective + a penalty requires global coordination Approximate with local operations by truncating Taylor s series of Hessian inverse kth term of series is k-hop neighbor sparse NN-k aggregates information from k-hop neighborhood NN-1 NN-2 NN-3 f 2 (x 2 ) f 5 (x 5 ) f 8 (x 8 ) f 1 (x 1 ) f 3 (x 3 ) f 6 (x 6 ) f 9 (x 9 ) f 4 (x 4 ) f 7 (x 7 ) f 10 (x 10 ) NN-k converges linearly always and exhibits a quadratic phase in a range Aryan Mokhtari Efficient Methods for Large-Scale Optimization 35

36 Decentralized Gradient Descent (DGD) Problem in distributed form min x 1,...,x n n f i (x i ), i=1 s.t. x i = x j, for j N i With nonnegative doubly stochastic weights w ij, DGD update at node i is x i,t+1 = j N i w ij x j,t α f i (x i,t ) Average of local and neighboring variables + local gradient descent Rewrite DGD in vector form (aggregate variable y := [x 1;... ; x n]), y t+1 = Zy t αh(y t) Weight matrix W, Z := W I. Gradient h(y) := [ f 1(x 1);... ; f n(x n)] Reorder terms in vector form DGD y t+1 = y t [ (I Z)y t + αh(y t) ] Gradient descent on function F (y) := 1 n 2 yt (I Z) y + α f i (x i ) i=1 Aryan Mokhtari Efficient Methods for Large-Scale Optimization 36

37 DGD as a penalty method (reconsidering the mystery of DGD) n Why do gradient descent on F (y) := α f i (x i ) yt (I Z) y? i=1 (I Z)y = 0 if and only if x i = x j Same is true of (I Z) 1/2 problem in distributed form is equivalent to min y n f i (x i ), s.t. (I Z) 1/2 y = 0 i=1 DGD is a penalty method that solves this (equivalent) problem Squared norm penalty 1 2 (I Z)1/2 y 2 with coefficient 1/α Converges to wrong solution. Not far from right if α is small Gradient descent in F (y) works Why not using Newton steps on F (y)? Aryan Mokhtari Efficient Methods for Large-Scale Optimization 37

38 Newton method for penalized objective function Penalized objective function F (y) = 1 n 2 yt (I Z)y + α f i (x i ) i=1 To implement Newton on F (y) need Hessians H t = I Z + αg t G t is block diagonal with blocks G ii,t = 2 f i (x i,t ) Hessian H t has the sparsity pattern of Z = sparsity pattern of graph Can be computed with local information + exchanges with neighbors Newton step depends on Hessian inverse d t := H 1 t g t Inverse of H t is, in general, not block sparse nor locally computable Aryan Mokhtari Efficient Methods for Large-Scale Optimization 38

39 Network Newton Hessian approximation Define diagonal matrix D t := αg t + 2(I diag(z)) Define block graph sparse matrix B := I 2diag(Z) + Z ( ) Split Hessian as H t = D t B = D 1/2 t I D 1/2 t BD 1/2 t D 1/2 t Use Taylor series (I X) 1 = k=0 Xk to write Hessian inverse as H 1 t = D 1/2 t k=0 ( ) kd D 1/2 t BD 1/2 1/2 t t Define NN-K step d (K) t := Ĥ(K) 1 t g t by truncating sum at Kth term Ĥ (K) 1 t := D 1/2 t K k=0 ( ) k D 1/2 t BD 1/2 1/2 t D t ( ) k D 1/2 t BD 1/2 t graph sparse D 1/2 t BD 1/2 t k-hop neighborhood sparse Aryan Mokhtari Efficient Methods for Large-Scale Optimization 39

40 Distributed computation of NN-K step The descent direction is d (K) t := Ĥ(K) 1 t g t If we define d (K) i,t as the local part of d (K) t d (k+1) i,t = D 1 ii,t j N i,j=i at node i, then B ij d (k) j,t D 1 ii,t g i,t Local piece of NN-(k + 1) step is computed as a function of Local matrices, local gradient components, local piece of NN-k step Pieces of the NN-k step of neighboring nodes. Can exchange. Aryan Mokhtari Efficient Methods for Large-Scale Optimization 40

41 NN-K Algorithm at node i (0) Initialize at x i,0. Repeat for times t = 0, 1,... (1) Exchange local iterates x i,t with neighboring nodes j N i. (2) Compute local gradient components g i,t = (1 w ii )x i,t j N i w ij x j,t + α f i (x i,t ) (3) Initialize NN step computation with NN-0 step d (0) i,t (4) Repeat for k = 0, 1,..., K 1 = D 1 ii,t g i,t (5) Exchange local elements d (k) i,t of NN-k step with neighbors j N i. (6) Compute local component of NN-(k + 1) step d (k+1) i,t = D 1 ii,t j N i,j=i (7) Update local iterate: x i,t+1 = x i,t + ɛ d (K) i,t B ij d (k) j,t D 1 ii,t g i,t Aryan Mokhtari Efficient Methods for Large-Scale Optimization 41

42 Assumptions Assumption 1 The local objective functions f i (x) are twice differentiable The Hessians 2 f i (x) have bounded eigenvalues mi 2 f i (x) MI Assumption 2 The local Hessians are Lipschitz continuous 2 f i (x) 2 f i (ˆx) L x ˆx Aryan Mokhtari Efficient Methods for Large-Scale Optimization 42

43 Linear convergence of NN-K Theorem For a specific choice of stepsize ɛ the sequence F (y t) converges to the optimal argument F (y ) at least linearly with constant 0 < 1 ζ α < 1, i.e., F (y t) F (y ) (1 ζ α) t (F (y 0) F (y )) Aryan Mokhtari Efficient Methods for Large-Scale Optimization 43

44 Superlinear convergence Theorem Theorem For specific values of Γ 1 and Γ 2 the sequence of weighted gradient norm satisfies ] D 1/2 t g t+1 (1 ɛ+ɛρ K+1 ) [1 + Γ 1(1 ζ) t 1 4 D 1/2 +ɛ t 1 gt 2 D 1/2 2 Γ 2 t 1 gt where ρ < 1. D 1 2 t g t+1 is upper bounded by linear and quadratic terms of D 1 2 Similar to the convergence analysis of Newton method with constant ɛ For t large enough Γ 1(1 ζ) t K large enough ɛ 1 linear term becomes smaller t 1 gt There is an interval in which the quadratic term dominates the linear term Rate of convergence is quadratic in that interval Aryan Mokhtari Efficient Methods for Large-Scale Optimization 44

45 Numerical results Convergence path for f (x) := 100 i=1 xt A i x/2 + b T i x 100 i=1 A i is ill-conditioned, Graph is 4 regular and α = 10 2 xi x 2 x 2 error = 1 n n i= DGD NN-0 NN-1 NN Number of iterations DGD is slower than different versions of NN-K NN-K with larger K converges faster in terms of number of iterations Aryan Mokhtari Efficient Methods for Large-Scale Optimization 45

46 Numerical results Number of required information exchanges to achieve accuracy e = 10 2 n = 100, d = {4,..., 10} and c.n.= {10 2, 10 3, 10 4 } Empirical distribution Empirical distribution DGD Number of information exchanges NN Number of information exchanges Empirical distribution Empirical distribution NN-0 Number of information exchanges NN Number of information exchanges Different versions of NN-k have almost similar performances DGD is slower than all versions of NN-K by an order of magnitude Aryan Mokhtari Efficient Methods for Large-Scale Optimization 46

47 Proposed research: Quasi-Newton extension Introduction Stochastic quasi-newton algorithms Proposed research: Incremental aggregated BFGS Distributed methods Proposed research: Quasi-Newton extension Proposed research: Stochastic decentralized methods Adaptive sample size algorithms Proposed research: Ada Newton Conclusions Aryan Mokhtari Efficient Methods for Large-Scale Optimization 47

48 Proposed research NN methods improve the curvature approximation in DGD They accelerate the convergence rate of DGD Quadratic phase However, they suffer from two major issues (I-1): They require computation of the Hessian (I-2): They converge to a neighborhood of the solution Solution to (I-1): Decentralized quasi-newton methods QN methods only require the computation of the objective gradients Solution to (I-2): Dual domain methods Dual methods converge to the exact solution using the Lagrangian idea Goal: Develop a distributed QN method that operates in the dual domain Aryan Mokhtari Efficient Methods for Large-Scale Optimization 48

49 Proposed research: Stochastic decentralized methods Introduction Stochastic quasi-newton algorithms Proposed research: Incremental aggregated BFGS Distributed methods Proposed research: Quasi-Newton extension Proposed research: Stochastic decentralized methods Adaptive sample size algorithms Proposed research: Ada Newton Conclusions Aryan Mokhtari Efficient Methods for Large-Scale Optimization 49

50 Proposed research Decentralized methods reduce the computational complexity and storage By a factor proportional to the number of processors (nodes) However, the No. of samples assigned to each node could be still massive Nodes can not afford the computational complexity of det. methods Solution: Stochastic approximation Approach 1: Use stochastic gradients in lieu of gradients Noise of stochastic approx. does not vanish Slow convergence Approach 2: Use variance reduction methods instead of gradients Noise of stoch. approx. vanishes Keeps the rate of det. methods Goal: Develop stochastic distributed methods that leverage variance reduction techniques Aryan Mokhtari Efficient Methods for Large-Scale Optimization 50

51 Numerical results Comparison of a saddle point method (EXTRA) and its stochastic approximations LR problem n = 20 nodes, N = 500 sample points q n = Error x t x sto-extra α = 10 3 sto-extra α = 10 2 DSA α = EXTRA α = Error x t x sto-extra α = 10 2 sto-extra α = 10 3 EXTRA α = DSA α = Number of itrations t Number of gradient evaluations DSA is the only stochastic decentralized method with linear converg. rate DSA requires less samples relative to EXTRA to achieve a target accuracy For e t = 10 10, each node in DSA requires 460 grad. comps. For e t = 10 10, each node in EXTRA requires 2000 = grad. comps. Aryan Mokhtari Efficient Methods for Large-Scale Optimization 51

52 Adaptive sample size algorithms Introduction Stochastic quasi-newton algorithms Proposed research: Incremental aggregated BFGS Distributed methods Proposed research: Quasi-Newton extension Proposed research: Stochastic decentralized methods Adaptive sample size algorithms Proposed research: Ada Newton Conclusions Aryan Mokhtari Efficient Methods for Large-Scale Optimization 52

53 Back to the ERM problem Our original goal was to solve the statistical loss problem w := argmin w R p L(w) = argmin E [f (w, Z)] w R p But since the distribution of Z is unknown we settle for the ERM problem w N 1 := argmin L N (w) = argmin w R p w R p N N f (w, z k ) Where the samples z k are drawn from a common distribution ERM approximates actual problem Don t need perfect solution k=1 Aryan Mokhtari Efficient Methods for Large-Scale Optimization 53

54 Regularized ERM problem From statistical learning we know that there exists a constant V N such that sup L(w) L N (w) V N, w w.h.p. V N = O(1/ N) from CLT. V N = O(1/N) sometimes [Bartlett et al 06] There is no need to minimize L N (w) beyond accuracy O(V N ) This is well known. In fact, this is why we can add regularizers to ERM w N := argmin w R N (w) = argmin L N (w) + cv N w 2 w 2 Adding the term (cv N /2) w 2 moves the optimum of the ERM problem But the optimum w N is still in a ball of order V N around w Aryan Mokhtari Efficient Methods for Large-Scale Optimization 54

55 Adaptive sample size methods Consider ERM problem R n for subset of n N uniformly chosen samples w n := argmin w R n(w) = argmin L n(w) + cvn w 2 w 2 Solutions w m for m samples and w n n samples close if m and n are close They are chosen from the same distribution Find approximate solution w m for ERM problem R m with m samples Increase sample size to n samples Use solution w m as warm start to compute approximate solution w n for R n Aryan Mokhtari Efficient Methods for Large-Scale Optimization 55

56 Adaptive sample size Newton method Ada Newton is a specific adaptive sample size method using Newton steps Find w m that solves R m to its statistical accuracy V m Apply single Newton iteration w n = w m 2 R n(w m) 1 R n(w m) If m and n close, we have w n within statistical accuracy of R n w n w m This works if statistical accuracy ball of R m is within Newton quadratic convergence ball of R n. Then, w m is within Newton quadratic convergence ball of R n A single Newton iteration yields w n within statistical accuracy of R n Aryan Mokhtari Efficient Methods for Large-Scale Optimization 56

57 Numerical results LR problem Protein homology dataset provided (KDD cup 2004) Number of samples N = 145, 751, dimension p = 74 Parameters V n = 1/n, c = 20, m 0 = 124, and α = SGD SAGA Newton Ada Newton RN(w) R N Number of passes Ada Newton achieves the statistical accuracy of the full training set with about two passes over the dataset Can we show that α = 2 is feasible in theory? Aryan Mokhtari Efficient Methods for Large-Scale Optimization 57

58 Proposed research: Ada Newton Introduction Stochastic quasi-newton algorithms Proposed research: Incremental aggregated BFGS Distributed methods Proposed research: Quasi-Newton extension Proposed research: Stochastic decentralized methods Adaptive sample size algorithms Proposed research: Ada Newton Conclusions Aryan Mokhtari Efficient Methods for Large-Scale Optimization 58

59 Proposed research If α is chosen properly, w m will be in the quadratic Nbhd. of Newton s method for the risk R n w n will stay within the statistical accuracy of R n Question: How should we choose α? Computational complexity of Ada Newton O(np 2 + p 3 ) How can we reduce the computational complexity of Ada Newton? There is room for error Don t need exact solution Hessian computation O(np 2 ) Sub-sampled Newton O(τp 2 ) Hessian inverse computation O(p 3 ) SVD truncation O(rp 2 ) Aryan Mokhtari Efficient Methods for Large-Scale Optimization 59

60 Conclusions Introduction Stochastic quasi-newton algorithms Proposed research: Incremental aggregated BFGS Distributed methods Proposed research: Quasi-Newton extension Proposed research: Stochastic decentralized methods Adaptive sample size algorithms Proposed research: Ada Newton Conclusions Aryan Mokhtari Efficient Methods for Large-Scale Optimization 60

61 Conclusions We studied different approaches to solve large-scale ERM problems Stochastic methods Distribute samples over time Distributed (decentralized) methods Distribute samples over processors Distributed stochastic methods Distribute samples over time and processors Adaptive sample size methods Leverage the closeness of solutions The presented SQN methods improve the convergence time of SGD The convergence rate is still slow (sublinear) PR: Incorporate incremental aggregation technique to reduce the noise of SA Aryan Mokhtari Efficient Methods for Large-Scale Optimization 61

62 Conclusions NN methods converge faster (quadratic+linear) than DGD (linear) (I-1) The convergence is inexact (I-2) Implementation of NN requires Hessian computation PR: Develop a quasi-newton method that operates in the dual domain (I-3) Local gradient computation could be costly (same as other dist. methods) PR: Explore the use of variance reduction methods Adaptive sample size version of Newton s method In practice we can double the size of the training set (α = 2) PR: What is the largest valid choice for α in theory? How can we reduce the computational complexity of Ada Newton? PR: Sub-sampled Newton Reduces C.C. of the Hessian computation PR: SVD truncation Reduces C.C. of the Hessian inverse computation Aryan Mokhtari Efficient Methods for Large-Scale Optimization 62

63 References A. Mokhtari and A. Ribeiro, RES: Regularized Stochastic BFGS Algorithm, IEEE Transaction on Signal Processing (TSP), vol. 62, no. 23, pp , December A. Mokhtari and A. Ribeiro, Global Convergence of Online Limited Memory BFGS, Journal of Machine Learning Research (JMLR), vol. 16, pp , A. Mokhtari, Q. Ling, and A. Ribeiro, Network Newton Distributed Optimization Methods, IEEE Transaction on Signal Processing (TSP), vol. 65, no. 1, pp , January A. Mokhtari, and A. Ribeiro, DSA: Decentralized double stochastic averaging gradient algorithm, Journal of Machine Learning Research (JMLR), vol. 17, no. 61, pp. 1-35, Aryan Mokhtari Efficient Methods for Large-Scale Optimization 63