arxiv: v4 [math.oc] 10 Jun 2017

Transcription

1 Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization arxiv: v4 math.oc] 0 Jun 07 Xiangru Lian, Yijun Huang, Yuncheng Li, and Ji Liu {lianxiangru, huangyj0, raingomm, ji.liu.uwisc}@gmail.com Department of Computer Science, University of Rochester Abstract Asynchronous parallel implementations of stochastic gradient SG have been broadly used in solving deep neural network and received many successes in practice recently. However, existing theories cannot explain their convergence and speedup properties, mainly due to the nonconvexity of most deep learning formulations and the asynchronous parallel mechanism. To fill the gaps in theory and provide theoretical supports, this paper studies two asynchronous parallel implementations of SG: one is over a computer network and the other is on a shared memory system. We establish an ergodic convergence rate O/ for both algorithms and prove that the linear speedup is achievable if the number of workers is bounded by is the total number of iterations. Our results generalize and improve existing analysis for convex minimization. Introduction The asynchronous parallel optimization recently received many successes and broad attention in machine learning and optimization Niu et al., 0, Li et al., 03, 04b, Yun et al., 03, Fercoq and Richtárik, 03, Zhang and wok, 04, arecek et al., 04, Tappenden et al., 05, Hong, 04]. It is mainly due to that the asynchronous parallelism largely reduces the system overhead comparing to the synchronous parallelism. The key idea of the asynchronous parallelism is to allow all workers work independently and have no need of synchronization or coordination. The asynchronous parallelism has been successfully applied to speedup many state-of-the-art optimization algorithms including stochastic gradient Niu et al., 0, Agarwal and Duchi, 0, Zhang et al., 04, Feyzmahdavian et al., 05, Paine et al., 03, ania et al., 05], stochastic coordinate descent Avron et al., 04, Liu et al., 04a, Sridhar et al., 03], dual stochastic coordinate ascent Tran et al., 05], and randomized aczmarz algorithm Liu et al., 04b]. Published at NIPS 05.

2 In this paper, we are particularly interested in the asynchronous parallel stochastic gradient algorithm AsySG for nonconvex optimization mainly due to its recent successes and popularity in deep neural network Bengio et al., 003, Dean et al., 0, Paine et al., 03, Zhang et al., 04, Li et al., 04a] and matrix completion Niu et al., 0, Petroni and Querzoni, 04, Yun et al., 03]. While some research efforts have been made to study the convergence and speedup properties of AsySG for convex optimization, people still know very little about its properties in nonconvex optimization. Existing theories cannot explain its convergence and excellent speedup property in practice, mainly due to the nonconvexity of most deep learning formulations and the asynchronous parallel mechanism. People even have no idea if its convergence is certified for nonconvex optimization, although it has been used widely in solving deep neural network and implemented on different platforms such as computer network and shared memory for example, multicore and multigpu system. To fill these gaps in theory, this paper tries to make the first attempt to study AsySG for the following nonconvex optimization problem min f x := E ξ Fx; ξ] x R n where ξ Ξ is a random variable and f x is a smooth but not necessarily convex function. The most common specification is that Ξ is an index set of all training samples Ξ = {,,, N} and Fx; ξ is the loss function with respect to the training sample indexed by ξ. We consider two popular asynchronous parallel implementations of SG: one is for the computer network originally proposed in Agarwal and Duchi, 0] and the other one is for the shared memory including multicore/multigpu system originally proposed in Niu et al., 0]. Note that due to the architecture diversity, it leads to two different algorithms. The key difference lies on that the computer network can naturally also efficiently ensure the atomicity of reading and writing the whole vector of x, while the shared memory system is unable to do that efficiently and usually only ensures efficiency for atomic reading and writing on a single coordinate of parameter x. The implementation on computer cluster is described by the consistent asynchronous parallel SG algorithm AsySG-con, because the value of parameter x used for stochastic gradient evaluation is consistent an existing value of parameter x at some time point. Contrarily, we use the inconsistent asynchronous parallel SG algorithm AsySG-incon to describe the implementation on the shared memory platform, because the value of parameter x used is inconconsistent, that is, it might not be the real state of x at any time point. This paper studies the theoretical convergence and speedup properties for both algorithms. We establish an asymptotic convergence rate of O/ for AsySG-con where is the total iteration number and is the size of minibatch. The linear speedup is proved to be achievable while the number of workers is bounded by O. For AsySG-incon, we establish an asymptotic convergence and speedup properties similar to AsySG-con. The intuition of the linear speedup of asynchronous parallelism for SG can be explained in the following: Recall that the serial The speedup for T workers is defined as the ratio between the total work load using one worker and the average work load using T workers to obtain a solution at the same precision. The linear speedup is achieved means that the speedup with T workers greater than ct for any values of T c 0, ] is a constant independent to T.

3 SG essentially uses the stochastic gradient to surrogate the accurate gradient. AsySG brings additional deviation from the accurate gradient due to using stale or delayed information. If the additional deviation is relatively minor to the deviation caused by the stochastic in SG, the total iteration complexity or convergence rate of AsySG would be comparable to the serial SG, which implies a nearly linear speedup. This is the key reason why AsySG works. The main contributions of this paper are highlighted as follows: Our result for AsySG-con generalizes and improves earlier analysis of AsySG-con for convex optimization in Agarwal and Duchi, 0]. Particularly, we improve the upper bound of the maximal number of workers to ensure the linear speedup from O /4 3/4 to O / / by a factor /4 /4 ; The proposed AsySG-incon algorithm provides a more accurate description than Hogwild! Niu et al., 0] for the lock-free implementation of AsySG on the shared memory system. Although our result does not strictly dominate the result for Hogwild! due to different problem settings, our result can be applied to more scenarios e.g., nonconvex optimization; Our analysis provides theoretical convergence and speedup guarantees for many recent successes of AsySG in deep learning. To the best of our knowledge, this is the first work that offers such theoretical support. Notation x denotes the global optimal solution to. x 0 denotes the l 0 norm of vector x, that is, the number of nonzeros in x; e i R n denotes the ith natural unit basis vector; We use E ξk, to denote the expectation with respect to a set of variables {ξ k,,, ξ k, }; E means taking the expectation in terms of all random variables. Gx; ξ is used to denote Fx; ξ for short; We use i f x and Gx; ξ i to denote the ith element of f x and Gx; ξ respectively. Assumption Throughout this paper, we make the following assumption for the objective function. All of them are quite common in the analysis of stochastic gradient algorithms. Assumption. We assume that the following holds: Unbiased Gradient: The stochastic gradient Gx; ξ is unbiased, that is to say, f x = E ξ Gx; ξ] 3

4 Bounded Variance: The variance of stochastic gradient is bounded: E ξ Gx; ξ f x σ, x. 3 Lipschitzian Gradient: The gradient function f is Lipschitzian, that is to say, f x f y L x y x, y. 4 Under the Lipschitzian gradient assumption, we can define two more constants L s and L max. Let s be any positive integer. Define L s to be the minimal constant satisfying the following inequality: f x f x + α i e i L s α i e i, S {,,..., n} and S s 5 i S i S Define L max as the minimum constant that satisfies: It can be seen that L max L s L. i f x i f x + αe i L max α, i {,,..., n}. 6 Related Work This section reviews stochastic gradient algorithms, synchronous parallel stochastic gradient algorithms, asynchronous parallel gradient algorithms, and asynchronous parallel stochastic gradient algorithms. Stochastic gradient is a very powerful approach for large scale optimization. The well known convergence rate of stochastic gradient is O/ for convex problems and O/ for strongly convex problems, for example, see Nemirovski et al., 009, oulines and Bach, 0]. A parallel algorithm of stochastic gradient is the dual averaging approach, which achieves the same convergence rate Xiao, 009]. The stochastic gradient is also closely related to online learning algorithms. Please refer to the online learning literature, for example, Crammer et al., 006, Shalev-Shwartz, 0, Yang et al., 04]. ost studies for stochastic gradient focus on convex optimization. For nonconvex optimization using SG, Ghadimi and Lan 03] proved an ergodic convergence rate of O/ for stochastic gradient under nonconvex optimization, which is consistent with the rate of stochastic gradient for convex problems. As for synchronous parallel stochastic gradient methods, Dekel et al. 0] proposed a mini-batch stochastic gradient algorithm multiple workers compute the stochastic gradient on data in parallel and need a synchronization step before modifying parameter x in every iteration. The convergence rate is proven to be O/ for convex optimization. In their follow up work Dekel et al., 00], they showed that a variant of their approach is asymptotically robust to asynchrony, as long as most processors remain synchronized for most of time. Precisely, this variant can be considered as a partially asynchronous parallel stochastic gradient algorithm. 4

5 Zinkevich et al., 00] studied a method where each node in the network runs the vanilla stochastic gradient method, using random subsets of the overall data set, and their solutions are only averaged in the final step. The asynchronous parallel algorithms received broad attention in optimization recently, although pioneer studies started from 980s Bertsekas and Tsitsiklis, 989]. Due to the rapid development of hardware resources, the asynchronous parallelism recently received many successes when applied to parallel stochastic gradient Niu et al., 0, Agarwal and Duchi, 0, Zhang et al., 04, Feyzmahdavian et al., 05, Paine et al., 03], stochastic coordinate descent Avron et al., 04, Liu et al., 04a], dual stochastic coordinate ascent Tran et al., 05], randomized aczmarz algorithm Liu et al., 04b], and AD Zhang and wok, 04]. Liu et al. 04a] and Liu and Wright 04] studied the asynchronous parallel stochastic coordinate descent algorithm with consistent read and inconsistent read respectively and prove the linear speedup is achievable if T On / for smooth convex functions and T On /4 for functions with smooth convex loss + nonsmooth convex separable regularization. Avron et al. 04] studied this asynchronous parallel stochastic coordinate descent algorithm in solving Ax = b where A is a symmetric positive definite matrix, and showed that the linear speedup is achievable if T On for consistent read and T On / for inconsistent read. Tran et al. 05] studied a semi-asynchronous parallel version of Stochastic Dual Coordinate Ascent algorithm which periodically enforces primal-dual synchronization in a separate thread. We review the asynchronous parallel stochastic gradient algorithms in the last. Agarwal and Duchi 0] analyzed the AsySG-con algorithm on computer cluster for convex smooth optimization and proved a convergence rate of O + T which implies that linear speedup is achieved when T is bounded by O /4 / 3/4. In comparison, our analysis for the more general nonconvex smooth optimization improves the upper bound by a factor /4 /4. A very recent work Feyzmahdavian et al., 05] extended the analysis in Agarwal and Duchi 0] to minimize functions in the form smooth convex loss + nonsmooth convex regularization and obtained similar results. Niu et al. 0] proposed a lock free asynchronous parallel implementation of SG on the shared memory system and described this implementation as Hogwild! algorithm. They proved a sublinear convergence rate O/ for strongly convex smooth objectives. Another recent work ania et al. 05] analyzed asynchronous stochastic optimization algorithms for convex functions by viewing it as a serial algorithm with the input perturbed by bounded noise and proved the convergences rates no worse than using traditional point of view for several algorithms. 3 Asynchronous parallel stochastic gradient for computer network This section considers the asynchronous parallel implementation of SG on computer network proposed by Agarwal and Duchi 0]. It has been successfully applied to the distributed neural network Dean et al., 0] and the parameter server Li et al., 04a] to solve deep neural network. 5

6 3. Algorithm Description: AsySG-con Algorithm AsySG-con Require: x 0,, {γ k } k=0,, Ensure: x : for k = 0,, do : Randomly select training samples indexed by ξ k,, ξ k,,...ξ k, ; 3: x k+ = x k γ k Gx k τ k,m, ξ k,m ; 4: end for The star in the star-shaped network is a master machine which maintains the parameter x. Other machines in the computer network serve as workers which only communicate with the master. All workers exchange information with the master independently and simultaneously, basically repeating the following steps: Select: randomly select a subset of training samples S Ξ; Pull: pull parameter x from the master; Compute: compute the stochastic gradient g ξ S Gx; ξ; Push: push g to the master. The master basically repeats the following steps: Aggregate: aggregate a certain amount of stochastic gradients g from workers; Sum: summarize all g s into a vector ; Update: update parameter x by x x γ. While the master is aggregating stochastic gradients from workers, it does not care about the sources of the collected stochastic gradients. As long as the total amount achieves the predefined quantity, the master will compute and perform the update on x. The update step is performed as an atomic operation workers cannot read the value of x during this step, which can be efficiently implemented in the network especially in the parameter server Li et al., 04a]. The key difference between this asynchronous parallel implementation of SG and the serial or synchronous parallel SG algorithm lies on that in the update step, some stochastic gradients g in might be computed from some early value of x instead of the current one, while in the serial SG, all g s are guaranteed to use the current value of x. There could be more than one machines in some networks, but all of them serves the same purpose and can be treated as a single machine. 6

7 The asynchronous parallel implementation substantially reduces the system overhead and overcomes the possible large network delay, but the cost is to use the old value of x in the stochastic gradient evaluation. We will show in Section 3. that the negative affect of this cost will vanish asymptotically. To mathematically characterize this asynchronous parallel implementation, we monitor parameter x in the master. We use the subscript k to indicate the kth iteration on the master. For example, x k denotes the value of parameter x after k updates, so on and so forth. We introduce a variable τ k,m to denote how many delays for x used in evaluating the mth stochastic gradient at the kth iteration. This asynchronous parallel implementation of SG on the star-shaped network is summarized by the AsySG-con algorithm, see Algorithm. The suffix con is short for consistent read. Consistent read means that the value of x used to compute the stochastic gradient is a real state of x no matter at which time point. Consistent read is ensured by the atomicity of the update step. When the atomicity fails, it leads to inconsistent read which will be discussed in Section 4. It is worth noting that on some non-star structures the asynchronous implementation can also be described as AsySG-con in Algorithm, for example, the cyclic delayed architecture and the locally averaged delayed architecture Agarwal and Duchi, 0, Figure ]. 3. Analysis for AsySG-con To analyze Algorithm, besides Assumption we make the following additional assumptions. Assumption. We assume that the following holds: Independence: All random variables in {ξ k,m } k=0,,,;,, in Algorithm are independent to each other; Bounded Age: All delay variables τ k,m s are bounded: max k,m τ k,m T. The independence assumption strictly holds if all workers select samples with replacement. Although it might not be satisfied strictly in practice, it is a common assumption made for the analysis purpose. The bounded delay assumption is much more important. As pointed out before, the asynchronous implementation may use some old value of parameter x to evaluate the stochastic gradient. Intuitively, the age or oldness should not be too large to ensure the convergence. Therefore, it is a natural and reasonable idea to assume an upper bound for ages. This assumption is commonly used in the analysis for asynchronous algorithms, for example, Niu et al., 0, Avron et al., 04, Liu and Wright, 04, Liu et al., 04a, Feyzmahdavian et al., 05, Liu et al., 04b]. It is worth noting that the upper bound T is roughly proportional to the number of workers. Under Assumptions and, we have the following convergence rate for nonconvex optimization. 7

8 Theorem. Assume that Assumptions and hold and the steplength sequence {γ k },, in Algorithm satisfies Lγ k + L Tγ k T κ= γ k+κ for all k =,,... 7 We have the following ergodic convergence rate for the iteration of Algorithm γ k γ k E f x k f x f x + γ k L + L γ k k j=k T γ j γ k where E denotes taking expectation in terms of all random variables in Algorithm. σ. 8 To evaluate the convergence rate, the commonly used metrics in convex optimization are not eligible, for example, f x k f and x k x. For nonsmooth optimization, we use the ergodic convergence as the metric, that is, the weighted average of the l norm of all gradients f x k, which is used in the analysis for nonconvex optimization Ghadimi and Lan, 03]. Although the metric used in nonconvex optimization is not exactly comparable to f x k f or x k x used in the analysis for convex optimization, it is not totally unreasonable to think that they are roughly in the same order. The ergodic convergence directly indicates the following convergence: If randomly select an index from {,,, } with probability {γ k / γ k}, then E f x is bounded by the right hand side of 8 and all bounds we will show in the following. Taking a close look at Theorem, we can properly choose the steplength γ k as a constant value and obtain the following convergence rate: Corollary. Assume that Assumptions and hold. Set the steplength γ k to be a constant γ If the delay parameter T is bounded by γ := f x f x Lσ. 9 4L f x f x σ T +, 0 then the output of Algorithm satisfies the following ergodic convergence rate min E f x k f E f x k {,,} k x f x 4 L σ. This corollary basically claims that when the total iteration number is greater than OT, the convergence rate achieves O/. Since this rate does not depend on the delay parameter T after sufficient number of iterations, the negative effect of using old values of x for stochastic gradient evaluation vanishes asymptoticly. In other words, if the total number of workers is bounded by O /, the linear speedup is achieved. 8

9 Note that our convergence rate O/ is consistent with the serial SG with = for convex optimization Nemirovski et al., 009], the synchronous parallel or mini-batch SG for convex optimization Dekel et al., 0], and nonconvex smooth optimization Ghadimi and Lan, 03]. Therefore, an important observation is that as long as the number of workers which is proportional to T is bounded by O /, the iteration complexity to achieve the same accuracy level will be roughly the same. In other words, the average work load for each worker is reduced by the factor T comparing to the serial SG. Therefore, the linear speedup is achievable if T O /. Since our convergence rate meets several special cases, it is tight. Next we compare with the analysis of AsySG-con for convex smooth optimization in Agarwal and Duchi 0, Corollary ]. They proved an asymptotic convergence rate O/, which is consistent with ours. But their results require T O /4 3/4 to guarantee linear speedup. Our result improves it by a factor O /4 /4. 4 Asynchronous parallel stochastic gradient for shared memory architecture This section considers a widely used lock-free asynchronous implementation of SG on the shared memory system proposed in Niu et al. 0]. Its advantages have been witnessed in solving SV, graph cuts Niu et al., 0], linear equations Liu et al., 04b], and matrix completion Petroni and Querzoni, 04]. While the computer network always involves multiple machines, the shared memory platform usually only includes a single machine with multiple cores / GPUs sharing the same memory. 4. Algorithm Description: AsySG-incon Algorithm AsySG-incon Require: x 0,, γ Ensure: x : for k = 0,, do : Randomly select training samples indexed by ξ k,, ξ k,,...ξ k, ; 3: Randomly select i k {,,..., n} with uniform distribution; 4: x k+ ik = x k ik γ G ˆx k,m ; ξ k,m ik ; 5: end for For the shared memory platform, one can exactly follow AsySG-con on the computer network using software locks, which is expensive 3. Therefore, in practice the lock free asynchronous 3 The time consumed by locks is roughly equal to the time of 0 4 floating-point computation. The additional cost for 9

10 parallel implementation of SG is preferred. This section considers the same implementation as Niu et al. 0], but provides a more precise algorithm description AsySG-incon than Hogwild! proposed in Niu et al. 0]. In this lock free implementation, the shared memory stores the parameter x and allows all workers reading and modifying parameter x simultaneously without using locks. All workers repeat the following steps independently, concurrently, and simultaneously: Read: read the parameter from the shared memory to the local memory without software locks we use ˆx to denote its value; Compute: sample a training data ξ and use ˆx to compute the stochastic gradient G ˆx; ξ locally; Update: update parameter x in the shared memory without software locks x x γg ˆx; ξ. Since we do not use locks in both read and update steps, it means that multiple workers may manipulate the shared memory simultaneously. It causes the inconsistent read at the read step, that is, the value of ˆx read from the shared memory might not be any state of x in the shared memory at any time point. For example, at time 0, the original value of x in the shared memory is a two dimensional vector a, b]; at time, worker W is running the read step and first reads a from the shared memory; at time, worker W updates the first component of x in the shared memory from a to a ; at time, worker W updates the second component of x in the shared memory from b to b ; at time 3, worker W reads the value of the second component of x in the shared memory as b. In this case, worker W eventually obtains the value of ˆx as a, b ], which is not a real state of x in the shared memory at any time point. Recall that in AsySG-con the parameter value obtained by any worker is guaranteed to be some real value of parameter x at some time point. To precisely characterize this implementation and especially represent ˆx, we monitor the value of parameter x in the shared memory. We define one iteration as a modification on any single component of x in the shared memory since the update on a single component can be considered to be atomic on GPUs and DSPs Niu et al., 0]. We use x k to denote the value of parameter x in the shared memory after k iterations and ˆx k to denote the value read from the shared memory and used for computing stochastic gradient at the kth iteration. ˆx k can be represented by x k with a few earlier updates missing ˆx k = x k x j+ x j j Jk where Jk {k, k,, 0} is a subset of index numbers of previous iterations. This way is also used in analyzing asynchronous parallel coordinate descent algorithms in Avron et al., 04, Liu and Wright, 04]. The kth update happened in the shared memory can be described as x k+ ik = x k ik γg ˆx k ; ξ k ik using locks is the waiting time during which multiple worker access the same memory address. 0

11 where ξ k denotes the index of the selected data and i k denotes the index of the component being updated at kth iteration. In the original analysis for the Hogwild! implementation Niu et al., 0], ˆx k is assumed to be some earlier state of x in the shared memory that is, the consistent read for simpler analysis, although it is not true in practice. One more complication is to apply the mini-batch strategy like before. Since the update step needs physical modification in the shared memory, it is usually much more time consuming than both read and compute steps are. If many workers run the update step simultaneously, the memory contention will seriously harm the performance. To reduce the risk of memory contention, a common trick is to ask each worker to gather multiple say stochastic gradients and write the shared memory only once. That is, in each cycle, run both update and compute steps for times before you run the update step. Thus, the mini-batch updates happen in the shared memory can be written as x k+ ik = x k ik γ G ˆx k,m ; ξ k,m ik 3 where i k denotes the coordinate index updated at the kth iteration, and G ˆx k,m ; ξ k,m is the mth stochastic gradient computed from the data sample indexed by ξ k,m and the parameter value denoted by ˆx k,m at the kth iteration. ˆx k,m can be expressed by: ˆx k,m = x k x j+ x j 4 j Jk,m where Jk, m {k, k,, 0} is a subset of index numbers of previous iterations. The algorithm is summarized in Algorithm from the view of the shared memory. 4. Analysis for AsySG-incon To analyze the AsySG-incon, we need to make a few assumptions similar to Niu et al. 0], Liu et al. 04b], Avron et al. 04], Liu and Wright 04]. Assumption 3. We assume that the following holds for Algorithm : Independence: All groups of variables {i k, {ξ k,m } } at different iterations from k = to are independent to each other. Bounded Age: Let T be the global bound for delay: Jk, m {k,...k T}, Jk, m T. k, m, so The independence assumption might not be true in practice, but it is probably the best assumption one can make in order to analyze the asynchronous parallel SG algorithm. This assumption was also used in the analysis for Hogwild! Niu et al., 0] and asynchronous randomized

12 aczmarz algorithm Liu et al., 04b]. The bounded delay assumption basically restricts the age of all missing components in ˆx k,m m, k. The upper bound T here serves a similar purpose as in Assumption. Thus we abuse this notation in this section. The value of T is proportional to the number of workers and does not depend on the size of mini-batch. The bounded age assumption is used in the analysis for asynchronous stochastic coordinate descent with inconsistent read Avron et al., 04, Liu and Wright, 04]. Under Assumptions and 3, we have the following results: Theorem 3. Assume that Assumptions and 3 hold and the constant steplength γ satisfies TL T n + T γ We have the following ergodic convergence rate for Algorithm n 3/ + L max γ. 5 t= E f x t n γ f x f x + L T Tγ σ + L max γσ. 6 n Taking a close look at Theorem 3, we can choose the steplength γ properly and obtain the following error bound: Corollary 4. Assume that Assumptions and 3 hold. Set the steplength to be a constant γ γ := f x f x n. 7 LT σ If the total iterations is greater than 6 f x f x L T n 3/ + 4T, 8 nσ then the output of Algorithm satisfies the following ergodic convergence rate 7 f E f x k x f x L T n σ. 9 This corollary indicates the asymptotic convergence rate achieves O/ when the total iteration number exceeds a threshold in the order of OT if n is considered as a constant. We can see that this rate and the threshold are consistent with the result in Corollary for AsySG-con. One may argue that why there is an additional factor n in the numerator of 9. That is due to the way we count iterations one iteration is defined as updating a single component of x. If we take into account this factor in the comparison to AsySG-con, the convergence rates for AsySG-con and AsySG-incon are essentially consistent. This comparison implies that the inconsistent read would not make a big difference from the consistent read.

13 Next we compare our result with the analysis of Hogwild! by Niu et al., 0]. In principle, our analysis and their analysis consider the same implementation of asynchronous parallel SG, but differ in the following aspects: our analysis considers the smooth nonconvex optimization which includes the smooth strongly convex optimization considered in their analysis; our analysis considers the inconsistent read model which meets the practice while their analysis assumes the impractical consistent read model. Although the two results are not absolutely comparable, it is still interesting to see the difference. Niu et al. 0] proved that the linear speedup is achievable if the maximal number of nonzeros in stochastic gradients is bounded by O and the number of workers is bounded by On /4. Our analysis does not need this prerequisite and guarantees the linear speedup as long as the number of workers is bounded by O. Although it is hard to say that our result strictly dominates Hogwild! in Niu et al. 0], our asymptotic result is eligible for more scenarios. We consider an extension of AsySG-incon for sparse stochastic gradients. In this scenario, we slightly change Steps 3 and 4 in Algorithm : 3: Uniformly select i k from the support set of g k := G ˆx k,m; ξ k,m ; 4: x k+ ik = x k ik γ g k 0 g k ik. The convergence rate in 9 can be slightly improved by taking the advantage of sparsity: E f x k 6 f x f x L T max k g k 0 σ. 0 The proof can be simply get by extending the proof for 9. 5 Experiments The successes of AsySG-con and AsySG-incon and their advantages over synchronous parallel algorithms have been widely witnessed in many applications such as deep neural network Dean et al., 0, Paine et al., 03, Zhang et al., 04, Li et al., 04a], matrix completion Niu et al., 0, Petroni and Querzoni, 04, Yun et al., 03], SV Niu et al., 0], and solving linear equations Liu et al., 04b]. We refer readers to these literatures for more comphrehensive comparison and empirical studies. This section mainly provides the empirical study to validate the speedup properties for completeness. We perform experiments for AsySG-con and AsySG-incon on computer cluster and multicore machine respectively. The main purpose of the following experiments is to validate the speedup property. We are particularly interested in two types of speedup: iteration speedup and running time speedup. The iteration speedup is exactly the speedup we discussed in the whole paper. 3

14 Given T workers, it is computed from the ratio iteration speedup of T workers = # of total iterations of the serial SG or using one worker # of total iterations using T workers T where # is the iteration count when the same level of precision achieved. This speedup is less affected by the hardware. The running time speedup is the actual speedup. It is defined with respect to the running time: running time speedup of T workers = running time for the serial SG or using one worker. running time of using T workers The running time speedup is seriously affected by the hardware. It is generally worse than the iteration speedup. 5. AsySG-con We implement AsySG-con for deep neural network based on the Caffe Jia et al., 04] package. Caffe is an open source code base of deep learning algorithms. We evaluate AsySG-con on two standard datasets provided in the Caffe package, LENET and CIFAR0-FULL. The neural network consists of convolution layer, nonlinear layer, max pool layer and fully connected layer rizhevsky et al., 0], and the detailed specification can be found on the Caffe website 4. LENET is a digit classifier network, training on the NIST dataset 5. CIFAR0-full has 0 classes of color images, training on the CIFAR0 dataset rizhevsky and Hinton, 009]. We first initialize a parameter server hosting the parameters, and then spawn up to 8 stochastic gradient workers. The point to point communication between the parameter server and gradient workers are handled by the PICH library 6. The parameter server and stochastic gradient workers run on separate machines, and each process uses a single core of a XeonR E5-430 CPU. The steplength γ for LENET is chosen as the default value. It means that this steplength has been tuned to be the optimal for the serial SG algorithm on LENET. The steplength we used for CIFAR0-FULL is chosen as the default value as well. We draw the curves of objective loss against iterations and running time in Figures and respectively, and report their speedups in Tables and. ore details about datasets and the parameter setting in experiments can be found in Table 3. We can observe that The iteration speedup is always better than the running time speedup, which meets our common sense; The speedups for both problems are comparable overall as shown in Tables and. The time speedup for CIFAR0-FULL is slightly more stable, while we notice that the time

15 speedup for LENET in Table suddenly drops to.88 with mpi-8 from 5.9 with mpi-7. That is because it hits the ceiling of communication bandwidth. The number of parameters in LENET is more than CIFAR0-FULL, thus requiring more communication cost. When the number of machines achieves a certain threshold in this case LENET, it is 8, the performance might become dramatically worse. loss loss ~ iter mpi- mpi- mpi-3 mpi-4 mpi-5 mpi-6 mpi-7 mpi-8 loss loss ~ ts000s mpi- mpi- mpi-3 mpi-4 mpi-5 mpi-6 mpi-7 mpi iter ts000s Figure : LENET. The AsySG-con algorithm is run on various numbers of machines from to 8 to solve LENET. The curves of the objective loss against the number of iteration and the running time are drawn in the left and the right graphs respectively. loss loss ~ iter mpi- mpi- mpi-3 mpi-4 mpi-5 mpi-6 mpi-7 mpi iter loss loss ~ ts000s mpi- mpi- mpi-3 mpi-4 mpi-5 mpi-6 mpi-7 mpi ts000s Figure : CIFAR0-FULL. The AsySG-con algorithm is run on various numbers of machines from to 8 to solve CIFAR0-FULL. The curves of the objective loss against the number of iteration and the running time are drawn in the left and the right graphs respectively. 5

16 Table : Iteration speedup and running time speedup of AsySG-con. LENET mpi- mpi- mpi-3 mpi-4 mpi-5 mpi-6 mpi-7 mpi-8 iteration speedup time speedup Table : Iteration speedup and running time speedup of AsySG-con. CIFAR0-FULL mpi- mpi- mpi-3 mpi-4 mpi-5 mpi-6 mpi-7 mpi-8 iteration speedup time speedup Table 3: ore details about LENET and CIFAR0-FULL. Datasets Type #Images inibatch #CONV #FC #Params LENET 8x8 grayscale ,080 CIFAR0-FULL 3x3 RGB , AsySG-incon We conduct the empirical study for AsySG-incon on the machine Intel Xeon architecture, which has 4 sockets and 0 cores for each socket. The synthetic data is generated from a full connected neural network with 5 layers and parameters totally. The total number of samples is The data size is about.5 GB. The input vector and all parameters are generated from i.i.d. Gaussian distribution. The output vector is constructed by applying the network parameter to the input vector plus some Gaussian random noise. We run AsySG-incon on various numbers of cores from to 3. The size of mini-batch is chosen as = 3 and the steplength is chosen as γ = Both parameters are chosen based on the best performance of the serial SG to achieve the precision 0 for the l norm of gradient. Figure 3 draws the curve of the l norm of gradients against the number of iterations and running time respectively. The speedup is reported in Table 4. We observe that the iteration speedup is almost linear while the running time speedup is slightly worse than the iteration speedup. The overall performance of AsySG-incon on the shared memory system is better than AsySG-con on the computer cluster. The reason is that the computer cluster suffers from serious communication delay and the delay bound T on the computer cluster is usually larger than on the shared memory system. 6

17 gradient Iteration Gradient 3 threads 8 threads 4 threads 0 threads 6 threads threads 8 threads 4 threads thread gradient Time Gradient 3 thread 8 threads 4 threads 0 threads 6 threads threads 8 threads 4 threads threads iterations x timeseconds Figure 3: Deep neural network for synthetic data using AsySG-incon. The AsySG-con algorithm is run on various numbers of machines from to 3. The curves of the objective loss against the number of iteration and the running time are drawn in the left and the right graphs respectively. Table 4: Iteration speedup and running time speedup of AsySG-incon synthetic data. thr- thr-4 thr-8 thr- thr-6 thr-0 thr-4 thr-8 thr-3 iteration speedup time speedup

18 6 Conclusion This paper studied two popular asynchronous parallel implementations for SG on computer cluster and shared memory system respectively. Two algorithms AsySG-con and AsySG-incon are used to describe two implementations. An asymptotic sublinear convergence rate is proven for both algorithms on nonconvex smooth optimization. This rate is consistent with the result of SG for convex optimization. The linear speedup is proven to achievable when the number of workers is bounded by, which improves the earlier analysis of AsySG-con for convex optimization in Agarwal and Duchi, 0]. The proposed AsySG-incon algorithm provides a more precise description for lock free implementation on shared memory system than Hogwild! Niu et al., 0]. Our result for AsySG-incon can be applied to more scenarios. Acknowledgements This project is supported by the NSF grant CNS , the NEC fellowship, and the startup funding at University of Rochester. We thank Professor Daniel Gildea and Professor Sandhya Dwarkadas at University of Rochester, Professor Stephen J. Wright at University of Wisconsin- adison, and anonymous meta-reviewers for their constructive comments and helpful advices. References A. Agarwal and J. C. Duchi. Distributed delayed stochastic optimization. NIPS, 0. H. Avron, A. Druinsky, and A. Gupta. Revisiting asynchronous linear solvers: Provable convergence rate through randomization. IPDPS, 04. Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin. A neural probabilistic language model. The Journal of achine Learning Research, 3:37 55, 003. D. P. Bertsekas and J. N. Tsitsiklis. Parallel and distributed computation: numerical methods, volume 3. Prentice hall Englewood Cliffs, NJ, Crammer, O. Dekel, J. eshet, S. Shalev-Shwartz, and Y. Singer. Online passive-aggressive algorithms. Journal of achine Learning Research, 7:55 585, 006. J. Dean, G. Corrado, R. onga,. Chen,. Devin,. ao, A. Senior, P. Tucker,. Yang, Q. V. Le, et al. Large scale distributed deep networks. NIPS, 0. O. Dekel, R. Gilad-Bachrach, O. Shamir, and L. Xiao. Robust distributed online prediction. arxiv preprint arxiv:0.370, 00. O. Dekel, R. Gilad-Bachrach, O. Shamir, and L. Xiao. Optimal distributed online prediction using mini-batches. Journal of achine Learning Research, 3:65 0, 0. 8

19 O. Fercoq and P. Richtárik. Accelerated, parallel and proximal coordinate descent. arxiv preprint arxiv:3.5799, 03. H. R. Feyzmahdavian, A. Aytekin, and. Johansson. An asynchronous mini-batch algorithm for regularized stochastic optimization. ArXiv e-prints, ay S. Ghadimi and G. Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIA Journal on Optimization, 34:34 368, 03.. Hong. A distributed, asynchronous and incremental algorithm for nonconvex optimization: An AD based approach. arxiv preprint arxiv:4.6058, 04. Y. Jia, E. Shelhamer, J. Donahue, S. arayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arxiv preprint arxiv: , 04. A. rizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Computer Science Department, University of Toronto, Tech. Rep, 4:7, 009. A. rizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. NIPS, pages , 0.. Li, L. Zhou, Z. Yang, A. Li, F. Xia, D. G. Andersen, and A. Smola. Parameter server for distributed machine learning. Big Learning NIPS Workshop, 03.. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su. Scaling distributed machine learning with the parameter server. OSDI, 04a.. Li, D. G. Andersen, A. J. Smola, and. Yu. Communication efficient distributed machine learning with the parameter server. NIPS, 04b. J. Liu and S. J. Wright. Asynchronous stochastic coordinate descent: Parallelism and convergence properties. arxiv preprint arxiv: , 04. J. Liu, S. J. Wright, C. Ré, V. Bittorf, and S. Sridhar. An asynchronous parallel stochastic coordinate descent algorithm. ICL, 04a. J. Liu, S. J. Wright, and S. Sridhar. An asynchronous parallel randomized kaczmarz algorithm. arxiv preprint arxiv: , 04b. H. ania, X. Pan, D. Papailiopoulos, B. Recht,. Ramchandran, and. I. Jordan. Perturbed iterate analysis for asynchronous stochastic optimization. arxiv preprint arxiv: , 05. J. arecek, P. Richtárik, and. Takác. Distributed block coordinate descent for minimizing partially separable functions. arxiv preprint arxiv: , 04. E. oulines and F. R. Bach. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. NIPS, 0. 9

20 A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIA Journal on Optimization, 94: , 009. F. Niu, B. Recht, C. Re, and S. Wright. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. NIPS, 0. T. Paine, H. Jin, J. Yang, Z. Lin, and T. Huang. Gpu asynchronous stochastic gradient descent to speed up neural network training. NIPS, 03. F. Petroni and L. Querzoni. Gasgd: stochastic gradient descent for distributed asynchronous matrix completion via graph partitioning. AC Conference on Recommender systems, 04. S. Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends in achine Learning, 4:07 94, 0. S. Sridhar, S. Wright, C. Re, J. Liu, V. Bittorf, and C. Zhang. An approximate, efficient LP solver for lp rounding. NIPS, 03. R. Tappenden,. Takáč, and P. Richtárik. On the complexity of parallel coordinate descent. arxiv preprint arxiv: , 05.. Tran, S. Hosseini, L. Xiao, T. Finley, and. Bilenko. Scaling up stochastic dual coordinate ascent. ICL, 05. L. Xiao. Dual averaging method for regularized stochastic learning and online optimization. NIPS, 009. T. Yang,. ahdavi, R. Jin, and S. Zhu. Regret bounded by gradual variation for online convex optimization. achine learning, 95:83 3, 04. H. Yun, H.-F. Yu, C.-J. Hsieh, S. Vishwanathan, and I. Dhillon. Nomad: Non-locking, stochastic multi-machine algorithm for asynchronous and decentralized matrix completion. arxiv preprint arxiv:3.093, 03. R. Zhang and J. wok. Asynchronous distributed AD for consensus optimization. ICL, 04. S. Zhang, A. Choromanska, and Y. LeCun. Deep learning with elastic averaging SGD. CoRR, abs/4.665, 04.. Zinkevich,. Weimer, L. Li, and A. J. Smola. Parallelized stochastic gradient descent. NIPS, 00. 0

21 Appendix: Proofs Proofs to Theorem Proof. From the Lipschitzisan gradient assumption 5, we have f x k+ f x k f x k, x k+ x k + L x k+ x k = f x k, γ k Gx k τk,m ; ξ k,m + γ k L Gx k τk,m ; ξ k,m. Taking expectation respect to ξ k, on both sides of, we have E ξk, f x k+ f x k γ k f x k, f x k τk,m + γ k L E ξ k, Gx k τk,m ; ξ k,m where we use the unbiased stochastic gradient assumption in. From the fact we have E ξk, f x k+ f x k γ k + γ k L E ξ k, a, b = a + b a b, f x k + f x k τk,m f x k f x k τk,m }{{} T Gx k τk,m ; ξ k,m. 3 }{{} T Next we estimate the upper bound of T and T. For T we have T =E ξk, G x k τk,m ; ξ k,m =E ξk, Gx k τk,m ; ξ k,m f x k τk,m + f x k τk,m

22 =E ξk, Gx k τk,m ; ξ k,m f x k τk,m ] + f x k τk,m + Gx k τk,m ; ξ k,m f x k τk,m, f x k τk,m =E ξk, Gx k τk,m ; ξ k,m f x k τk,m + f x k τk,m =E ξk, + m<m Gx k τk,m ; ξ k,m f x k τk,m ] + f x k τk,m σ + f x k τk,m where the forth equality is due to = Gx k τk,m ; ξ k,m f x k τk,m, Gx k τk,m ; ξ k,m f x k τk,m Gx k τk,m ; ξ k,m f x k τk,m E ξk, E ξk, Gx k τk,m ; ξ k,m f x k τk,m =0,, f x k τk,m f x k τk,m 4 and the last inequality is due to the assumption 3 and Gx k τk,m ; ξ k,m f x k τk,m, Gx ; ξ k τk,m k,m f x k τk,m E ξk, m<m =E ξk, E k,m m<m =E ξk, m<m Gx k τk,m ; ξ k,m f x k τk,m, Gx ; ξ k τk,m k,m f x k τ k,m Gx k τk,m ; ξ k,m f x k τk,m, E k,m Gx k τk,m ; ξ k,m f x k τk,m =0. 5 We next turn to T : T = f x k f x k τk,m = f x k f x k τk,m

23 L f x k f x k τk,m x k x k τk,m L max x k x k τk,m k {,,} =L x k x k τk,µ. let µ := arg max m {,,} x k x k τk,m where the second inequality is from the Lipschitzian gradient assumption 5. It follows that T L xk x k τk,µ =L k xj+ x j j=k τ k,µ =L k γ j G x j τj,m ; ξ j,m j=k τ k,µ =L k γ j j=k τ k,µ ] G x j τj,m ; ξ j,m f x j τj,m + L k ] γ j G x j τj,m ; ξ j,m f x j τj,m j=k τ k,µ }{{} T 3 k j=k τ k,µ γ j + f x j τj,m k γ j f x j τj,m j=k τ k,µ }{{} T 4 where the last inequality uses the fact that a + b a + b for any real vectors a and b. Taking the expectation in terms of {ξ j, j {k τ k,µ,..., k }} for T 3, we have E ξj,,j {k τ k,µ,...,k }T 3 =E ξj,,j {k τ k,µ,...,k } =E ξj,,j {k τ k,µ,...,k } + E ξj,,j {k τ k,µ,...,k } k γ j j=k τ k,µ k j=k τ k,µ γ j ] Gx j τj,m ; ξ j,m f x j τj,m Gx j τj,m ; ξ j,m f x j τj,m ] k j >j k τ k,µ γ j γ j ] Gx j τ j,m ; ξ j,m ij f x j τ j,m 3 ] Gx j τ j,m ; ξ j,m f x j τ j,m, 6

24 =E ξj,,j {k τ k,µ,...,k } =E ξj,,j {k τ k,µ,...,k } k j=k T k j=k τ k,µ γ j k j=k τ k,µ γ j Gx j τj,m ; ξ j,m f x j τj,m ] Gx j τj,m ; ξ j,m f x j τj,m ] γ j σ 7 where the second last equality is due to 5 and the third equality is due to E ξj,k j k τ k,µ k j >j k τ k,µ γ j γ j ] Gx j τ j,m ; ξ j,m f x j τ j,m, ] Gx j τ j,m ; ξ j,m f x j τ j,m ] =E ξj,k j k τ k,µ γ j γ j E j Gx j τ j,m ; ξ j,m f x j τ j,m, k j >j k τ k,µ ] Gx j τ j,m ; ξ j,m f x j τ j,m =0. Taking the expectation in terms of ξ j, for T 4, we have E ξj,,j {k τ k,µ,...,k }T 4 =E ξj,,j {k τ k,µ,...,k } T k γ j j=k τ k,µ k j=k τ k,µ γ j E ξ j,,j {k τ k,µ,...,k } f x j τj,m where the last inequality uses the upper bound of the delay age: τ k,µ T. f x j τj,m 8 We take full expectation on both sides of 6 and substitute ET 3 and ET 4 by their upper bounds in 7 and 8 respectively: ET L k j=k T γ k j σ + T γ j E j=k τ k,µ f x j τj,m. 9 4

25 Applying the upper bounds for ET in 9 and ET in 4 to 3, and take full expectation on both sides, we obtain E f x k+ f x k γ k E f x k + E f x k τk,m L k γ k j σ + T γ ] j E f x j τj,m j=k T j=k τ k,µ + γ k L σ + E f x k τk,m γ k γ E f x k + k L γ k E f x k τk,m γ + k L + L k γ k γ j σ j=k T + L k Tγ k γ j E f x j τj,m. 30 j=k T Summarizing the inequality 30 from k = to k =, we have E f x + f x γ k E γ + k L + L T = + + γ k γ k f x k + + L γ k k k j=k T γ j j=k T γ j E γ k E f x k L γ k L γ k E + L T T κ= + L k γ k γ j j=k T f x k + γ k L γ k E σ f x j τj,m γ k+κ γ k E σ γ k L 5 f x k τk,m f x k τk,m + L k γ k γ j σ j=k T

26 where the last inequality is due to 7. Note that x is the global optimization point. Thus we have γ k It completes the proof. γ k E f x k f x f x + γk L + L γ k k j=k T γ j γ k σ. Proofs to Corollary Proof. From 9 and 0, we have If follows that γ γl + L T γ LT T + + T 4T +, which implies that the condition 7 in Theorem is satisfied globally. Then we can safely apply 8 in Theorem : i= E f x i f x f x + γ L + L Tγ 3 σ γ = f x f x + Lσ γ + L Tσ γ γ f x f x + Lσ γ γ f x f x =4 L σ, where the second last inequality is due to 3 and the last equality uses 9. It completes the proof. Proofs to Theorem 3 Proof. From the Lipschitzisan gradient assumption 5, we have f x k+ f x k γ ik f x k, G ˆx k,m ; ξ k,m ik Taking the expectation of i k on both sides of 3, we have E ik f x k+ f x k γ n f x k, G ˆx k,m ; ξ k,m + L maxγ + L maxγ n G ˆx k,m ; ξ k,m ik. 3 G ˆx k,m ; ξ k,m. 33 6

27 Taking the expectation of ξ k, on both sides of 33, we obtain E ξk,,i k f x k+ ] f x k γ n E ξ k, f x k, G ˆx k,m ; ξ k,m + L maxγ n = f x k γ f x n k, f ˆx k,m + L maxγ E ξk, n = f x k γ n f x k + + L maxγ E ξk, n G ˆx k,m ; ξ k,m }{{} T f ˆx k,m E ξk, G ˆx k,m ; ξ k,m G ˆx k,m ; ξ k,m f x k f ˆx k,m }{{} T 34 where the last equation is deduced by the fact a, b = a + b a b. We next consider T and T respectively. For T, we have E ξk, T =E ξk, =E ξk, G ˆx k,m ; ξ k,m G ˆx k,m ; ξ k,m f ˆx k,m + f ˆx k,m =E ξk, G ˆx k,m ; ξ k,m f ˆx k,m ] + f ˆx k,m + G ˆx k,m ; ξ k,m f ˆx k,m, f ˆx k,m =E ξk, G ˆx k,m ; ξ k,m f ˆx k,m + f ˆx k,m =E ξk, G ˆx k,m ; ξ k,m f ˆx k,m + + m<m f ˆx k,m G ˆx k,m ; ξ k,m f ˆx k,m, G ˆx k,m ; ξ k,m f ˆx k,m ] 7