ECS289: Scalable Machine Learning

Transcription

1 ECS289: Scalable Machine Learning Cho-Jui Hsieh UC Davis Sept 29, 2016

2 Outline Convex vs Nonconvex Functions Coordinate Descent Gradient Descent Newton s method Stochastic Gradient Descent

3 Numerical Optimization Numerical Optimization: min X f (X ) Can be applied to computer science, economics, control engineering, operating research,... Machine Learning: find a model that minimizes the prediction error.

4 Properties of the Function Smooth function: functions with continuous derivative. Example: ridge regression argmin w 1 2 X w y 2 + λ 2 w 2 Non-smooth function: Lasso, primal SVM Lasso: argmin w SVM: argmin w 1 2 X w y 2 + λ w 1 n max(0, 1 y i w T x i ) + λ 2 w 2 i=1

5 Convex Functions A function is convex if: x 1, x 2, t [0, 1], f (tx 1 + (1 t)x 2 ) tf (x 1 ) + (1 t)f (x 2 ) No local optimum (why?) Figure from Wikipedia

6 Convex Functions If f (x) is twice differentiable, then f is convex if and only if 2 f (x) 0 Optimal solution may not be unique: has a set of optimal solutions S Gradient: capture the first order change of f : f (x + αd) = f (x) + α f (x) T d + O(α 2 ) If f is convex and differentiable, we have the following optimality condition: x S if and only if f (x) = 0

7 Strongly Convex Functions f is strongly convex if there exists an m > 0 such that f (y) f (x) + f (x) T (y x) + m 2 y x 2 2 A strongly convex function has a unique global optimum x (why?) If f is twice differentiable, then f is strongly convex if and only if 2 f (x) mi for all x Gradient descent, coordinate descent will converge linearly (will see later)

8 Nonconvex Functions If f is nonconvex, most algorithms can only converge to stationary points x is a stationary point if and only if f ( x) = 0 Three types of stationary points: (1) Global optimum (2) Local optimum (3) Saddle point Example: matrix completion, neural network,... Example: f (x, y) = 1 2 (xy a)2

9 Coordinate Descent

10 Coordinate Descent Update one variable at a time Coordinate Descent: repeatedly perform the following loop Step 1: pick an index i Step 2: compute a step size δ by (approximately) minimizing Step 3: x i x i + δ argmin f (x + δe i ) δ

11 Coordinate Descent (update sequence) Three types of updating order: Cyclic: update sequence x 1, x 2,..., x }{{ n, x } 1, x 2,..., x n,... }{{} 1st outer iteration 2nd outer iteration Randomly permute the sequence for each outer iteration (faster convergence in practice) Some interesting theoretical analysis for this recently C.-P. Lee and S. J. Wright, Random Permutations Fix a Worst Case for Cyclic Coordinate Descent. 2016

12 Coordinate Descent (update sequence) Three types of updating order: Cyclic: update sequence x 1, x 2,..., x }{{ n, x } 1, x 2,..., x n,... }{{} 1st outer iteration 2nd outer iteration Randomly permute the sequence for each outer iteration (faster convergence in practice) Some interesting theoretical analysis for this recently C.-P. Lee and S. J. Wright, Random Permutations Fix a Worst Case for Cyclic Coordinate Descent Random: each time pick a random coordinate to update Typical way: sample from uniform distribution Sample from uniform distribution vs sample from biased distribution P. Zhao and T. Zhang, Stochastic Optimization with Importance Sampling for Regularized Loss Minimization. In ICML 2015 D. Csiba, Z. Qu and P. Richtarik, Stochastic Dual Coordinate Ascent with Adaptive Probabilities. In ICML 2015

13 Greedy Coordinate Descent Greedy: choose the most important coordinate to update How to measure the importance? By first derivative: i f (x) By first and second derivative: i f (x)/ 2 ii f (x) By maximum reduction of objective function i = argmax i=1,...,n ( f (x) min δ f (x + δe i ) Need to consider the time complexity for variable selection Useful for kernel SVM (see lecture 6) )

14 Extension: block coordinate descent Variables are divided into blocks {X 1,..., X p }, where each X i is a subset of variables and X 1 X 2,..., X p = {1,..., n}, X i X j = ϕ, i, j Each time update a X i by (approximately) solving the subproblem within the block Example: alternating minimization for matrix completion (2 blocks). (See lecture 7)

15 Coordinate Descent (convergence) Converge to optimal if f ( ) is convex and smooth Has a linear convergence rate if f ( ) is strongly convex Linear convergence: error f (x t ) f (x ) decays as β, β 2, β 3,... for some β < 1. Local linear convergence: an algorithm converges linearly after x x K for some K > 0

16 Coordinate Descent: other names Alternating minimization (matrix completion) Iterative scaling (for log-linear models) Decomposition method (for kernel SVM) Gauss Seidel (for linear system when the matrix is positive definite)...

17 Gradient Descent

18 Gradient Descent Gradient descent algorithm: repeatedly conduct the following update: x t+1 x t α f (x t ) where α > 0 is the step size It is a fixed point iteration method: x α f (x) = x if and only if x is an optimal solution Step size too large diverge; too small slow convergence

19 Gradient Descent: successive approximation At each iteration, form an approximation of f ( ): f (x t + d) f x t (d) :=f (x t ) + f (x t ) T d d T ( 1 α I )d =f (x t ) + f (x t ) T d + 1 2α d T d Update solution by x t+1 x t + argmin d f x t (d) d = α f (x t ) is the minimizer of argmin d f x t (d) d will decrease the original objective function f if α (step size) is small enough

20 Gradient Descent: successive approximation However, the function value will decrease if Condition 1: f x (d) f (x + d) for all d Condition 2: f x (0) = f (x) Why? f (x t + d ) f x t (d ) f x t (0) = f (x t ) Condition 2 is satisfied by construction of f x t Condition 1 is satisfied if 1 α I 2 f (x) for all x

21 Gradient Descent: step size A twice differentiable function has L-Lipchitz continuous gradient if and only if 2 f (x) LI x In this case, Condition 2 is satisfied if α < 1 L Theorem: gradient descent converges if α < 1 L Theorem: gradient descent converges linearly with α < 1 L strongly convex if f is

22 Gradient Descent In practice, we do not know L Step size α too large: the algorithm diverges Step size α too small: the algorithm converges very slowly

23 Gradient Descent: line search d is a descent direction if and only if (d ) T f (x) < 0 Armijo rule bakctracking line search: Try α = 1, 1 2, 1 4,... until it staisfies where 0 < γ < 1 f (x + αd ) f (x) + γα(d ) T f (x) Figure from

24 Gradient Descent: line search Gradient descent with line search: Converges to optimal solutions if f is smooth Converges linearly if f is strongly convex However, each iteration requires evaluating f several times Several other step-size selection approaches (an ongoing research topic, especially for stochastic gradient descent)

25 Gradient Descent: applying to ridge regression Input: X R N d, y R N, initial w (0) Output: Solution w 1 := argmin w 2 X w y 2 + λ 2 w 2 1: t = 0 2: while not converged do 3: Compute the gradient 4: Choose step size α t g = X T (X w y) + λw 5: Update w w α t g 6: t t + 1 7: end while Time complexity: O(nnz(X )) per iteration

26 Proximal Gradient Descent How can we apply gradient descent to solve the Lasso problem? argmin w 1 2 X w y 2 + λ w 1 }{{} non differentiable General composite function minimization: argmin f (x) := {g(x) + h(x)} x where g is smooth and convex, h is convex but may be non-differentiable Usually assume h is simple (for computational efficiency)

27 Proximal Gradient Descent: successive approximation At each iteration, form an approximation of f ( ): f (x t + d) f x t (d) :=g(x t ) + g(x t ) T d d T ( 1 α I )d + h(x t + d) =g(x t ) + g(x t ) T d + 1 2α d T d + h(x t + d) Update solution by x t+1 x t + argmin d f x t (d) This is called proximal gradient descent Usually d = argmin d f x t (d) has a closed form solution

28 Proximal Gradient Descent: l 1 -regularization The subproblem: x t+1 =x t + argmin g(x t ) T d + 1 d 2α d T d + λ x t + d u (x t α g(x t )) 2 + λα u 1 = argmin u =S(x t α g(x t ), αλ), where S is the soft-thresholding operator defined by a z if a > z S(a, z) = a + z if a < z 0 if a [ z, z]

29 Proximal Gradient: soft-thresholding Figure from

30 Proximal Gradient Descent for Lasso Input: X R N d, y R N, initial w (0) Output: Solution w 1 := argmin w 2 X w y 2 + λ w 1 1: t = 0 2: while not converged do 3: Compute the gradient 4: Choose step size α t g = X T (X w y) 5: Update w S(w α t g, α t λ) 6: t t + 1 7: end while Time complexity: O(nnz(X )) per iteration

31 Newton s Method

32 Newton s Method Iteratively conduct the following updates: where α is the step size x x α 2 f (x) 1 f (x) If α = 1: converges quadratically when x t is close enough to x : x t+1 x K x t x 2 for some constant K. This means the error f (x t ) f (x ) decays quadratically: β, β 2, β 4, β 8, β 16,... Only need few iterations to converge in this quadratic convergence region

33 Newton s Method However, Newton s update rule is more expensive than gradient descent/coordinate descent

34 Newton s Method Need to compute 2 f (x) 1 f (x) Closed form solution: O(d 3 ) for solving a d dimensional linear system Usually solved by another iterative solver: gradient descent coordinate descent conjugate gradient method... Examples: primal L2-SVM/logistic regression, l 1 -regularized logistic regression,...

35 Newton s Method At each iteration, form an approximation of f ( ): f (x t + d) f x t (d) := f (x t ) + f (x t ) T d + 1 2α d T 2 f (x)d Update solution by x t+1 x t + argmin d f x t (d) When x is far away from x, needs line search to gaurantee convergence Assume LI 2 f (x) mi for all x, then α m L gaurantee the objective function value deacrease because L m 2 f (x) 2 f (y) x, y In practice, we often just use line search.

36 Proximal Newton What if f (x) = g(x) + h(x) and h(x) is non-smooth (h(x) = x 1 )? At each iteration, form an approximation of f ( ): f (x t + d) f x t (d) := g(x t ) + g(x t ) T d + α 2 d T 2 g(x)d + h(x + d) Update solution by x t+1 x t + argmin d fx t (d) Need another iterative solver for solving the subproblem

37 Stochastic Gradient Method

38 Stochastic Gradient Method: Motivation Widely used for machine learning problems (with large number of samples) Given training samples x 1,..., x n, we usually want to solve the following empirical risk minimization (ERM) problem: argmin w where each l i ( ) is the loss function n l i (x i ), i=1 Minimize the summation of individual loss on each sample

39 Stochastic Gradient Method Assume the objective function can be written as f (x) = 1 n f i (x) n Stochastic gradient method: i=1 Iterative conducts the following updates 1 Choose an index i (uniform) randomly 2 x t+1 x t η t f i (x t ) η t > 0 is the step size

40 Stochastic Gradient Method Assume the objective function can be written as f (x) = 1 n f i (x) n Stochastic gradient method: i=1 Iterative conducts the following updates 1 Choose an index i (uniform) randomly 2 x t+1 x t η t f i (x t ) η t > 0 is the step size Why does SG work? E i [ f i (x)] = 1 n n f i (x) = f (x) i=1

41 Stochastic Gradient Method Assume the objective function can be written as f (x) = 1 n f i (x) n Stochastic gradient method: i=1 Iterative conducts the following updates 1 Choose an index i (uniform) randomly 2 x t+1 x t η t f i (x t ) η t > 0 is the step size Why does SG work? E i [ f i (x)] = 1 n n f i (x) = f (x) i=1 Is it a fixed point method? No if η > 0 because x η i f (x ) x Is it a descent method? No, because f (x t+1 ) f (x t )

42 Stochastic Gradient Step size η has to decay to 0 (e.g., η t = Ct a for some constant a, C) SGD converges sub-linearly Many variants proposed recently SVRG, SAGA (2013, 2014): variance reduction AdaGrad (2011): adaptive learning rate RMSProp (2012): estimate learning rate by a running average of gradient. Adam (2015): adaptive moment estimation Widely used in machine learning

43 SGD in Deep Learning AdaGrad: adaptive step size for each parameter Update rule at the t-th iteration: Compute g = f (x) Estimate the second moment: G ii = G ii + g 2 Parameter update: x i x i η Gii g i Proposed for convex optimization in: i for all i for all i Adaptive subgradient methods for online learning and stochastic optimization (JMLR 2011) Adaptive Bound Optimization for Online Convex Optimization (COLT 2010)

44 SGD in Deep Learning AdaGrad: adaptive step size for each parameter Update rule at the t-th iteration: Compute g = f (x) Estimate the second moment: G ii = G ii + g 2 Parameter update: x i x i η Gii g i i for all i for all i Adam (Kingma and Ba, 2015): maintain both first moment and second moment Update rule at the t-th iteration: Compute g = f (x) m = β 1 m + (1 β 1 )g ˆm = m t /(1 β1 t ) (estimate of first moment) v = β 2 v + (1 β 2 )g 2 ˆv = v t /(1 β2 t ) (estimate of second moment) x x η ˆm/(ˆv + ɛ) All the operations are element-wise

45 Stochastic Gradient: Variance Reduction SGD is not a fixed point method: even if x is optimal, x η i f (x ) x Reason: E[ i f (x )] = f (x ) = 0 but variance can be large Variance reduction: reduce the variance so that variance 0 when t.

46 SVRG SVRG (Johnson and Zhang, 2013) For t = 1, 2, 1 x = x 2 g = 1 n n i=1 f i( x) 3 For s = 1, 2, 1 Randomly pick i 2 x x η( f i (x) f i ( x) + g ) }{{} a i Can easily show E[a i ] = 0 (unbiased) Var[a i ] = 0 when x = x = x Have linear convergence rate

47 Stochastic Gradient: applying to ridge regression Objective function: argmin w 1 n n (w T x i y i ) 2 + λ w 2 i=1 How to write as argmin w 1 n n i=1 f i(w)? How to decompose into n components?

48 Stochastic Gradient: applying to ridge regression Objective function: argmin w 1 n n (w T x i y i ) 2 + λ w 2 i=1 How to write as argmin w 1 n n i=1 f i(w)? First approach: f i (w) = (w T x i y i ) 2 + λ w 2 Update rule: w t+1 w t 2η t (w T x i y i )x i 2η t λw = (1 2η t λ)w 2η t (w T x i y i )x i

49 Stochastic Gradient: applying to ridge regression Objective function: argmin w 1 n n (w T x i y i ) 2 + λ w 2 i=1 How to write as argmin w 1 n n i=1 f i(w)? First approach: f i (w) = (w T x i y i ) 2 + λ w 2 Update rule: w t+1 w t 2η t (w T x i y i )x i 2η t λw = (1 2η t λ)w 2η t (w T x i y i )x i Need O(d) complexity per iteration even if data is sparse

50 Stochastic Gradient: applying to ridge regression Objective function: argmin w 1 n n (w T x i y i ) 2 + λ w 2 i=1 How to write as argmin w 1 n n i=1 f i(w)? First approach: f i (w) = (w T x i y i ) 2 + λ w 2 Update rule: w t+1 w t 2η t (w T x i y i )x i 2η t λw = (1 2η t λ)w 2η t (w T x i y i )x i Need O(d) complexity per iteration even if data is sparse Solution: store w = sv where s is a scalar

51 Stochastic Gradient: applying to ridge regression Objective function: Second approach: argmin w 1 n n (w T x i y i ) 2 + λ w 2 i=1 define Ω i = {j X ij 0} for i = 1,..., n define n j = {i X ij 0} for j = 1,..., d define f i (w) = (w T x i y i ) 2 + j Ω i λn n j wj 2

52 Stochastic Gradient: applying to ridge regression Objective function: Second approach: argmin w 1 n n (w T x i y i ) 2 + λ w 2 i=1 define Ω i = {j X ij 0} for i = 1,..., n define n j = {i X ij 0} for j = 1,..., d define f i (w) = (w T x i y i ) 2 + j Ω i Update rule when selecting index i: λn n j wj 2 w t+1 j wj t 2η t (xi T w t y i )X ij 2ηt λn wj t, n j j Ω i Solution: update can be done in O( Ω i ) operations

53 Coming up Next class: Parallel Optimization Questions?