Deep Learning II: Momentum & Adaptive Step Size

Transcription

1 Deep Learning II: Momentum & Adaptive Step Size CS 760: Machine Learning Spring 2018 Mark Craven and David Page 1

2 Goals for the Lecture You should understand the following concepts: momentum Nesterov momentum adaptive step size AdaGrad RMSProp Adam Thanks Yujia Bao! 2

3 Background: Mini-batch gradient descent Procedure: Pick a mini-batch of examples from the data For each example: Forward propagation to get the output; Backward propagation to get the gradient; Update the weights using the average gradient. Common mini-batch size: 32, 64, 128. Note: Forward/Backward propagation can both be done simultaneously for many examples using matrix multiplication, and this is fast on GPUs!

4 Mini-batch gradient descent Recall the forward prop for a MLP: z "#$ = f "#$ W " z " + b " where f "#$ is the activation function for the (k + 1)st layer, W " i [j] is the weights from z " [j] to z "#$ [i] If we have n examples, we can define Z " = z " $ z " 3, B " = b " b " and express the forward prop as Z "#$ = f " W " Z " + B "

5 An example

6 Weight Update Methods Steepest Descent (Review) Momentum (Review) Nesterov Accelerated Gradient AdaGrad [Duchi et al., 2011] RMSprop [Tieleman & Hinton, 2012] Adam [Kingma & Ba, 2015] Element-wise adaptive learning rate

7 Steepest Descent Algorithm: Step 1. w (8#$) = w (8) η => (A) Very simple. Just one step. Poor convergence if the function surface is more steep in one dimension than in another. Let s see an example!

8 Steepest Descent Objective: L w = w $ C + 25w C C -50w C 2w $ Gradient: L = 2w $ 50w C Small over w $ direction, large over w C direction!

9 Steepest Descent Zigzag behavior! Move very slowly on the flat direction, oscillate on the steep direction.

10 Trick 7: Momentum Algorithm: Step 1.v (8#$) = μv (8) η => (A) Interpret v as velocity, 1 μ as friction. v stores the moving average of the gradients. Step 2. w (8#$) = w (8) + v (8#$) A ball rolling down from the surface of the loss function. Initialize v (J) to zero. Common settings for μ ranges between 0.5 to 0.99.

11 Trick 7: Momentum Accelerate convergence on the direction that the gradient keeps the same sign over the past few iterations. If μ is large, momentum will overshoot the target, but the overall convergence is still faster than steepest descent in practice. As μ goes smaller, momentum behaves more similar to steepest descent.

12

13 Nesterov Accelerated Gradient Momentum Nesterov Accelerated Gradient w (8#$) Negative gradient direction: η => (A) #LM (A) Speed direction: μv (8) Actual update Speed direction: μv (8) w (8#$) Actual update w (8) Negative gradient direction: η => (A) w (8)

14 An example Suppose w (8) is already optimal, but v (8) 0. Speed direction: μv (8) w (8) Momentum w (8#$) Actual update Negative gradient direction: 0 Nesterov Accelerated Gradient Speed direction: μv (8) w (8) w (8#$) Actual update Negative gradient direction: η => (A) #LM (A) Look ahead! The actual update is more reliable than Momentum.

15 Nesterov Accelerated Gradient Algorithm: Step 1.v (8#$) = μv (8) η => (A) #LM (A) Step 2.w (8#$) = w (8) + v (8#$) Nesterov Accelerated Gradient has better performance than momentum [Ilya et al., 2013].

16

17 Trick 8: Adaptive Learning Rate

18 Some Notes on our Notation in this Lecture For a scalar η and vector v, ηv is a vector Likewise P v is a vector, just as η $ v is a vector By between two vectors, we mean element-wise multiplication rather than dot product

19 AdaGrad (Adaptive Gradient) Algorithm: Step 1. G (8#$) = G (8) + => (A) C G is the sum of the square gradient. Step 2.w (8#$) = w (8) P W (AXY) #$J[\ =?? (A) Scale the learning rate by 1/ G and follow the negative gradient direction. Common setting for η is Different learning rate for each w (each w has its own G). Increase the learning rate on the flat direction. Decrease the learning rate on the steep direction.

20 AdaGrad (Adaptive Gradient) Consider the previous example: Gradient: L w = w $ C + 25w C C L = 2w $ 50w C Initial w (J) is 1, 1. η ( 2) w ($) = C + 10^_ η ( 50) 50 C + 10^_ Will follow the diagonal direction! η η

21

22 AdaGrad (Adaptive Gradient) Algorithm: Step 1. G (8#$) = G (8) + => (A) C Step 2. w (8#$) = w (8) P W (AXY) #$J[\ =?? (A) Why that slow? Learning rate goes down monotonically as t increases. Stop learning!!!

23 RMSProp (Root Mean Square Propagation) Algorithm: C =?? (A) G is a moving exponential average of the square gradient. Step 1. G (8#$) = γg (8) + 1 γ Step 2. w (8#$) = w (8) P W (AXY) #$J[\ =?? (A) Scale the learning rate by 1/ G and follow the negative gradient direction. Default settings: γ = 0.9, η =

24

25 Another example (saddle point) Now consider minimizing L w = w $ e + w C C The problem is unbounded and (0,0) is a saddle point. At 0.01, 1, we have L = How will these methods behave if we start at this point?

26

27 Adam (Adaptive Moment Estimation) Algorithm [Simplified Version]: Step 1. m (8#$) = β $ m (8) + 1 β $ => (A) m is the exponential moving average of the gradients (similar as the speed in the momentum method); C =?? (A) v is the exponential moving average of the square gradients. Step 2. v (8#$) = β C v (8) + (1 β C ) Step 3. w (8#$) = w (8) P m(8#$) M AXY #$J[i Default settings: β $ = 0.9, β C = 0.999, η =

28 Adam (Adaptive Moment Estimation) Algorithm [Complete Version]: Step 1.m (8#$) = β $ m (8) + 1 β $ => (A) Step 2. v (8#$) = β C v (8) + (1 β C ) C =?? (A) Step 3. mj (8#$) = $ $^k Y A m (8#$) Step 4. vl (8#$) = $ $^k m A v (8#$) Bias Correction (since we set m (J), v (J) to 0) Step 5. w (8#$) = w (8) P Ml (AXY) #$J [i mj (8#$)

29

30

31 Adam (Adaptive Moment Estimation) Image credit: Kingma & Ba, 2015 Network configurations: Input layer: 28x28 2 hidden layers (1000 ReLUs each) Output: 10 SoftMax Cross-entropy loss. In practice, Adam is the default choice to use. [Fei- Fei Li, Andrej Karpathy, Justin Johnson]