Introduction to Deep Learning

Transcription

1 Introduction to Deep Learning A. G. Schwing & S. Fidler University of Toronto, 2015 A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

2 Outline 1 Universality of Neural Networks 2 Learning Neural Networks 3 Deep Learning 4 Applications 5 References A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

3 What are neural networks? Let s ask Biological Input layer Hidden layer Output layer Input #1 Computational Input #2 Input #3 Input #4 Output A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

4 What are neural networks?...neural networks (NNs) are computational models inspired by biological neural networks [...] and are used to estimate or approximate functions... [Wikipedia] A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

5 What are neural networks? Origins: Traced back to threshold logic [W. McCulloch and W. Pitts, 1943] Perceptron [F. Rosenblatt, 1958] A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

6 What are neural networks? Use cases Classification Playing video games Captcha Neural Turing Machine (e.g., learn how to sort) Alex Graves A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

7 What are neural networks? Example: input x parameters w 1, w 2, b A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

8 What are neural networks? Example: input x parameters w 1, w 2, b x R w 1 w 2 h 1 f b R A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

9 How to compute the function? Forward propagation/pass, inference, prediction: Given input x and parameters w, b Compute (latent variables/) intermediate results in a feed-forward manner Until we obtain output function f x R w 1 w 2 h 1 f b R A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

10 How to compute the function? Forward propagation/pass, inference, prediction: Given input x and parameters w, b Compute (latent variables/) intermediate results in a feed-forward manner Until we obtain output function f A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

11 How to compute the function? Example: input x, parameters w 1, w 2, b x R w 1 w 2 h 1 f b R A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

12 How to compute the function? Example: input x, parameters w 1, w 2, b x R w 1 h w 2 1 f h 1 = σ(w 1 x + b) b R f = w 2 h 1 Sigmoid function: σ(z) = 1/(1 + exp( z)) A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

13 How to compute the function? Example: input x, parameters w 1, w 2, b x R w 1 h w 2 1 f h 1 = σ(w 1 x + b) b R f = w 2 h 1 Sigmoid function: σ(z) = 1/(1 + exp( z)) x = ln 2, b = ln 3, w 1 = 2, w 2 = 2 h 1 =? f =? A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

14 How to compute the function? Given parameters, what is f for x = 0, x = 1, x = 2,... f = w 2 σ(w 1 x + b) A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

15 f How to compute the function? Given parameters, what is f for x = 0, x = 1, x = 2,... f = w 2 σ(w 1 x + b) x A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

16 Let s mess with parameters: x R w 1 h 1 w 2 f h 1 = σ(w 1 x + b) f = w 2 h 1 b R σ(z) = 1/(1 + exp( z)) A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

17 Let s mess with parameters: x R w 1 h 1 w 2 f h 1 = σ(w 1 x + b) f = w 2 h 1 b R σ(z) = 1/(1 + exp( z)) w 1 = 1.0, b changes A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

18 Let s mess with parameters: f x R w 1 h 1 w 2 f h 1 = σ(w 1 x + b) f = w 2 h 1 b R σ(z) = 1/(1 + exp( z)) w 1 = 1.0, b changes 0.4 b = b = 0 b = x A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

19 Let s mess with parameters: f x R w 1 h 1 w 2 f h 1 = σ(w 1 x + b) f = w 2 h 1 b R σ(z) = 1/(1 + exp( z)) 1 w 1 = 1.0, b changes b = 0, w 1 changes b = b = 0 b = x A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

20 Let s mess with parameters: f x R w 1 h 1 w 2 f h 1 = σ(w 1 x + b) f = w 2 h 1 b R σ(z) = 1/(1 + exp( z)) w 1 = 1.0, b changes b = 0, w 1 changes b = b = 0 b = x 0.6 = 0 fw1 0.4 w 1 = w 1 = 1.0 w 1 = x A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

21 Let s mess with parameters: f x R w 1 h 1 w 2 f h 1 = σ(w 1 x + b) f = w 2 h 1 b R σ(z) = 1/(1 + exp( z)) w 1 = 1.0, b changes b = 0, w 1 changes b = b = 0 b = x 0.6 = 0 fw1 0.4 w 1 = w 1 = 1.0 w 1 = x Keep in mind the step function. A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

22 How to use Neural Networks for binary classification? Feature/Measurement: x Output: How likely is the input to be a cat? y x A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

23 f How to use Neural Networks for binary classification? Feature/Measurement: x Output: How likely is the input to be a cat? x A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

24 f How to use Neural Networks for binary classification? Feature/Measurement: x Output: How likely is the input to be a cat? x A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

25 f How to use Neural Networks for binary classification? Feature/Measurement: x Output: How likely is the input to be a cat? x A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

26 f How to use Neural Networks for binary classification? Shifted feature/measurement: x Output: How likely is the input to be a cat? Previous features x A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

27 f f How to use Neural Networks for binary classification? Shifted feature/measurement: x Output: How likely is the input to be a cat? Previous features x Shifted features x A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

28 f f How to use Neural Networks for binary classification? Shifted feature/measurement: x Output: How likely is the input to be a cat? Previous features x Shifted features x Learning/Training means finding the right parameters. A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

29 f So far we are able to scale and translate sigmoids. How well can we approximate an arbitrary function? With the simple model we are obviously not going very far. Features are good Simple classifier x A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

30 f f So far we are able to scale and translate sigmoids. How well can we approximate an arbitrary function? With the simple model we are obviously not going very far. Features are good Simple classifier Features are noisy More complex classifier x x A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

31 f f So far we are able to scale and translate sigmoids. How well can we approximate an arbitrary function? With the simple model we are obviously not going very far. Features are good Simple classifier Features are noisy More complex classifier x x How can we generalize? A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

32 Let s use more hidden variables: b 1 h 1 x R w 1 w 2 w 3 w 4 f h 1 = σ(w 1 x + b 1 ) h 2 = σ(w 3 x + b 2 ) f = w 2 h 1 + w 4 h 2 h 2 b 2 A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

33 Let s use more hidden variables: f b 1 h 1 x R w 1 w 2 w 3 w 4 f h 1 = σ(w 1 x + b 1 ) h 2 = σ(w 3 x + b 2 ) f = w 2 h 1 + w 4 h 2 h 2 b 2 Combining two step functions gives a bump x w 1 = 100, b 1 = 40, w 3 = 100, b 2 = 60, w 2 = 1, w 4 = 1 A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

34 So let s simplify: b 1 x R w 1 h 1 w 2 w 3 w 4 f h 2 b 2 Bump(x 1, x 2, h) f We simplify a pair of hidden nodes to a bump function: Starts at x 1 Ends at x 2 Has height h A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

35 Now we can represent bumps very well. How can we generalize? A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

36 Now we can represent bumps very well. How can we generalize? f Bump(0.0, 0.2, h 1 ) Bump(0.2, 0.4, h 2 ) Bump(0.4, 0.6, h 3 ) f Bump(0.6, 0.8, h 4 ) Bump(0.8, 1.0, h 5) Target Approximation x More bumps gives more accurate approximation. Corresponds to a single layer network. A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

37 Universality: theoretically we can approximate an arbitrary function So we can learn a really complex cat classifier Where is the catch? A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

38 Universality: theoretically we can approximate an arbitrary function So we can learn a really complex cat classifier Where is the catch? Complexity, we might need quite a few hidden units Overfitting, memorize the training data A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

39 Generalizations are possible to A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

40 Generalizations are possible to include more input dimensions capture more output dimensions employ multiple layers for more efficient representations See for a great read! A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

41 How do we find the parameters to obtain a good approximation? How do we tell a computer to do that? A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

42 How do we find the parameters to obtain a good approximation? How do we tell a computer to do that? Intuitive explanation: Compute approximation error at the output Propagate error back by computing individual contributions of parameters to error [Fig. from H. Lee] A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

43 Example for backpropagation of error: Target function: 5x 2 Approximation: f (x, w) Domain of interest: x {0, 1, 2, 3} Error: e(w) = (5x 2 f (x, w)) 2 x {0,1,2,3} A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

44 Example for backpropagation of error: Target function: 5x 2 Approximation: f (x, w) Domain of interest: x {0, 1, 2, 3} Error: Program of interest: min w e(w) = e(w) = min w x {0,1,2,3} x {0,1,2,3} (5x 2 f (x, w)) 2 (5x 2 f (x, w)) 2 }{{} l(x,w) How to optimize? A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

45 Example for backpropagation of error: Target function: 5x 2 Approximation: f (x, w) Domain of interest: x {0, 1, 2, 3} Error: Program of interest: min w e(w) = e(w) = min w x {0,1,2,3} x {0,1,2,3} (5x 2 f (x, w)) 2 (5x 2 f (x, w)) 2 }{{} l(x,w) How to optimize? Gradient descent A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

46 Gradient descent min e(w) w A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

47 Gradient descent min e(w) w Algorithm: start with w 0, t = 0 1 Compute gradient g t = e 2 Update w t+1 = w t ηg t 3 Set t t + 1 w w=wt A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

48 Chain rule is important to compute gradients: min w e(w) = min w x {0,1,2,3} (5x 2 f (x, w)) 2 }{{} l(x,w) A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

49 Chain rule is important to compute gradients: min w Loss function: l(x, w) e(w) = min w x {0,1,2,3} (5x 2 f (x, w)) 2 }{{} l(x,w) A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

50 Chain rule is important to compute gradients: min w Loss function: l(x, w) Squared loss Log loss Hinge loss e(w) = min w x {0,1,2,3} (5x 2 f (x, w)) 2 }{{} l(x,w) A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

51 Chain rule is important to compute gradients: min w Loss function: l(x, w) Squared loss Log loss Hinge loss Derivatives: e(w) = min w e(w) w = = x {0,1,2,3} x {0,1,2,3} (5x 2 f (x, w)) 2 }{{} l(x,w) l(x, w) w A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

52 Chain rule is important to compute gradients: min w Loss function: l(x, w) Squared loss Log loss Hinge loss Derivatives: e(w) = min w e(w) w = = x {0,1,2,3} x {0,1,2,3} x {0,1,2,3} (5x 2 f (x, w)) 2 }{{} l(x,w) l(x, w) w l(x, w) f (x, w) f w A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

53 Slightly more complex example: Composite function represented as a directed a-cyclic graph l(x, w) = f 1 (w 1, f 2 (w 2, f 3 (...))) f 1 (w 1, f 2 ) f f 1 1 w f 1 2 w 1 f 2 (w 2, f 3 ) f 1 f 2 f 2 f 3 f 1 f 2 f 2 w 2 w 2 f 3 (...) Repeated application of chain rule for efficient computation of all gradients A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

54 Back propagation doesn t work well for deep sigmoid networks: Diffusion of gradient signal (multiplication of many small numbers) Attractivity of many local minima (random initialization is very far from good points) Requires a lot of training samples Need for significant computational power A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

55 Back propagation doesn t work well for deep sigmoid networks: Diffusion of gradient signal (multiplication of many small numbers) Attractivity of many local minima (random initialization is very far from good points) Requires a lot of training samples Need for significant computational power Solution: 2 step approach Greedy layerwise pre-training Perform full fine tuning at the end A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

56 Why go deep? Representation efficiency (fewer computational units for the same function) Hierarchical representation (non-local generalization) Combinatorial sharing (re-use of earlier computation) Works very well [Fig. from H. Lee] A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

57 To obtain more flexibility/non-linearity we use additional function prototypes: A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

58 To obtain more flexibility/non-linearity we use additional function prototypes: Sigmoid Rectified linear unit (ReLU) Pooling Dropout Convolutions A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

59 Convolutions What do the numbers mean? See Sanja s lecture 14 for the answers... [Fig. adapted from A. Krizhevsky] A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

60 f conv (, ) A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

61 f conv (, ) = A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

62 Max Pooling What is happening here? [Fig. adapted from A. Krizhevsky] A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

63 A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

64 Rectified Linear Unit (ReLU) A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

65 Rectified Linear Unit (ReLU) Drop information if smaller than zero Fixes the problem of vanishing gradients to some degree A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

66 Rectified Linear Unit (ReLU) Drop information if smaller than zero Fixes the problem of vanishing gradients to some degree Dropout A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

67 Rectified Linear Unit (ReLU) Drop information if smaller than zero Fixes the problem of vanishing gradients to some degree Dropout Drop information at random Kind of a regularization, enforcing redundancy A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

68 A famous deep learning network called AlexNet. The network won the ImageNet competition in How many parameters? Given an image, what is happening? Inference Time: about 2ms per image when processing many images in parallel on the GPU Training Time: a few days given a single recent GPU [Fig. adapted from A. Krizhevsky] A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

69 Demo A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

70 Neural networks have been used for many applications: Classification and Recognition in Computer Vision Text Parsing in Natural Language Processing Playing Video Games Stock Market Prediction Captcha Demos: Russ website Antonio Places website A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

71 Classification in Computer Vision: ImageNet Challenge Since it s the end of the semester, let s find the beach... A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

72 Classification in Computer Vision: ImageNet Challenge A place to maybe prepare for exams... A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

73 Links: Tutorials: Toronto Demo by Russ and students: MIT Demo by Antonio and students: Honglak Lee: workshop-tutorial-final.pdf Yann LeCun: yann/talks/lecun-ranzato-icml2013.pdf Richard Socher: A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39

74 Videos: Video games: Captcha: Stock exchange: A. G. Schwing & S. Fidler (UofT) CSC420: Intro to Image Understanding / 39