Introduction to Deep Learning

Transcription

1 Introduction to Deep Learning Some slides and images are taken from: David Wolfe Corne Wikipedia Geoffrey A. Hinton

2 Feedforward networks for function approximation and classification Feedforward networks Deep tensor networks ConvNets

3 x1 x2 w1 w2 Dendrites Terminal Branches of Axon x3 w3 Σ Axon xn wn

4 A single artificial neuron 1 w0 x1 w1 xn wn

5 A single artificial neuron bias node 1 w0 x1 w1 nonlinear activation function inputs xn wn summation output / activation of the neuron weights

6 Activation functions

7 Activation functions Traditionally used

8 Activation functions Currently most widely used. Empirically easier to train and results in sparse networks. Nair and Hinton: Rectified linear units improve restricted Boltzmann machines ICLM 10, (2010) Glorot, Bordes and Bengio: Deep sparse rectifier neural networks. PMLR 15: (2011)

9 Perceptron (Rosenblatt 1957) Used as a binary classifier - equivalent to support vector machine (SVM) 1 w0 x1 w1 xn wn

10 Perceptron (Rosenblatt 1957) 1 w0 x1 w1 xn wn

11 Second-generation neural networks (~1985) Slide credit : Geoffrey Hinton Back-propagate error signal to get derivatives for learning Compare outputs with correct answer to get error signal outputs hidden layers input vector

12 Second-generation neural networks (~1985) Error at output for a given example: Compare outputs with correct answer to get error signal Slide credit : Geoffrey Hinton j outputs wji i hidden layers input vector

13 Second-generation neural networks (~1985) Error at output for a given example: Compare outputs with correct answer to get error signal Slide credit : Geoffrey Hinton Error sensitivity at output neuron j: j outputs Backpropagate error sensitivity to neuron i: Sensitivity on weight wji: wji i hidden layers input vector

14 Second-generation neural networks (~1985) Error at output for a given example: Compare outputs with correct answer to get error signal Slide credit : Geoffrey Hinton Error sensitivity at output neuron j: j outputs Backpropagate error sensitivity to neuron i: Sensitivity on weight wji: wji i hidden layers Update weight wji: input vector learning rate

15 A decision boundary perspective on learning Initial random weights

16 A decision boundary perspective on learning Present a training instance / adjust the weights

17 A decision boundary perspective on learning Present a training instance / adjust the weights

18 A decision boundary perspective on learning Present a training instance / adjust the weights

19 A decision boundary perspective on learning Present a training instance / adjust the weights

20 A decision boundary perspective on learning Eventually.

21 Nonlinear versus linear models NNs use nonlinear f(x) so they can draw complex boundaries, but keep the data unchanged Kernel methods only draw straight lines, but transform the data first in a way that makes it linearly separable

22 Universal Representation Theorem Networks with a single hidden layer can represent any function F(x) with arbitrary precision in the large hidden layer size limit However, that doesn t mean, networks with single hidden layers are efficient in representing arbitrary functions. For many datasets, deep networks can represent the function F(x) even with narrow layers.

23 What does a neural network learn?

24 Feature detectors

25 What is this unit doing?

26 Hidden layer units become self-organized feature detectors strong +ve weight low/zero weight 63

27 what does this unit detect? strong +ve weight low/zero weight 63

28 what does this unit detect? strong +ve weight low/zero weight it will send strong signal for a horizontal line in the top row, ignoring everywhere else 63

29 what does this unit detect? strong +ve weight low/zero weight 63

30 what does this unit detect? strong +ve weight low/zero weight Strong signal for a dark area in the top left corner 63

31 What features might you expect a good NN to learn, when trained with data like this?

32 Horizontal lines 1 63

33 Horizontal lines 1 63

34 Small circles 1 63

35 Small circles 1 But what about position invariance? Our example unit detectors were tied to specific parts of the image

36 Deep Networks

37 successive layers can detect higher-level features detect lines in Specific positions etc Higher level detetors ( horizontal line, RHS vertical lune upper loop, etc v etc

38 successive layers can detect higher-level features detect lines in Specific positions etc Higher level detetors ( horizontal line, RHS vertical lune upper loop, etc v etc What does this unit detect?

39 So: multiple layers make sense

40 So: multiple layers make sense Multiple layers are also found in the brain, e.g. visual cortex

41 But: until recently deep networks could not be efficiently trained

42

43 Convolutional Neural Networks

44

45 Convolutional Kernel / Filter Apply convolutions

46

47 Convolutional filters perform image processing

48 Example: filters in face recognition

49 MNIST dataset

50 Misclassified examples

51 Applications to molecular systems

52 1) Learning to represent (effective) energy function Behler-Parrinello network Total energy Cartesian coordinates Internal coordinates Neural networks (may be shared for same atom types) Atomic energies Behler and Parrinello, PRL 98, (2007)

53 1) Learning to represent (effective) energy function Schuett, Arbabzadah, Chmiela, Müller & Tkatchenko, Nature Communications 8, (2017)

54 2) Generator networks Gómez-Bombarelli,, Aspuru-Guzik: Automatic Chemical Design using Variational Autoencoders (2016)

55 2) Generator networks Gómez-Bombarelli,, Aspuru-Guzik: Automatic Chemical Design using Variational Autoencoders (2016)

56 2) Generator networks Gómez-Bombarelli,, Aspuru-Guzik: Automatic Chemical Design using Variational Autoencoders (2016)

57 3) VAMPnets Mardt, Pasquali, Wu & Noé: VAMPnets - deep learning of molecular kinetics (2017)

58 3) VAMPnets Mardt, Pasquali, Wu & Noé: VAMPnets - deep learning of molecular kinetics (2017)