(Feed-Forward) Neural Networks Dr. Hajira Jabeen, Prof. Jens Lehmann

Transcription

1 (Feed-Forward) Neural Networks Dr. Hajira Jabeen, Prof. Jens Lehmann

2 Outline In the previous lectures we have learned about tensors and factorization methods. RESCAL is a bilinear model for SRL that can be formulated as tensor factorization problem. Furthermore, we learned about optimization techniques which can be applied for learning score-based models. Today we will learn about a new class of algorithms which can be applied for SRL, namely neural networks. We will see how to apply them for SRL in the following lecture. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 2

3 Artificial Neurons source: Wikipedia Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 3

4 Artificial Neurons source: Wikipedia There are many different types of neurons in the nervous system, and they are quite complicated. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 3

5 Artificial Neurons source: Wikipedia There are many different types of neurons in the nervous system, and they are quite complicated. Neurons are connected to each other with synapses. Thus, they form a network. This complicates things even more. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 3

6 Artificial Neurons source: Wikipedia There are many different types of neurons in the nervous system, and they are quite complicated. Neurons are connected to each other with synapses. Thus, they form a network. This complicates things even more. How to understand this system? Basic idea: reduce the neuron to its essentials. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 3

7 Artificial Neurons source: Wikipedia There are many different types of neurons in the nervous system, and they are quite complicated. Neurons are connected to each other with synapses. Thus, they form a network. This complicates things even more. How to understand this system? Basic idea: reduce the neuron to its essentials. There are a number of greatly simplified neuron models. One of the simplest models is by McCulloch and Pitts. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 3

8 Artificial Neurons A neuron receives input from a number of other neurons. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 4

9 Artificial Neurons A neuron receives input from a number of other neurons. These inputs come in the form of spikes short pulses of electrical current. We average these spikes over time and represent them with a single number: the spike frequency ν. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 4

10 Artificial Neurons A neuron receives input from a number of other neurons. These inputs come in the form of spikes short pulses of electrical current. We average these spikes over time and represent them with a single number: the spike frequency ν. The spikes arrive at the neuron s membrane and alter the electrical potential at this point. For each neuron we keep track of its membrane potential u. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 4

11 Artificial Neurons A neuron receives input from a number of other neurons. These inputs come in the form of spikes short pulses of electrical current. We average these spikes over time and represent them with a single number: the spike frequency ν. The spikes arrive at the neuron s membrane and alter the electrical potential at this point. For each neuron we keep track of its membrane potential u. There are excitatory and inhibitory connections between neurons, and they can be of different strength. We model the effect of a connection on the membrane potential as the product of the synaptic weight w with the spike frequency. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 4

12 Artificial Neurons A neuron receives input from a number of other neurons. These inputs come in the form of spikes short pulses of electrical current. We average these spikes over time and represent them with a single number: the spike frequency ν. The spikes arrive at the neuron s membrane and alter the electrical potential at this point. For each neuron we keep track of its membrane potential u. There are excitatory and inhibitory connections between neurons, and they can be of different strength. We model the effect of a connection on the membrane potential as the product of the synaptic weight w with the spike frequency. The weight is positive for excitatory and negative for inhibitory connections. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 4

13 Artificial Neurons A neuron receives input from a number of other neurons. These inputs come in the form of spikes short pulses of electrical current. We average these spikes over time and represent them with a single number: the spike frequency ν. The spikes arrive at the neuron s membrane and alter the electrical potential at this point. For each neuron we keep track of its membrane potential u. There are excitatory and inhibitory connections between neurons, and they can be of different strength. We model the effect of a connection on the membrane potential as the product of the synaptic weight w with the spike frequency. The weight is positive for excitatory and negative for inhibitory connections. The absolute value of the weight is small for weak connections and large for strong connections. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 4

14 Artificial Neurons When the neuron s membrane potential exceeds a threshold then the neuron emits a spike (which can propagate to multiple receivers) and resets its membrane potential. The spike frequency as a function of the incoming power is a non-linear transfer function (also simply called non-linearity): We call such a function a sigmoid or sigmoidal function. The standard formula is 1 ν = σ(u) = 1 + e u. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 5

15 Artificial Neurons Now assume we have a pool of neurons, numbered 1,..., n. Let u i be the membrane potential and ν i be the firing rate of neuron number i, and let w ji be the synaptic weight of the connection from i to j (which is zero if the neurons are not connected). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 6

16 Artificial Neurons Now assume we have a pool of neurons, numbered 1,..., n. Let u i be the membrane potential and ν i be the firing rate of neuron number i, and let w ji be the synaptic weight of the connection from i to j (which is zero if the neurons are not connected). Then we arrive at the model u j n w ji ν i i=1 ν j σ(u j ). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 6

17 Artificial Neurons Now assume we have a pool of neurons, numbered 1,..., n. Let u i be the membrane potential and ν i be the firing rate of neuron number i, and let w ji be the synaptic weight of the connection from i to j (which is zero if the neurons are not connected). Then we arrive at the model u j n w ji ν i i=1 ν j σ(u j ). This model draws inspiration from biology. However, it is so abstract that in the end it has little in common with its biological counterpart. It should rather be viewed as a computational unit in a mathematical learning machine. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 6

18 Artificial Neurons The relation u j n i=1 w ji ν i is familiar: if ν i are the inputs, then this is a linear function, which can be understood as a Perceptron model. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 7

19 Artificial Neurons The relation u j n i=1 w ji ν i is familiar: if ν i are the inputs, then this is a linear function, which can be understood as a Perceptron model. The sigmoid does not change the way decisions are made: ν j σ ( n i=1 w ji ν i ). The threshold changes to 1/2 (instead of zero), but the decision boundary remains the same. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 7

20 Artificial Neurons The relation u j n i=1 w ji ν i is familiar: if ν i are the inputs, then this is a linear function, which can be understood as a Perceptron model. The sigmoid does not change the way decisions are made: ν j σ ( n i=1 w ji ν i ). The threshold changes to 1/2 (instead of zero), but the decision boundary remains the same. In other words, the Perceptron is a model of a single neuron. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 7

21 Artificial Neurons The relation u j n i=1 w ji ν i is familiar: if ν i are the inputs, then this is a linear function, which can be understood as a Perceptron model. The sigmoid does not change the way decisions are made: ν j σ ( n i=1 w ji ν i ). The threshold changes to 1/2 (instead of zero), but the decision boundary remains the same. In other words, the Perceptron is a model of a single neuron. Usually many neurons process the input, so we have multiple Perceptrons in parallel: Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 7

22 Layered Neural Networks Let ν (0) R m denote the vector of firing rates coming from the inputs (sensors, data), and let ν (1) R n denote the vector of firing rates of the neurons. Let W R n m be the matrix with entries w ji. Also, let σ denote the component-wise application of the transfer function. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 8

23 Layered Neural Networks Let ν (0) R m denote the vector of firing rates coming from the inputs (sensors, data), and let ν (1) R n denote the vector of firing rates of the neurons. Let W R n m be the matrix with entries w ji. Also, let σ denote the component-wise application of the transfer function. The computation can be written in compact form: ν (1) = σ(w ν (0) ) Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 8

24 Layered Neural Networks Let ν (0) R m denote the vector of firing rates coming from the inputs (sensors, data), and let ν (1) R n denote the vector of firing rates of the neurons. Let W R n m be the matrix with entries w ji. Also, let σ denote the component-wise application of the transfer function. The computation can be written in compact form: ν (1) = σ(w ν (0) ) Neurons can not only receive input from sensors, but also from other neurons. What if we feed the outputs into another layer of neurons? Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 8

25 Layered Neural Networks The resulting architecture is called a (layered) feed-forward neural network, or multi layer Perceptron (MLP). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 9

26 Layered Neural Networks The resulting architecture is called a (layered) feed-forward neural network, or multi layer Perceptron (MLP). This example network has an input layer with 5 nodes, two hidden layers with 8 and 6 neurons, and an output layer with 2 neurons. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 9

27 Layered Neural Networks The resulting architecture is called a (layered) feed-forward neural network, or multi layer Perceptron (MLP). This example network has an input layer with 5 nodes, two hidden layers with 8 and 6 neurons, and an output layer with 2 neurons. The size of the input and output layers is determined by the problem (dimension of the vectors x and y), but number and size of the hidden layers is arbitrary. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 9

28 Layered Neural Networks Now let ν (0) denote the vector of inputs, let ν (i) denote the vector of firing rates in layer i, and let W (i) denote the matrix of connections from layer i 1 to layer i. Then we have the overall model: ( ν (n) = σ out (W (n) σ W (n 1) σ (... σ(w (1) ν (0) )... ))). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 10

29 Layered Neural Networks Now let ν (0) denote the vector of inputs, let ν (i) denote the vector of firing rates in layer i, and let W (i) denote the matrix of connections from layer i 1 to layer i. Then we have the overall model: ( ν (n) = σ out (W (n) σ W (n 1) σ (... σ(w (1) ν (0) )... ))). The model processes a data point x as follows: Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 10

30 Layered Neural Networks Now let ν (0) denote the vector of inputs, let ν (i) denote the vector of firing rates in layer i, and let W (i) denote the matrix of connections from layer i 1 to layer i. Then we have the overall model: ( ν (n) = σ out (W (n) σ W (n 1) σ (... σ(w (1) ν (0) )... ))). The model processes a data point x as follows: Set ν (0) = x. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 10

31 Layered Neural Networks Now let ν (0) denote the vector of inputs, let ν (i) denote the vector of firing rates in layer i, and let W (i) denote the matrix of connections from layer i 1 to layer i. Then we have the overall model: ( ν (n) = σ out (W (n) σ W (n 1) σ (... σ(w (1) ν (0) )... ))). The model processes a data point x as follows: Set ν (0) = x. Apply the model, i.e., compute ν (n) from ν (0). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 10

32 Layered Neural Networks Now let ν (0) denote the vector of inputs, let ν (i) denote the vector of firing rates in layer i, and let W (i) denote the matrix of connections from layer i 1 to layer i. Then we have the overall model: ( ν (n) = σ out (W (n) σ W (n 1) σ (... σ(w (1) ν (0) )... ))). The model processes a data point x as follows: Set ν (0) = x. Apply the model, i.e., compute ν (n) from ν (0). Output ŷ = ν (n) (which is hopefully close to the true label y). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 10

33 Layered Neural Networks ( ν (n) = σ out (W (n) σ W (n 1) σ (... σ(w (1) ν (0) )... ))) Hidden layers employ a sigmoid transfer function. The transfer function σ out of the output layer is chosen task specific: Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 11

34 Layered Neural Networks ( ν (n) = σ out (W (n) σ W (n 1) σ (... σ(w (1) ν (0) )... ))) Hidden layers employ a sigmoid transfer function. The transfer function σ out of the output layer is chosen task specific: Regression problems usually need an unbounded range of values. Then a sigmoid is not appropriate. The identity function is used in this case (so-called linear output neurons). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 11

35 Layered Neural Networks ( ν (n) = σ out (W (n) σ W (n 1) σ (... σ(w (1) ν (0) )... ))) Hidden layers employ a sigmoid transfer function. The transfer function σ out of the output layer is chosen task specific: Regression problems usually need an unbounded range of values. Then a sigmoid is not appropriate. The identity function is used in this case (so-called linear output neurons). For classification the range of values does not matter. Either linear or sigmoid output layer neurons can be used. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 11

36 Layered Neural Networks ( ν (n) = σ out (W (n) σ W (n 1) σ (... σ(w (1) ν (0) )... ))) Hidden layers employ a sigmoid transfer function. The transfer function σ out of the output layer is chosen task specific: Regression problems usually need an unbounded range of values. Then a sigmoid is not appropriate. The identity function is used in this case (so-called linear output neurons). For classification the range of values does not matter. Either linear or sigmoid output layer neurons can be used. The linear function u = W ν is usually extended to an affine function u = W ν + b by means of a so-called bias neuron. This neuron has a constant firing rate of one and is input to all other neurons, with connection weights b i. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 11

37 Layered Neural Networks Hidden layers employ a sigmoid transfer function. The transfer function σ out of the output layer is chosen task specific: Regression problems usually need an unbounded range of values. Then a sigmoid is not appropriate. The identity function is used in this case (so-called linear output neurons). For classification the range of values does not matter. Either linear or sigmoid output layer neurons can be used. The linear function u = W ν is usually extended to an affine function u = W ν + b by means of a so-called bias neuron. This neuron has a constant firing rate of one and is input to all other neurons, with connection weights b i. This is effectively the same as embedding affine functions into linear functions, one dimension up. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 11 ( ν (n) = σ out (W (n) σ W (n 1) σ (... σ(w (1) ν (0) )... )))

38 Layered Neural Networks A layered neural network alternates the application of two types of transformations: Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 12

39 Layered Neural Networks A layered neural network alternates the application of two types of transformations: 1 A linear map: left multiplication with the matrix W (i). This matrix is a parameter of the model, so it can be subject to learning. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 12

40 Layered Neural Networks A layered neural network alternates the application of two types of transformations: 1 A linear map: left multiplication with the matrix W (i). This matrix is a parameter of the model, so it can be subject to learning. 2 A non-linear function: component-wise transfer function σ. This function is fixed. It has no parameters that can be adjusted. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 12

41 Layered Neural Networks A layered neural network alternates the application of two types of transformations: 1 A linear map: left multiplication with the matrix W (i). This matrix is a parameter of the model, so it can be subject to learning. 2 A non-linear function: component-wise transfer function σ. This function is fixed. It has no parameters that can be adjusted. The non-linearities are not adaptive, but they are nevertheless important! Without them the model would collapse into the linear map W = W (n)... W (1). Then all computations were linear. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 12

42 Layered Neural Networks A layered neural network alternates the application of two types of transformations: 1 A linear map: left multiplication with the matrix W (i). This matrix is a parameter of the model, so it can be subject to learning. 2 A non-linear function: component-wise transfer function σ. This function is fixed. It has no parameters that can be adjusted. The non-linearities are not adaptive, but they are nevertheless important! Without them the model would collapse into the linear map W = W (n)... W (1). Then all computations were linear. It turns out that a neural network with sigmoid transfer functions is indeed far more powerful than a linear model. In a sense, it can compute everything. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 12

43 Universal Approximation Property Theorem. Let σ : R R be a continuous, non-constant, bounded, and monotonically increasing function. Let K R m be compact, and let C(K) denote the space of continuous functions K R. Then, given a function g C(K) and an accuracy ε > 0, there exists a hidden layer size N N and a set of coefficients w (1) i R m, w (2) i, b i R (for i {1,..., N}), such that f : K R ; f (x) = N i=1 is an ε-approximation of g, that is, w (2) i ( ) σ (w (1) i ) T x + b i f g := max f (x) g(x) < ε. x K Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 13

44 Universal Approximation Property Theorem. Let σ : R R be a continuous, non-constant, bounded, and monotonically increasing function. Let K R m be compact, and let C(K) denote the space of continuous functions K R. Then, given a function g C(K) and an accuracy ε > 0, there exists a hidden layer size N N and a set of coefficients w (1) i R m, w (2) i, b i R (for i {1,..., N}), such that f : K R ; f (x) = N i=1 is an ε-approximation of g, that is, w (2) i ( ) σ (w (1) i ) T x + b i f g := max f (x) g(x) < ε. x K Corollary. The theorem extends trivially to multiple outputs. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 13

45 Universal Approximation Property Theorem. Let σ : R R be a continuous, non-constant, bounded, and monotonically increasing function. Let K R m be compact, and let C(K) denote the space of continuous functions K R. Then, given a function g C(K) and an accuracy ε > 0, there exists a hidden layer size N N and a set of coefficients w (1) i R m, w (2) i, b i R (for i {1,..., N}), such that f : K R ; f (x) = N i=1 is an ε-approximation of g, that is, w (2) i ( ) σ (w (1) i ) T x + b i f g := max f (x) g(x) < ε. x K Corollary. The theorem extends trivially to multiple outputs. Corollary. Neural networks with a single sigmoidal hidden layer and linear output layer are universal approximators. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 13

46 Universal Approximation Property This means that for a given target function g there exists a sequence of networks ( f k that converges (pointwise) to )k N the target function. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 14

47 Universal Approximation Property This means that for a given target function g there exists a sequence of networks ( f k that converges (pointwise) to )k N the target function. Usually, as the networks come closer and closer to g, they will need more and more hidden neurons. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 14

48 Universal Approximation Property This means that for a given target function g there exists a sequence of networks ( f k that converges (pointwise) to )k N the target function. Usually, as the networks come closer and closer to g, they will need more and more hidden neurons. A network with fixed layer sizes can only model a subspace of all continuous functions. Its dimensionality is limited by the number of weights. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 14

49 Universal Approximation Property This means that for a given target function g there exists a sequence of networks ( f k that converges (pointwise) to )k N the target function. Usually, as the networks come closer and closer to g, they will need more and more hidden neurons. A network with fixed layer sizes can only model a subspace of all continuous functions. Its dimensionality is limited by the number of weights. The continuous functions form an infinite dimensional vector space. Therefore arbitrarily large hidden layer sizes are needed. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 14

50 Universal Approximation Property This means that for a given target function g there exists a sequence of networks ( f k that converges (pointwise) to )k N the target function. Usually, as the networks come closer and closer to g, they will need more and more hidden neurons. A network with fixed layer sizes can only model a subspace of all continuous functions. Its dimensionality is limited by the number of weights. The continuous functions form an infinite dimensional vector space. Therefore arbitrarily large hidden layer sizes are needed. The universal approximation property is not as special as it seems. For example, polynomials are universal approximators (Weierstraß theorem). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 14

51 Other Neural Networks In this lecture we cover only feed-forward neural networks, because this is the most basic and most relevant class for supervised (and unsupervised) learning and the class applied to SRL. A broad range of other network types has been developed. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 15

52 Other Neural Networks In this lecture we cover only feed-forward neural networks, because this is the most basic and most relevant class for supervised (and unsupervised) learning and the class applied to SRL. A broad range of other network types has been developed. The linear+sigmoid processing model can be replaced by other functions. For example, this leads to radial basis function models. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 15

53 Other Neural Networks In this lecture we cover only feed-forward neural networks, because this is the most basic and most relevant class for supervised (and unsupervised) learning and the class applied to SRL. A broad range of other network types has been developed. The linear+sigmoid processing model can be replaced by other functions. For example, this leads to radial basis function models. Convolutional Neural Networks (CNNs) are inspired by the organization of the animal visual cortex. Each neuron receives input only from a local patch (corresponding to the receptive field in real neurons). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 15

54 Other Neural Networks In this lecture we cover only feed-forward neural networks, because this is the most basic and most relevant class for supervised (and unsupervised) learning and the class applied to SRL. A broad range of other network types has been developed. The linear+sigmoid processing model can be replaced by other functions. For example, this leads to radial basis function models. Convolutional Neural Networks (CNNs) are inspired by the organization of the animal visual cortex. Each neuron receives input only from a local patch (corresponding to the receptive field in real neurons). Synapses can form loops. This requires the introduction of time delays. Then we speak of Recurrent Neural Networks (RNNs). These are even more powerful models: they are not simple mappings, but stateful computers. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 15

55 Other Neural Networks In this lecture we cover only feed-forward neural networks, because this is the most basic and most relevant class for supervised (and unsupervised) learning and the class applied to SRL. A broad range of other network types has been developed. The linear+sigmoid processing model can be replaced by other functions. For example, this leads to radial basis function models. Convolutional Neural Networks (CNNs) are inspired by the organization of the animal visual cortex. Each neuron receives input only from a local patch (corresponding to the receptive field in real neurons). Synapses can form loops. This requires the introduction of time delays. Then we speak of Recurrent Neural Networks (RNNs). These are even more powerful models: they are not simple mappings, but stateful computers. Auto-encoders, (restricted) Boltzmann machines, and self-organizing maps are used for unsupervised learning. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 15

56 Other Neural Networks In this lecture we cover only feed-forward neural networks, because this is the most basic and most relevant class for supervised (and unsupervised) learning and the class applied to SRL. A broad range of other network types has been developed. The linear+sigmoid processing model can be replaced by other functions. For example, this leads to radial basis function models. Convolutional Neural Networks (CNNs) are inspired by the organization of the animal visual cortex. Each neuron receives input only from a local patch (corresponding to the receptive field in real neurons). Synapses can form loops. This requires the introduction of time delays. Then we speak of Recurrent Neural Networks (RNNs). These are even more powerful models: they are not simple mappings, but stateful computers. Auto-encoders, (restricted) Boltzmann machines, and self-organizing maps are used for unsupervised learning. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 15

57 Deep Learning Many NNs used e.g. for image processing are really deep, i.e., they consist of 10 layers or more. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 16

58 Deep Learning Many NNs used e.g. for image processing are really deep, i.e., they consist of 10 layers or more. We speak of deep learning. This has become one of the dominant buzz-words of the field (next to big data). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 16

59 Deep Learning Many NNs used e.g. for image processing are really deep, i.e., they consist of 10 layers or more. We speak of deep learning. This has become one of the dominant buzz-words of the field (next to big data). A central concept of deep learning is that lower layers extract basic features (e.g., edge detectors), while higher layers compose them to complex features (complex cells, invariant object detectors). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 16

60 Deep Learning Many NNs used e.g. for image processing are really deep, i.e., they consist of 10 layers or more. We speak of deep learning. This has become one of the dominant buzz-words of the field (next to big data). A central concept of deep learning is that lower layers extract basic features (e.g., edge detectors), while higher layers compose them to complex features (complex cells, invariant object detectors). This is in rough correspondence with our understanding of how the visual cortex processes images. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 16

61 Deep Learning Many NNs used e.g. for image processing are really deep, i.e., they consist of 10 layers or more. We speak of deep learning. This has become one of the dominant buzz-words of the field (next to big data). A central concept of deep learning is that lower layers extract basic features (e.g., edge detectors), while higher layers compose them to complex features (complex cells, invariant object detectors). This is in rough correspondence with our understanding of how the visual cortex processes images. Recently deep learning revolutionized a lot of fields like image and language procession, machine translations etc. It was also part of Alpha-Go. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 16

62 From Models to Learners The class of functions represented by neural networks is rich enough to represent the solution to any problem, provided that the network is big enough. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 17

63 From Models to Learners The class of functions represented by neural networks is rich enough to represent the solution to any problem, provided that the network is big enough. With a large enough hidden layer the network can approximate the optimal hypothesis arbitrarily well. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 17

64 From Models to Learners The class of functions represented by neural networks is rich enough to represent the solution to any problem, provided that the network is big enough. With a large enough hidden layer the network can approximate the optimal hypothesis arbitrarily well. However, this is not helpful in practice. It does not tell us how to actually set the network size, let alone the weights. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 17

65 From Models to Learners The class of functions represented by neural networks is rich enough to represent the solution to any problem, provided that the network is big enough. With a large enough hidden layer the network can approximate the optimal hypothesis arbitrarily well. However, this is not helpful in practice. It does not tell us how to actually set the network size, let alone the weights. Until now we have defined neural networks as a class of models. We do not have a learning rule yet. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 17

66 From Models to Learners The class of functions represented by neural networks is rich enough to represent the solution to any problem, provided that the network is big enough. With a large enough hidden layer the network can approximate the optimal hypothesis arbitrarily well. However, this is not helpful in practice. It does not tell us how to actually set the network size, let alone the weights. Until now we have defined neural networks as a class of models. We do not have a learning rule yet. Neural networks are trained based on stochastic gradient descent as described in the following. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 17

67 (Online) Steepest Descent Training Let w denote a vector collecting all weights of a neural network. This is a linearized version of all of its weight matrices. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 18

68 (Online) Steepest Descent Training Let w denote a vector collecting all weights of a neural network. This is a linearized version of all of its weight matrices. Let f w be the mapping represented by the network for particular weights w. Let E(w) = 1 S L(f w (x i ), y i ) i S denote the error of the network, as a function of the weights. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 18

69 (Online) Steepest Descent Training Let w denote a vector collecting all weights of a neural network. This is a linearized version of all of its weight matrices. Let f w be the mapping represented by the network for particular weights w. Let E(w) = 1 S L(f w (x i ), y i ) i S denote the error of the network, as a function of the weights. The sum may run over the whole data set ( S = n, batch mode), over small subsets ( S n, mini batches), or only over a single example ( S = 1, online mode). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 18

70 (Online) Steepest Descent Training E(w) = 1 S L(f w (x i ), y i ) i S The batch error is what we have called the empirical risk w.r.t. the loss function L. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 19

71 (Online) Steepest Descent Training E(w) = 1 S L(f w (x i ), y i ) i S The batch error is what we have called the empirical risk w.r.t. the loss function L. Its minimization is a straightforward learning strategy. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 19

72 (Online) Steepest Descent Training E(w) = 1 S L(f w (x i ), y i ) i S The batch error is what we have called the empirical risk w.r.t. the loss function L. Its minimization is a straightforward learning strategy. This is usually what we want when training a neural network. So why care about online and mini batch errors? Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 19

73 (Online) Steepest Descent Training E(w) = 1 S L(f w (x i ), y i ) i S The batch error is what we have called the empirical risk w.r.t. the loss function L. Its minimization is a straightforward learning strategy. This is usually what we want when training a neural network. So why care about online and mini batch errors? The reason is that the online error is much faster to compute, namely by a factor of n (size of the data set). Thus its use allows for many more gradient descent steps. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 19

74 (Online) Steepest Descent Training Assume we have computed the gradient of the error w E(w) with respect to the weights. Then we can perform a step of gradient descent with learning rate η to update the wights. w w η w E(w). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 20

75 Backpropagation Now we come to the computation of the error gradient w E(w). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 21

76 Backpropagation Now we come to the computation of the error gradient w E(w). The error is a simple sum over loss terms of the form E(w) = L(f w (x), y). We compute w E(w) in the following. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 21

77 Backpropagation Now we come to the computation of the error gradient w E(w). The error is a simple sum over loss terms of the form E(w) = L(f w (x), y). We compute w E(w) in the following. We write this error as ( ( ( E(w) = L σ W (n) σ W (n 1) σ (... W (1) x... ))) ), y, where W (k) are the weight matrices and σ is the component-wise non-linearity. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 21

78 Backpropagation Now we come to the computation of the error gradient w E(w). The error is a simple sum over loss terms of the form E(w) = L(f w (x), y). We compute w E(w) in the following. We write this error as ( ( ( E(w) = L σ W (n) σ W (n 1) σ (... W (1) x... ))) ), y, where W (k) are the weight matrices and σ is the component-wise non-linearity. The gradient can be calculated by the chain rule. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 21

79 Backpropagation Now we come to the computation of the error gradient w E(w). The error is a simple sum over loss terms of the form E(w) = L(f w (x), y). We compute w E(w) in the following. We write this error as ( ( ( E(w) = L σ W (n) σ W (n 1) σ (... W (1) x... ))) ), y, where W (k) are the weight matrices and σ is the component-wise non-linearity. The gradient can be calculated by the chain rule. Backpropagation is an algorithm for doing this fast. It will be introduced in the next lecture. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 21

80 Summary We have introduced a simple neuron model which is composed out of linear input and a non-linear transfer/activation function. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 22

81 Summary We have introduced a simple neuron model which is composed out of linear input and a non-linear transfer/activation function. We have composed multiple neurons in parallel to layers, and multiple layers in sequence to feed forward neural networks (multi layer Perceptrons, MLPs). Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 22

82 Summary We have introduced a simple neuron model which is composed out of linear input and a non-linear transfer/activation function. We have composed multiple neurons in parallel to layers, and multiple layers in sequence to feed forward neural networks (multi layer Perceptrons, MLPs). Neural networks are universal function approximators. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 22

83 Summary We have introduced a simple neuron model which is composed out of linear input and a non-linear transfer/activation function. We have composed multiple neurons in parallel to layers, and multiple layers in sequence to feed forward neural networks (multi layer Perceptrons, MLPs). Neural networks are universal function approximators. Networks can be trained by online or batch gradient descent. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 22

84 Summary We have introduced a simple neuron model which is composed out of linear input and a non-linear transfer/activation function. We have composed multiple neurons in parallel to layers, and multiple layers in sequence to feed forward neural networks (multi layer Perceptrons, MLPs). Neural networks are universal function approximators. Networks can be trained by online or batch gradient descent. The error gradient can be computed efficiently with the backpropagation algorithm. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 22

85 Summary We have introduced a simple neuron model which is composed out of linear input and a non-linear transfer/activation function. We have composed multiple neurons in parallel to layers, and multiple layers in sequence to feed forward neural networks (multi layer Perceptrons, MLPs). Neural networks are universal function approximators. Networks can be trained by online or batch gradient descent. The error gradient can be computed efficiently with the backpropagation algorithm. The weights usually end up in a local optimum, not in the global optimum. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 22

86 Summary We have introduced a simple neuron model which is composed out of linear input and a non-linear transfer/activation function. We have composed multiple neurons in parallel to layers, and multiple layers in sequence to feed forward neural networks (multi layer Perceptrons, MLPs). Neural networks are universal function approximators. Networks can be trained by online or batch gradient descent. The error gradient can be computed efficiently with the backpropagation algorithm. The weights usually end up in a local optimum, not in the global optimum. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 22

87 Acknowledgments Acknowledgments: We thank Tobias Glasmachers for providing us the material for this class which was taken from his lecture Machine Learning - Supervised Methods at the Ruhr-Uni-Bochum. Dr. Hajira Jabeen, Prof. Jens Lehmann (Feed-Forward) Neural Networks 23