Deep Neural Networks

Transcription

1 Deep Neural Networks DT2118 Speech and Speaker Recognition Giampiero Salvi KTH/CSC/TMH VT / 45

2 Outline State-to-Output Probability Model Artificial Neural Networks Perceptron Multi Layer Perceptron Error Backpropagation Hybrid HMM-MLP Deep Learning (Initialization) Deep Neural Networks Restricted Boltzmann Machines Deep Belief Networks 2 / 45

3 Outline State-to-Output Probability Model Artificial Neural Networks Perceptron Multi Layer Perceptron Error Backpropagation Hybrid HMM-MLP Deep Learning (Initialization) Deep Neural Networks Restricted Boltzmann Machines Deep Belief Networks 3 / 45

4 State-to-Output Probability Model... z n 1 z n z n x n 1 x n x n+1... is responsible for the discriminative power of the whole model GMMs used because easy to train and adapt discriminative training can improve results 4 / 45

5 State-to-Output Probability Model... z n 1 z n z n x n 1 x n x n+1... is responsible for the discriminative power of the whole model Alternatives: artificial neural networks (ANNs) deep neural networks (DNNs) support vector machines (SVMs) not used for ASR 4 / 45

6 Outline State-to-Output Probability Model Artificial Neural Networks Perceptron Multi Layer Perceptron Error Backpropagation Hybrid HMM-MLP Deep Learning (Initialization) Deep Neural Networks Restricted Boltzmann Machines Deep Belief Networks 5 / 45

7 Perceptron Known since the 1950 s [4] Input Transfer layer Sum Function x 0 Output x 1 x 2 x 3 w 0 w 1 w 2 w 3 w 4 f y x 4 [4] F. Rosenblatt. The perceptron: A perceiving and recognizing automaton. Tech. rep Cornell Aeronautical Laboratory, / 45

8 Perceptron input/output where y = f ( b + i w i x i ) f (z) = e z sigmoid f (z) = ez e z e z + e z hyperbolic tangent f (z) = max(0, z) rectified linear unit Equivalent to logistic regression (b = w 0 x 0 bias) 7 / 45

9 Perceptron: Linear Classification Learning adjust weights to correct errors y 2 a y 1 y 1 8 / 45

10 Multi-layer Perceptron [3] Output y 1 y 2 y 3 Hidden h 0 h 1 h 2 h 3 h 4 h 5 Input x 0 x 1 x 2 x 3 x 4 [3] F. Rosenblatt. Principles of neurodynamics. perceptrons and the theory of brain mechanisms. Tech. rep. DTIC Document, / 45

11 Universal Approximation Theorem First proposed by Gybenko [1] one single hidden layer and finite but appropriate number of neurons can approximate any function in R N with mild constraints [1] G. Gybenko. Approximation by superposition of sigmoidal functions. In: Mathematics of Control, Signals and Systems 2.4 (1989), pp / 45

12 Multi-layer Perceptron: Training Backpropagation algorithm [5] output unit hidden layer bias unit input units x 0 x 1 x 2 x d [5] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. Tech. rep. DTIC Document, / 45

13 Multi-layer Perceptron: Training Backpropagation algorithm [5] E output unit hidden layer bias unit input units x 0 x 1 x 2 x d [5] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. Tech. rep. DTIC Document, / 45

14 Learning Criteria Ideally minimise Expected Loss: J EL = E [ J(W, B, o, y) ] = J(W, B, o, y)p(o)do where o = features, y = labels but we do not know p(o) Use empirical learning criteria instead: Mean Square Error (MSE) Cross Entropy (CE) o 12 / 45

15 Mean Square Error Criterion J MSE = 1 M M J MSE (W, B, o m, y m ) m=1 J MSE (W, B, o m, y m ) = 1 v L y 2 2 = 1 ( v L y ) T ( v L y ) 2 13 / 45

16 Cross Entropy Criterion J CE = 1 M M J CE (W, B, o m, y m ) m=1 J CE (W, B, o m, y m ) = C i=1 y i log v L i Equivalent to minimising Kullback-Leibler divergence (KLD) 14 / 45

17 Update rules Wt+1 l Wt l ɛ Wt l bt+1 l bt l ɛ bt l To compute W l t and b l t we need the gradient of the criterion function. Key trick: chain rule of gradients f (g(x)): f x = f g g x 15 / 45

18 Backpropagation: Properties weights only depend on neighbouring variables algorithm finds local optimum sensitive to initialisation 16 / 45

19 Practical Issues preprocessing: Cepstral Mean Normalisation initialisation: random (symmetry breaking), linear range of activation function regularisation (weight decay, dropout) batch size selection sample randomisation momentum learning rate and stopping criterion 17 / 45

20 Output Layer Regression tasks: Linear layer v L = z L = W L v L 1 + b L Classification tasks: Softmax layer v L i = softmax i (z L ) = e zl i C j=1 ezl j 18 / 45

21 Probabilistic Interpretation 1. vi L [0, 1] i 2. C j=1 v i L = 1 Output activations are posterior probabilities of the classes given the observations v L i In speech: P(state sounds) = P(i o) 19 / 45

22 Hybrid HMM+Multi Layer Perceptron Figure from Yu and Deng 20 / 45

23 Combining probabilities HMMs use likelihoods P(sound state) MLPs and DNNs estimate posteriors P(state sound) We can combine with Bayes: P(sound state) = P(state sound)p(sound) P(state) P(state) can be estimated from the training set P(sound) is constant and can be ignored Use scaled likelihoods: P(sound state) = P(state sound) P(state) 21 / 45

24 Time Dependent and Recurrent ANNs 22 / 45

25 ANNs in ASR: Advantages discriminative in nature powerful time model: Time-Delayed Neural Networks (TDNNs) Recurrent Neural Networks (RNNs) 23 / 45

26 ANNs in ASR: Disadvantages training requires state level annotations (no EM available) usually annotations obtained with forced alignment not easy to adapt 24 / 45

27 Outline State-to-Output Probability Model Artificial Neural Networks Perceptron Multi Layer Perceptron Error Backpropagation Hybrid HMM-MLP Deep Learning (Initialization) Deep Neural Networks Restricted Boltzmann Machines Deep Belief Networks 25 / 45

28 Deep Neural Network Output y 1 y 2 y 3 Hidden 4 h 30 h 31 h 32 h 33 h 34 h 35 Hidden 3 h 20 h 21 h 22 h 23 h 24 h 25 Hidden 2 h 10 h 11 h 12 h 13 h 14 h 15 Input x 0 x 1 x 2 x 3 x 4 26 / 45

29 DNN: Motivation depth abstraction good initialisation (see later) fast computers, large datasets 27 / 45

30 DNN and MLPs no conceptual difference from MLPs Backpropagation alone not powerful enough local minima vanishing gradients 28 / 45

31 Deep Learning for Acoustic Modelling most promising technique at the moment pioneered by Geoffry Hinton (Univ. Toronto) most of the large companies are using it (Microsoft, Google, Nuance, IBM) unifies properties of generative and discriminative models 29 / 45

32 Deep Learning: Idea discriminative stuff observation label

33 Deep Learning: Idea discriminative stuff observation label

34 Deep Learning: Idea discriminative stuff generative stuff observation label observation label

35 Deep Learning: Idea discriminative stuff generative stuff observation label observation label 30 / 45

36 Deep Learning: Idea discriminative stuff generative stuff observation label observation label... but HMMs with Gaussian Mixure Models are also generative: why is this better? 30 / 45

37 Deep Learning: Idea #2 1. initialise DNN with Restricted Boltzmann Machines (RBM) that can be trained unsupervised 2. use fast learning procedure (Hinton) 3. use ridiculous amounts of unlabelled (cheap) data to train a ridiculous number of parameters in an unsupervised fashion 4. at the end, use small amounts of labelled (expensive) data and backpropagation to learn the labels 31 / 45

38 Restricted Boltzmann Machines (RBMs) First called Harmonium [6] binary nodes: Bernoulli distribution continuous nodes: Gaussian-Benoulli [6] P. Smolensky. Information processing in dynamical systems: Foundations of harmony theory. In: Department of Computer Science, University of Colorado, Boulder, Chap / 45

39 Restricted Boltzmann Machines (RBMs) Energy (Benoulli): E(v, h) = a T v b T h h T Wv Energy (Gaussian-Benoulli): E(v, h) = 1 2 (v a)t (v a) b T h h T Wv 33 / 45

40 RBM: Probabilistic Interpretation P(v, h) = e E(v,h) v,h e E(v,h) Posteriors (conditional independence): P(h v) = = i P(h i v) and P(v h) = = i P(v i h) 34 / 45

41 Binary Units: Cond Prob Posterior equals sigmoid function!! P(h i = 1 v) = = e (b i1+1w i, v) e (b i1+1w i, v) + e (b i0+0w i, v) e (b i1+1w i, v) e (b i1+1w i, v) + 1 = σ(b i 1 + 1W i, v) Same as Multi Layer Perceptron (viable for initialisation!) 35 / 45

42 Gaussian Units: Cond Prob with P(v h) = N (v; µ, Σ) µ = W T h + a Σ = I 36 / 45

43 RBM Training Stochastic Gradient Descend (minimise the negative log likelihood) J NLL (W, a, b, v) = log P(v) = F (v)+log v F (v) e where ( ) F (v) = log e E(v,h) is the free energy of the system. BUT: the gradient can not be computed exactly h 37 / 45

44 RBM Gradient J NLL (W, a, b, v) θ = F (v) θ ṽ F (ṽ) p(ṽ) θ first term increases prob of training data second term decreases prob density defined by the model 38 / 45

45 RBM Stochastic Gradient The general form is: [ E(v, h) θ J NLL (W, a, b, v) = θ data ] E(v, h) θ model Example: visible layer wij J NLL (W, a, b, v) = [ v i h j data v i h j model ] 39 / 45

46 Gibbs Sampling v i h j model computed with sampling Sample joint distribution of N variables, one at a time: P(X i X i ) where X i are all the other variables BUT: it takes exponential time to compute exactly 40 / 45

47 Contrastive Divergence Two tricks: 1. initialise the chain with a training sample 2. do not wait for convergence 41 / 45

48 RBMs and Deep Belief Networks Figure from Yu and Deng 42 / 45

49 Deep Belief Networks: Training Yee-Whye Teh (one of Hinton s students) observed that DBNs can be trained greedily for each layer: 1. train a RBM unsupervised 2. excite the network with training data to produce outputs 3. use the outputs to train next RBM 43 / 45

50 Final Step: Supervised Training Output y 1 y 2 y 3 Hidden 4 h 30 h 31 h 32 h 33 h 34 h 35 Hidden 3 h 20 h 21 h 22 h 23 h 24 h 25 Hidden 2 h 10 h 11 h 12 h 13 h 14 h 15 Input x 0 x 1 x 2 x 3 x 4 44 / 45

51 Deep Learning: Performance state-of-the-art on most ASR tasks Made people from University of Toronto, Microsoft, Google and IBM write a paper together [2] experiments with learning the features from speech signal Yu and Deng s book has many examples [2] G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury. Deep Neural Networks for Acoustic Modeling in Speech Recognition. In: IEEE Signal Processing Magazine (2012) 45 / 45