Lecture 3. Gaussian Mixture Models and Introduction to HMM s. Michael Picheny, Bhuvana Ramabhadran, Stanley F. Chen, Markus Nussbaum-Thom

Transcription

1 Lecture 3 Gaussian Mixture Models and Introduction to HMM s Michael Picheny, Bhuvana Ramabhadran, Stanley F. Chen, Markus Nussbaum-Thom Watson Group IBM T.J. Watson Research Center Yorktown Heights, New York, USA {picheny,bhuvana,stanchen,nussbaum}@us.ibm.com 3 February 2016

2 Administrivia 2 / 106 Lab 1 due Friday, February 12th at 6pm. Should have received username and password. Courseworks discussion has been started. TA office hours: Minda, Wed 2-4pm, EE lounge, 13th floor Mudd; Srihari, Thu 2-4pm, 122 Mudd.

3 Feedback Muddiest topic: MFCC(7), DTW(5), DSP (4), PLP (2) Comments (2+ votes): More examples/spend more time on examples (8) Want slides before class (4) More explanation of equations (3) Engage students more; ask more questions to class (2) 3 / 106

4 Where Are We? 4 / 106 Can extract features over time (MFCC, PLP, others) that... Characterize info in speech signal in compact form. Vector of features extracted 100 times a second

5 DTW Recap Training: record audio A w for each word w in vocab. Generate sequence of MFCC features A w (template for w). Test time: record audio A test, generate sequence of MFCC features A test. For each w, compute distance(a test, A w) using DTW. Return w with smallest distance. DTW computes distance between words represented as sequences of feature vectors... While accounting for nonlinear time alignment. Learned basic concepts (e.g., distances, shortest paths)... That will reappear throughout course. 5 / 106

6 What are Pros and Cons of DTW? 6 / 106

7 Pros 7 / 106 Easy to implement.

8 Cons: It s Ad Hoc Distance measures completely heuristic. Why Euclidean? Weight all dimensions of feature vector equally? - ugh!! Warping paths heuristic. Human derived constraints on warping paths like weights, etc - ugh!! Doesn t scale well Run DTW for each template in training data - what if large vocabulary? -ugh!! Plenty other issues. 8 / 106

9 Can We Do Better? Key insight 1: Learn as much as possible from data. e.g., distance measure; warping functions? Key insight 2: Use probabilistic modeling. Use well-described theories and models from... Probability, statistics, and computer science... Rather than arbitrary heuristics with ill-defined properties. 9 / 106

10 Next Two Main Topics Gaussian Mixture models (today) A probabilistic model of... Feature vectors associated with a speech sound. Principled distance between test frame... And set of template frames. Hidden Markov models (next week) A probabilistic model of... Time evolution of feature vectors for a speech sound. Principled generalization of DTW. 10 / 106

11 Part I 11 / 106 Gaussian Distributions

12 Gauss 12 / 106 Johann Carl Friedrich Gauss ( ) "Greatest Mathematician since Antiquity"

13 Gauss s Dog 13 / 106

14 The Scenario 14 / 106 Compute distance between test frame and frame of template Imagine 2d feature vectors instead of 40d for visualization.

15 Problem Formulation 15 / 106 What if instead of one training sample, have many?

16 Ideas 16 / 106 Average training samples; compute Euclidean distance. Find best match over all training samples. Make probabilistic model of training samples.

17 Where Are We? 17 / Gaussians in One Dimension 2 Gaussians in Multiple Dimensions 3 Estimating Gaussians From Data

18 Problem Formulation, Two Dimensions 18 / 106 Estimate P(x 1, x 2 ), the frequency... That training sample occurs at location (x 1, x 2 ).

19 Let s Start With One Dimension 19 / 106 Estimate P(x), the frequency... That training sample occurs at location x.

20 The Gaussian or Normal Distribution 20 / 106 P µ,σ 2(x) = N (µ, σ 2 ) = 1 e (x µ) 2 2σ 2 2πσ Parametric distribution with two parameters: µ = mean (the center of the data). σ 2 = variance (how wide data is spread).

21 Visualization 21 / 106 Density function: µ 4σ µ 2σ µ µ + 2σ µ + 4σ Sample from distribution: µ 4σ µ 2σ µ µ + 2σ µ + 4σ

22 Properties of Gaussian Distributions Is valid probability distribution. 1 e (x µ) 2 2σ 2 dx = 1 2πσ Central Limit Theorem: Sums of large numbers of identically distributed random variables tend to Gaussian. Lots of different types of data look bell-shaped. Sums and differences of Gaussian random variables... Are Gaussian. If X is distributed as N (µ, σ 2 )... ax + b is distributed as N (aµ + b, (aσ) 2 ). Negative log looks like weighted Euclidean distance! ln 2πσ + (x µ)2 2σ 2 22 / 106

23 Where Are We? 23 / Gaussians in One Dimension 2 Gaussians in Multiple Dimensions 3 Estimating Gaussians From Data

24 Gaussians in Two Dimensions 24 / 106 N (µ 1, µ 2, σ 2 1, σ 2 2) = 1 2πσ 1 σ 2 1 r 2 e 1 2(1 r 2 ) (x 1 µ 1 ) 2 σ 2 1 «2rx 1 x 2 + (x 2 µ 2 )2 σ 1 σ 2 σ 2 2 If r = 0, simplifies to 1 e (x 1 µ 1) 2σ 1 2 2πσ e (x 2 µ 2) 2σ 2 2 = N (µ 1, σ1)n 2 (µ 2, σ2) 2 2πσ2 i.e., like generating each dimension independently.

25 25 / 106 Example: r = 0, σ 1 = σ 2 x 1, x 2 uncorrelated. Knowing x 1 tells you nothing about x 2.

26 26 / 106 Example: r = 0, σ 1 σ 2 x 1, x 2 can be uncorrelated and have unequal variance.

27 27 / 106 Example: r > 0, σ 1 σ 2 x 1, x 2 correlated. Knowing x 1 tells you something about x 2.

28 Generalizing to More Dimensions 28 / 106 If we write following matrix: Σ = σ1 2 rσ 1 σ 2 rσ 1 σ 2 σ2 2 then another way to write two-dimensional Gaussian is: N (µ, Σ) = 1 (2π) d/2 Σ where x = (x 1, x 2 ), µ = (µ 1, µ 2 ). 1 e 2 (x µ)t Σ 1 (x µ) 1/2 More generally, µ and Σ can have arbitrary numbers of components. Multivariate Gaussians.

29 Diagonal and Full Covariance Gaussians Let s say have 40d feature vector. How many parameters in covariance matrix Σ? The more parameters,... The more data you need to estimate them. In ASR, usually assume Σ is diagonal d params. This is why like having uncorrelated features! 29 / 106

30 Computing Gaussian Log Likelihoods Why log likelihoods? Full covariance: log P(x) = d 2 ln(2π) 1 2 ln Σ 1 2 (x µ)t Σ 1 (x µ) Diagonal covariance: log P(x) = d d 2 ln(2π) ln σ i 1 2 i=1 d (x i µ i ) 2 /σi 2 i=1 Again, note similarity to weighted Euclidean distance. Terms on left independent of x; precompute. A few multiplies/adds per dimension. 30 / 106

31 Where Are We? 31 / Gaussians in One Dimension 2 Gaussians in Multiple Dimensions 3 Estimating Gaussians From Data

32 Estimating Gaussians Give training data, how to choose parameters µ, Σ? Find parameters so that resulting distribution... Matches data as well as possible. Sample data: height, weight of baseball players / 106

33 Maximum-Likelihood Estimation (Univariate) 33 / 106 One criterion: data matches distribution well... If distribution assigns high likelihood to data. Likelihood of string of observations x 1, x 2,..., x N is... Product of individual likelihoods. N L(x1 N 1 µ, σ) = e (x i µ) 2 2σ 2 2πσ i=1 Maximum likelihood estimation: choose µ, σ... That maximizes likelihood of training data. (µ, σ) MLE = arg max L(x1 N µ, σ) µ,σ

34 Why Maximum-Likelihood Estimation? Assume we have correct model form. Then, as the number of training samples increases... ML estimates approach true parameter values (consistent) ML estimators are the best! (efficient) ML estimation is easy for many types of models. Count and normalize! 34 / 106

35 What is ML Estimate for Gaussians? Much easier to work with log likelihood L = ln L: L(x1 N µ, σ) = N 2 ln 2πσ2 1 N (x i µ) 2 2 σ 2 Take partial derivatives w.r.t. µ, σ: L(x1 N µ, σ) N (x i µ) = µ σ 2 i=1 i=1 L(x1 N µ, σ) = N σ 2 2σ + N (x i µ) 2 2 σ 4 Set equal to zero; solve for µ, σ 2. µ = 1 N x i σ 2 = 1 N N i=1 i=1 N (x i µ) 2 i=1 35 / 106

36 What is ML Estimate for Gaussians? 36 / 106 Multivariate case. N µ = 1 N Σ = 1 N i=1 x i N (x i µ) T (x i µ) i=1 What if diagonal covariance? Estimate params for each dimension independently.

37 Example: ML Estimation 37 / 106 Heights (in.) and weights (lb.) of 1033 pro baseball players. Noise added to hide discretization effects. stanchen/e6870/data/mlb_data.dat height weight

38 Example: ML Estimation 38 /

39 Example: Diagonal Covariance 39 / 106 µ 1 = 1 ( ) = µ 2 = 1 ( ) = σ1 2 = 1 [ ( ) 2 + ( ) 2 + ) ] 1033 = 5.43 σ2 2 = 1 [ ( ) 2 + ( ) 2 + ) ] 1033 =

40 Example: Diagonal Covariance 40 /

41 Example: Full Covariance 41 / 106 Mean; diagonal elements of covariance matrix the same. Σ 12 = Σ 21 = 1 [( ) ( ) ( ) ( ) + )] = µ = [ ] Σ =

42 Example: Full Covariance 42 /

43 Recap: Gaussians Lots of data looks Gaussian. Central limit theorem. ML estimation of Gaussians is easy. Count and normalize. In ASR, mostly use diagonal covariance Gaussians. Full covariance matrices have too many parameters. 43 / 106

44 Part II 44 / 106 Gaussian Mixture Models

45 Problems with Gaussian Assumption 45 / 106

46 Problems with Gaussian Assumption Sample from MLE Gaussian trained on data on last slide. Not all data is Gaussian! 46 / 106

47 Problems with Gaussian Assumption 47 / 106 What can we do? What about two Gaussians? where p 1 + p 2 = 1. P(x) = p 1 N (µ 1, Σ 1 ) + p 2 N (µ 2, Σ 2 )

48 Gaussian Mixture Models (GMM s) More generally, can use arbitrary number of Gaussians: P(x) = j p j 1 (2π) d/2 Σ j 1 e 2 (x µ j )T Σ 1 j (x µ j ) 1/2 where j p j = 1 and all p j 0. Also called mixture of Gaussians. Can approximate any distribution of interest pretty well... If just use enough component Gaussians. 48 / 106

49 Example: Some Real Acoustic Data 49 / 106

50 Example: 10-component GMM (Sample) 50 / 106

51 Example: 10-component GMM (µ s, σ s) 51 / 106

52 ML Estimation For GMM s Given training data, how to estimate parameters... i.e., the µ j, Σ j, and mixture weights p j... To maximize likelihood of data? No closed-form solution. Can t just count and normalize. Instead, must use an optimization technique... To find good local optimum in likelihood. Gradient search Newton s method Tool of choice: The Expectation-Maximization Algorithm. 52 / 106

53 Where Are We? 53 / The Expectation-Maximization Algorithm 2 Applying the EM Algorithm to GMM s

54 Wake Up! 54 / 106 This is another key thing to remember from course. Used to train GMM s, HMM s, and lots of other things. Key paper in 1977 by Dempster, Laird, and Rubin [2]; citations to date. "the innovative Dempster-Laird-Rubin paper in the Journal of the Royal Statistical Society received an enthusiastic discussion at the Royal Statistical Society meeting...calling the paper "brilliant""

55 What Does The EM Algorithm Do? Finds ML parameter estimates for models... With hidden variables. Iterative hill-climbing method. Adjusts parameter estimates in each iteration... Such that likelihood of data... Increases (weakly) with each iteration. Actually, finds local optimum for parameters in likelihood. 55 / 106

56 What is a Hidden Variable? A random variable that isn t observed. Example: in GMMs, output prob depends on... The mixture component that generated the observation But you can t observe it Important concept. Let s discuss!!!! 56 / 106

57 Mixtures and Hidden Variables 57 / 106 So, to compute prob of observed x, need to sum over... All possible values of hidden variable h: P(x) = h P(h, x) = h P(h)P(x h) Consider probability distribution that is a mixture of Gaussians: P(x) = p j N (µ j, Σ j ) j Can be viewed as hidden model. h Which component generated sample. P(h) = p j ; P(x h) = N (µ j, Σ j ). P(x) = h P(h)P(x h)

58 The Basic Idea If nail down hidden value for each x i,... Model is no longer hidden! e.g., data partitioned among GMM components. So for each data point x i, assign single hidden value h i. Take h i = arg max h P(h)P(x i h). e.g., identify GMM component generating each point. Easy to train parameters in non-hidden models. Update parameters in P(h), P(x h). e.g., count and normalize to get MLE for µ j, Σ j, p j. Repeat! 58 / 106

59 The Basic Idea 59 / 106 Hard decision: For each x i, assign single h i = arg max h P(h, x i )... With count 1. Test: what is P(h, x i ) for Gaussian distribution? Soft decision: For each x i, compute for every h... the Posterior prob P(h x i ) = P(h,x i ) Ph. P(h,x i ) Also called the fractional count e.g., partition event across every GMM component. Rest of algorithm unchanged.

60 The Basic Idea, using more Formal Terminology Initialize parameter values somehow. For each iteration... Expectation step: compute posterior (count) of h for each x i. P(h x i ) = P(h, x i) h P(h, x i) Maximization step: update parameters. Instead of data x i with hidden h, pretend... Non-hidden data where... (Fractional) count of each (h, x i ) is P(h x i ). 60 / 106

61 Example: Training a 2-component GMM 61 / 106 Two-component univariate GMM; 10 data points. The data: x 1,..., x , 7.6, 4.2, 2.6, 5.1, 4.0, 7.8, 3.0, 4.8, 5.8 Initial parameter values: p 1 µ 1 σ 2 1 p 2 µ 2 σ Training data; densities of initial Gaussians.

62 The E Step 62 / 106 x i p 1 N 1 p 2 N 2 P(x i ) P(1 xi ) P(2 xi ) P(h x i ) = P(h, x i) h P(h, x i) = p h N h P(x i ) h {1, 2}

63 The M Step View: have non-hidden corpus for each component GMM. For hth component, have P(h x i ) counts for event x i. Estimating µ: fractional events. µ = 1 N N x i µ h = i=1 i 1 P(h x i ) N P(h x i )x i i=1 µ 1 = ( ) = 3.98 Similarly, can estimate σ 2 h with fractional events. 63 / 106

64 The M Step (cont d) 64 / 106 What about the mixture weights p h? To find MLE, count and normalize! p 1 = = 0.59

65 The End Result 65 / 106 iter p 1 µ 1 σ1 2 p 2 µ 2 σ

66 First Few Iterations of EM 66 / 106 iter 0 iter 1 iter 2

67 Later Iterations of EM 67 / 106 iter 2 iter 3 iter 10

68 Why the EM Algorithm Works 68 / 106 x = (x 1, x 2,...) = whole training set; h = hidden. θ = parameters of model. Objective function for MLE: (log) likelihood. L(θ) = log P θ (x) = log P θ (x, h) log P θ (h x) Form expectation with respect to θ n, the estimate of θ on the n th estimation iteration: P θ n(h x) log P θ (x) = h h h P θ n(h x) log P θ (x, h) P θ n(h x) log P θ (h x) rewrite as : log P θ (x) = Q(θ θ n ) + H(θ θ n )

69 Why the EM Algorithm Works 69 / 106 log P θ (x) = Q(θ θ n ) + H(θ θ n ) What is Q? In the Gaussian example above Q is just P θ n(h x) log p h N x (µ h, Σ h ) h It can be shown (using Gibb s inequality) that H(θ θ n ) H(θ n θ n ) for any θ θ n So that means that any choice of θ that increases Q will increase log P θ (x). Typically we just pick θ to maximize Q altogether, can often be done in closed form.

70 The E Step 70 / 106 Compute Q.

71 The M Step 71 / 106 Maximize Q with respect to θ Then repeat - E/M, E/M till likelihood stops improving significantly. That s the E-M algorithm in a nutshell!

72 Discussion EM algorithm is elegant and general way to... Train parameters in hidden models... To optimize likelihood. Only finds local optimum. Seeding is of paramount importance. 72 / 106

73 Where Are We? 73 / The Expectation-Maximization Algorithm 2 Applying the EM Algorithm to GMM s

74 Another Example Data Set 74 / 106

75 Question: How Many Gaussians? Method 1 (most common): Guess! Method 2: Bayesian Information Criterion (BIC)[1]. Penalize likelihood by number of parameters. BIC(C k ) = k { 1 2 n j log Σ j } Nk(d + 1 d(d + 1)) 2 j=1 k = Gaussian components. d = dimension of feature vector. n j = data points for Gaussian j; N = total data points. Discuss! 75 / 106

76 The Bayesian Information Criterion 76 / 106 View GMM as way of coding data for transmission. Cost of transmitting model number of params. Cost of transmitting data log likelihood of data. Choose number of Gaussians to minimize cost.

77 Question: How To Initialize Parameters? Set mixture weights p j to 1/k (for k Gaussians). Pick N data points at random and... Use them to seed initial values of µ j. Set all σ s to arbitrary value... Or to global variance of data. Extension: generate multiple starting points. Pick one with highest likelihood. 77 / 106

78 Another Way: Splitting Start with single Gaussian, MLE. Repeat until hit desired number of Gaussians: Double number of Gaussians by perturbing means... Of existing Gaussians by ±ɛ. Run several iterations of EM. 78 / 106

79 Question: How Long To Train? i.e., how many iterations of EM? Guess. Look at performance on training data. Stop when change in log likelihood per event... Is below fixed threshold. Look at performance on held-out data. Stop when performance no longer improves. 79 / 106

80 The Data Set 80 / 106

81 Sample From Best 1-Component GMM 81 / 106

82 The Data Set, Again 82 / 106

83 20-Component GMM Trained on Data 83 / 106

84 20-Component GMM µ s, σ s 84 / 106

85 Acoustic Feature Data Set 85 / 106

86 5-Component GMM; Starting Point A 86 / 106

87 5-Component GMM; Starting Point B 87 / 106

88 5-Component GMM; Starting Point C 88 / 106

89 Solutions With Infinite Likelihood 89 / 106 Consider log likelihood; two-component 1d Gaussian. ( ) N 1 ln p 1 e (x i µ 2 1) 1 2σ p 2 e (x i µ 2 2) 2σ 2 2 2πσ1 2πσ2 i=1 If µ 1 = x 1, above reduces to ( 1 ln 2 + 2πσ πσ 2 e 1 2 (x 1 µ 2 ) 2 σ 2 2 ) + N... i=2 which goes to as σ 1 0. Only consider finite local maxima of likelihood function. Variance flooring. Throw away Gaussians with count below threshold.

90 Recap GMM s are effective for modeling arbitrary distributions. State-of-the-art in ASR for decades (though may be superseded by NNs at some point, discuss later in course) The EM algorithm is primary tool for training GMM s (and lots of other things) Very sensitive to starting point. Initializing GMM s is an art. 90 / 106

91 References 91 / 106 S. Chen and P.S. Gopalakrishnan, Clustering via the Bayesian Information Criterion with Applications in Speech Recognition, ICASSP, vol. 2, pp , A.P. Dempster, N.M. Laird, D.B. Rubin, Maximum Likelihood from Incomplete Data via the EM Algorithm, Journal of the Royal Stat. Society. Series B, vol. 39, no. 1, 1977.

92 What s Next: Hidden Markov Models Replace DTW with probabilistic counterpart. Together, GMM s and HMM s comprise... Unified probabilistic framework. Old paradigm: New paradigm: w = arg min distance(a test, A w) w vocab w = arg max P(A test w) w vocab 92 / 106

93 Part III 93 / 106 Introduction to Hidden Markov Models

94 Introduction to Hidden Markov Models 94 / 106 The issue of weights in DTW. Interpretation of DTW grid as Directed Graph. Adding Transition and Output Probabilities to the Graph gives us an HMM! The three main HMM operations.

95 Another Issue with Dynamic Time Warping 95 / 106 Weights are completely heuristic! Maybe we can learn weights from data? Take many utterances...

96 Learning Weights From Data For each node in DP path, count number of times move up right and diagonally. Normalize number of times each direction taken by total number of times node was actually visited (= C/N) Take some constant times reciprocal as weight (αn/c) Example: particular node visited 100 times. Move 40 times; 20 times; 40 times. Set weights to 2.5, 5, and 2.5, (or 1, 2, and 1). Point: weight distribution should reflect... Which directions are taken more frequently at a node. Weight estimation not addressed in DTW... But central part of Hidden Markov models. 96 / 106

97 DTW and Directed Graphs 97 / 106 Take following Dynamic Time Warping setup: Let s look at representation of this as directed graph:

98 DTW and Directed Graphs 98 / 106 Another common DTW structure: As a directed graph: Can represent even more complex DTW structures... Resultant directed graphs can get quite bizarre.

99 Path Probabilities Let s assign probabilities to transitions in directed graph: a ij is transition probability going from state i to state j, where j a ij = 1. Can compute probability P of individual path just using transition probabilities a ij. 99 / 106

100 Path Probabilities It is common to reorient typical DTW pictures: Above only describes path probabilities associated with transitions. Also need to include likelihoods associated with observations. 100 / 106

101 Path Probabilities As in GMM discussion, let us define likelihood of producing observation x i from state j as b j (x i ) = m c jm 1 (2π) d/2 Σ jm 1 e 2 (x i µ jm ) T Σ 1 jm (x i µ jm ) 1/2 where c jm are mixture weights associated with state j. This state likelihood is also called the output probability associated with state. 101 / 106

102 Path Probabilities 102 / 106 In this case, likelihood of entire path can be written as:

103 Hidden Markov Models The output and transition probabilities define a Hidden Markov Model or HMM. Since probabilities of moving from state to state only depend on current and previous state, model is Markov. Since only see observations and have to infer states after the fact, model is hidden. One may consider HMM to be generative model of speech. Starting at upper left corner of trellis, generate observations according to permissible transitions and output probabilities. Not only can compute likelihood of single path... Can compute overall likelihood of observation string... As sum over all paths in trellis. 103 / 106

104 HMM: The Three Main Tasks Compute likelihood of generating string of observations from HMM (Forward algorithm). Compute best path from HMM (Viterbi algorithm). Learn parameters (output and transition probabilities) of HMM from data (Baum-Welch a.k.a. Forward-Backward algorithm). 104 / 106

105 Part IV 105 / 106 Epilogue

106 Course Feedback 1 Was this lecture mostly clear or unclear? What was the muddiest topic? 2 Other feedback (pace, content, atmosphere)? 106 / 106