Mixture Models & EM algorithm Lecture 21

Transcription

1 Mixture Models & EM algorithm Lecture 21 David Sontag New York University Slides adapted from Carlos Guestrin, Dan Klein, Luke Dan Weld, Vibhav Gogate, and Andrew Moore

2 The Evils of Hard Assignments? Clusters may overlap Some clusters may be wider than others Distances can be deceiving!

3 ProbabilisOc Clustering Try a probabilisoc model! allows overlaps, clusters of different size, etc. Can tell a genera&ve story for data P(X Y) P(Y) Challenge: we need to esomate model parameters without labeled Ys Y X 1 X 2?? ?? ?? ?? ??

4 The General GMM assumpoon P(Y): There are k components P(X Y): Each component generates data from a mul<variate Gaussian with mean μ i and covariance matrix Σ i Each data point is sampled from a genera&ve process: 1. Choose component i with probability P(y=i) 2. Generate datapoint ~ N(m i, Σ i ) Gaussian mixture model (GMM) µ 1 µ 2 µ 3

5 What Model Should We Use? Depends on X! Here, maybe Gaussian Naïve Bayes? MulOnomial over clusters Y (Independent) Gaussian for each X i given Y p(y i = y k )=θ k Y X 1 X 2?? ?? ?? ?? ??

6 Could we make fewer assumpoons? What if the X i co- vary? What if there are mulople peaks? Gaussian Mixture Models! P(Y) soll mulonomial P(X Y) is a mulovariate Gaussian distribuoon: P(X = x j Y = i) = 1 (2! ) m/2! i exp # " 1 1/2 2 x " µ j i $ % ( ) T! "1 i ( x j " µ i ) & ' (

7 MulOvariate Gaussians P(X = P(X=x x j Y j )= = i) = 1 (2! ) m/2! i exp # " 1 1/2 2 x j " µ i $ % ( ) T! "1 i ( x j " µ i ) & ' ( Σ idenoty matrix

8 MulOvariate Gaussians P(X = P(X=x x j Y j )= = i) = 1 (2! ) m/2! i exp # " 1 1/2 2 x j " µ i $ % ( ) T! "1 i ( x j " µ i ) & ' ( Σ = diagonal matrix X i are independent ala Gaussian NB

9 MulOvariate Gaussians P(X = P(X=x x j Y j )= = i) = 1 (2! ) m/2! i exp # " 1 1/2 2 x j " µ i $ % ( ) T! "1 i ( x j " µ i ) & ' ( Σ = arbitrary (semidefinite) matrix: - specifies rotaoon (change of basis) - eigenvalues specify relaove elongaoon

10 Eigenvalue, λ, of Σ MulOvariate Gaussians Covariance matrix, Σ, = degree to which x i vary together 1 P(X = x j Y = i) = (2! ) m/2! i exp # P(X=x " 1 1/2 2 x j )= j " µ i $ % ( ) T! "1 i ( x j " µ i ) & ' (

11 Mixtures of Gaussians (1) Old Faithful Data Set Time to ErupOon DuraOon of Last ErupOon

12 Mixtures of Gaussians (1) Old Faithful Data Set Single Gaussian Mixture of two Gaussians

13 Mixtures of Gaussians (2) Combine simple models into a complex model: Component Mixing coefficient K=3

14 Mixtures of Gaussians (3)

15 EliminaOng Hard Assignments to Clusters Model data as mixture of mulovariate Gaussians

16 EliminaOng Hard Assignments to Clusters Model data as mixture of mulovariate Gaussians

17 EliminaOng Hard Assignments to Clusters Model data as mixture of mulovariate Gaussians Shown is the posterior probability that a point was generated from i th Gaussian: Pr(Y = i x)

18 ML esomaoon in supervised semng Univariate Gaussian Mixture of Mul&variate Gaussians ML esomate for each of the MulOvariate Gaussians is given by: n k! x n! k ML = 1 n # k k j=1 n j=1 µ ML = 1 n ( x j " µ ) ML x j " µ ML ( ) T Just sums over x generated from the k th Gaussian

19 MLE: That was easy! But what if unobserved data? argmax θ j P(y j,x j ) θ: all model parameters eg, class probs, means, and variances But we don t know y j s!!! Maximize marginal likelihood: K argmax θ j P(x j ) = argmax j k=1 P(Y j =k, x j )

20 How do we opomize? Closed Form? Maximize marginal likelihood: argmax θ j P(x j ) = argmax j k=1 P(Y j =k, x j ) Almost always a hard problem! Usually no closed form soluoon Even when lgp(x,y) is convex, lgp(x) generally isn t For all but the simplest P(X), we will have to do gradient ascent, in a big messy space with lots of local opomum K

21 Learning general mixtures of Gaussian P(y = k x j ) 1 (2π) m / 2 Σ k exp 1 1/ 2 2 x j µ k ( ) T Σ 1 ( k x j µ ) k P(y = k) Marginal likelihood: m P(x j ) = P(x j, y = k) j =1 = m j =1 m K k =1 K j =1 k =1 1 (2π) m / 2 Σ k exp 1 1/ 2 2 x j µ k ( ) T Σ 1 k ( x j µ k ) P(y = k) Need to differenoate and solve for μ k, Σ k, and P(Y=k) for k=1..k There will be no closed form soluoon, gradient is complex, lots of local opomum Wouldn t it be nice if there was a be<er way!?!

22 ExpectaOon MaximizaOon

23 The EM Algorithm A clever method for maximizing marginal likelihood: argmax θ j P(x j ) = argmax θ j k=1 K P(Y j =k, x j ) A type of gradient ascent that can be easy to implement (eg, no line search, learning rates, etc.) Alternate between two steps: Compute an expectaoon Compute a maximizaoon Not magic: s&ll op&mizing a non- convex func&on with lots of local op&ma The computaoons are just easier (oqen, significantly so!)

24 EM: Two Easy Steps Objec<ve: argmax θ lg j k=1 K P(Y j =k, x j θ) = j lg k=1 K P(Y j =k, x j θ) Data: {x j j=1.. n} Nota<on a bit inconsistent Parameters = θ=λ E- step: Compute expectaoons to fill in missing y values according to current parameters, θ For all examples j and values k for Y j, compute: P(Y j =k x j, θ) M- step: Re- esomate the parameters with weighted MLE esomates Set θ = argmax θ j k P(Y j =k x j, θ) log P(Y j =k, x j θ) Especially useful when the E and M steps have closed form soluoons!!!

25 EM algorithm: Pictorial View Given a set of Parameters and training data Class assignment is probabilisoc or weighted (soq EM) Supervised learning problem Relearn the parameters based on the new training data Class assignment is hard (hard EM) EsOmate the class of each training example using the parameters yielding new (weighted) training data

26 Simple example: learn means only! Consider: 1D data Mixture of k=2 Gaussians Variances fixed to σ=1 DistribuOon over classes is uniform Just need to esomate μ 1 and μ m K m K P(x,Y j = k) exp 1 2σ 2 j =1 k =1 j =1 k =1 =2 x µ k 2 P(Y j = k)

27 EM for GMMs: only learning means Iterate: On the t th iteraoon let our esomates be λ t = { μ 1 (t), μ 2 (t) μ K (t) } E- step Compute expected classes of all datapoints M- step ( ) exp 1 P Y j = k x j,µ 1...µ K 2σ 2 x j µ k Compute most likely new μs given class expectaoons 2 P Y j = k ( ) µ k = m j =1 m j =1 P( Y j = k x ) j P( Y j = k x ) j x j

28 E.M. for General GMMs Iterate: On the t th iteraoon let our esomates be λ t = { μ 1 (t), μ 2 (t) μ K (t), Σ 1 (t), Σ 2 (t) Σ K (t), p 1 (t), p 2 (t) p K (t) } E- step Compute expected classes of all datapoints for each class M- step P( Y j = k x j,λ ) (t t p ) k p( (t x j µ ) (t ) k,σ ) k Compute weighted MLE for μ given expected classes above ( t +1 µ ) k = P Y j = k x j,λ t x j j j ( ) P( Y j = k x j,λ ) t p k (t +1) = j ( t +1 Σ ) k = P( Y j = k x j,λ ) t m t +1 P Y j = k x j,λ t x j µ k j ( ) Just evaluate a Gaussian at x j j p k (t) is shorthand for esomate of P(y=k) on t th iteraoon ( ) ( t +1) [ ] x j µ k P( Y j = k x j,λ ) t m = #training examples [ ] T

29 Gaussian Mixture Example: Start

30 Aqer first iteraoon

31 Aqer 2nd iteraoon

32 Aqer 3rd iteraoon

33 Aqer 4th iteraoon

34 Aqer 5th iteraoon

35 Aqer 6th iteraoon

36 Aqer 20th iteraoon

37 What if we do hard assignments? Iterate: On the t th iteraoon let our esomates be λ t = { μ 1 (t), μ 2 (t) μ K (t) } E- step Compute expected classes of all datapoints P( Y j = k x j,µ 1...µ ) K exp 1 2 x 2σ 2 j µ k P Y j = k M- step Compute most likely new μs given class expectaoons µ k = m j =1 m j =1 P( Y j = k x ) j P( Y j = k x ) j x j ( ) x j µ k = δ Y j = k, x j m j =1 ( ) ( ) δ Y j = k, x j Equivalent to k- means clustering algorithm!!! δ represents hard assignment to most likely or nearest cluster

38 Lets look at the math behind the magic! Next lecture, we will argue that EM: OpOmizes a bound on the likelihood Is a type of coordinate ascent Is guaranteed to converge to a (oqen local) opoma