EM & Variational Bayes

Transcription

1 EM & Variational Bayes Hanxiao Liu September 9, / 19

2 Outline 1. EM Algorithm 1.1 Introduction 1.2 Example: Mixture of vmfs 2. Variational Bayes 2.1 Introduction 2.2 Example: Bayesian Mixture of Gaussians 2 / 19

3 MLE by Gradient Ascent Goal: maximize L θ; X = log p X θ w.r.t θ Gradient Ascent GA One-step view: θ t+1 L θ t ; X + θ t Two-step view: 1. Q θ; θ t = L θ t ; X + θ t θ L θ t ; X 1 2 θ t θ θ t+1 argmax θ Q θ; θ t Drawbacks 1. L can be too complicated to work with 2. Too general to be efficient for structured problems 3 / 19

4 MLE by EM Expectation-maximization EM 1. Expectation: Q θ; θ t = E Z X,θ t L θ; X, Z 2. Maximization: θ t+1 argmax θ Q θ; θ t Replace L θ; X }{{} log-likelihood by L θ; X, Z }{{} complete log-likelihood L θ; X, Z is a random function w.r.t Z use the expected function as a surrogate 4 / 19

5 why EM is superior A comparison between Q θ, θ t, i.e., the local concave model 1. EM Q θ; θ t = E Z X,θ t L θ; X, Z = L θ; X D KL p Z X, θ t p Z X, θ + C 2. GA Q θ; θ t = L θ t ; X + θ t θ L θ t ; X 1 θ t θ / 19

6 Example: vmf mixture Notations { } X = {x i } n i=1, θ = π k 1, {µ i, κ i } k i=1 Z = {z ij {0, 1}} Log-likelihood zij = 1 = x i the j-th mixture component n n k L θ; X = log p x i θ = log i=1 i=1 j=1 Complete log-likelihood π j vmf x i µ j, κ j } {{ } log sum coupling n n k L θ; X, Z = log p x i, z i θ = z ij log π j vmf x i µ j, κ j i=1 i=1 j=1 6 / 19

7 E-step Compute Q where θ; θ t = EZ X,θ t L θ; X, Z Q π, µ, κ; π t, µ t, κ t = E Z X,π t,µ t,κ t n k i=1 j=1 z ij log π j vmf x i µ j, κ j n k = wij t log vmf x i µ j, κ j + wij t log π j i=1 j=1 wij t = E zij X,π t,µ t,κ t [z ij] = p z ij = 1 x i, π t, µ t, κ t πj t vmf x i µ t j, κ t j = ku=1 πu t vmf x i µ t u, κ t u 7 / 19

8 M-step Maximize Q π, µ, κ; π t, µ t, κ t = n k i=1 j=1 wij t log vmf x i µ j, κ j +wij t log π j w.r.t π, µ and κ s.t. π 1 = 1 and µ j 2 = 1, j [k] To impose constraints, maximize Q = Q + λ 1 π 1 + k j=1 ν j 1 µ j µ j 8 / 19

9 M-step Q π, µ, κ; π t, µ t, κ t = n k i=1 j=1 wij t log vmf x i µ j, κ j + wij t log π j + λ 1 π 1 + k j=1 ν j 1 µ j µ j Updating π t j Combining k j π j = k j w t ij = 1 with πj Q = ni=1 w t ij π j λ = 0 = π t+1 j n i=1 wt ij n 9 / 19

10 M-step Q π, µ, κ; π t, µ t, κ t = n k i=1 j=1 wij t log vmf x i µ j, κ j + wij t log π j + λ 1 π 1 + k j=1 ν j 1 µ j µ j Updating µ t j log vmf x i µ j, κ j = κ j µ j x i + C w.r.t µ j = µ t+1 j r j r j 2 µj Q = κ j n i=1 w t ijx i ν j µ j = 0 where r j = n i=1 w t ijx i 10 / 19

11 / 19 M-step Updating κ t j C p κ j = p κ 2 1 j 2π p 2 I p 2 1κ j the recurrence property of modified Bessel function 1 κj log I p 2 1 κ j = p 2 1 κ j + I p κ j 2 I p 2 1κ j n κj Q = wij t I p κ j 2 I p i=1 2 1 κ j + µ j x i = 0 = I p 2 κ j 2 1 κ j = r j = κj t+1 I p r jp r 3 j 1 r 2 j [?] where r j = n i=1 wt ij µ j x i n i=1 wt ij

12 An alternative view of EM EM - original definition 1. Expectation: Q θ; θ t = E Z X,θ t L θ; X, Z why? 2. Maximization: θ t+1 argmax θ Q θ; θ t L θ; X = E q log p X θ [ ] ] p X, Z θ q Z = E q log + E q [log q Z p Z X, θ }{{}}{{} VLBq,θ D KL qz pz X,θ EM - coordinate ascent 1. q t+1 = argmax q VLB q, θ t 2. θ t+1 = argmax θ VLB q t+1, θ Show the equivalence? 12 / 19

13 Bayes Inference Notations θ : hyper parameters Z : hidden variables + random parameters Goals 1. find a good posterior q Z p Z X; θ 2. estimate θ by Empirical Bayes, i.e., maximize L θ; X w.r.t θ [ ] ] p X, Z θ q Z L θ; X = E q log + E q [log q Z p Z X, θ }{{}}{{} VLBq,θ D KL qz pz X,θ both goals can be achieved via the same procedure as EM 13 / 19

14 Variational Bayes Inference One should have q p Z; X, θ by alternating between 1. q t+1 = argmax q VLB q, θ t 2. θ t+1 = argmax θ VLB q t+1, θ However, we do not want q to be too complicated e.g., Q θ; θ t = E q L θ; X, Z can be intractable Solution: modify the first step as q t+1 = argmax q Q VLB q, θ t Q - some tractable distribution families Recall: without Q, q t+1 p Z X, θ t 14 / 19

15 Variational Bayes Inference Goal: solve argmax q Q VLB q, θ t usually, Q = {q q Z = M i=1 q i Z i = } M i=1 q i Coordinate ascent VLB q j ; q j, θ t = E q log p X, Z; θ t q Z = E q log p X, Z; θ t = E qj E q j log p log qj = E q j log p X, Z; θ t = D KL log q j E q j log p M E q log q i i=1 X, Z; θ t E qj log q j + C X, Z; θ t + C 15 / 19

16 / 19 Example: Bayes Mixture of Gaussians Consider putting a prior over the means in GM 2 For k = 1, 2... K, µ k N 0, τ 2 For i = 1, 2... N 1. z i Mult π 2. x i N µ zi, σ 2 p z, µ X = = p X z, µ p z p µ p X Ni=1 p z i p x i z i, µ K k=1 p µ k Ni=1 z p z i p x i z i, µ K k=1 p µ k dµ N K q z, µ = q z i ; φ i q i=1 k=1 µ k ; µ k, σ k 2

17 Example: Bayes Mixture of Gaussians log q z j = E q \z j log p z, µ, X N K = E q \z j log p z i + log p x i z i, µ + log p µ k i=1 k=1 = log p z j + E qµzj log p x j z j, µ zj + C [ ] 1 ] = log π zj + x j E qµzj µzj }{{} 2 E qµ zj [µ 2 z j + C }{{} µ zj µ 2 z j + σ z 2 j By observation q z j Mult, we can update φ j accordingly 17 / 19

18 Example: Bayes Mixture of Gaussians log q µ j = E q \µ j log p z, µ, X N K = E q \µ j log p z i + log p x i z i, µ zi + log p µ k i=1 k=1 N K = E q \µ j δ zi =k log N x i µ k + log p µ j + C i=1 k=1 N = E zi [δ zi =j] log N x i µ j + log p µ j + C }{{} i=1 φ j i Observing that q µ j N, µ j and σ 2 j can be updated accordingly 18 / 19

19 Stay tuned Next topics LDA Wanli Bayes vmf 19 / 19