Exponential Family and Maximum Likelihood, Gaussian Mixture Models and the EM Algorithm. by Korbinian Schwinger

Transcription

1 Exponential Family and Maximum Likelihood, Gaussian Mixture Models and the EM Algorithm by Korbinian Schwinger

2 Overview Exponential Family Maximum Likelihood The EM Algorithm Gaussian Mixture Models

3 Exponential Family A density function of a random variable x is member of the exponential family if the distribution function can be written as : n f x =h x exp b i=1 w i t i x

4 Gauss in 1 dimension Density function with the Parameters μ=0 and σ²=1 f x x = 1 x 2 2 e 2

5 Gauss in 1 dimension Distribution function with the Parameters μ=0 and σ²=1 x F x x = x e 2 dx

6 Gauss in 2 dimensions μ becomes a vector, the variance becomes the covariance-matrix, which shows the relationship between the 2 Coordinates : = 1 2 = ij =E [ x i i x j j ] 22

7 Gauss in 2 dimensions Density with the Parameters : = 10 1 =

8 Gauss in n-dimensions The parameters become : = 1 2 n n = n 1 n 2 n nn ij =E [ x i i x j j ]

9 Maximum Likelihood We have a density function p(x Θ) which depends on parameters in Θ We want to find out, which parameters in Θ are the most probable First, we need a set of data X of size N With this set we can create a new density function N p X = i=1 p x i =L X

10 Maximum Likelihood Best set of parameters maximizes the likelihood function max =argmax L X Often log(l(θ X)) is maximized If distribution is Gaussian the maximum can be found with the derivative

11 Maximum Likelihood - example We have got a random variable with a gaussian distribution with the Parameters μ and σ² The likelihood function is L x ;, 2 = 1 n 2 n i=1 exp x i 2 2 2

12 Maximum Likelihood - example Now we have to calculate the logaritm of the likelihood function n ln L x ;, 2 = n ln 2 ln i=1 x i 2 2 2

13 Maximum Likelihood - example Next step is to create the derivative of the loglikelihood ln L n = i=1 x i 2 =! 0 ln L = n n i=1 x i 2 3 =! 0

14 Maximum Likelihood - example At the end we have got : = x n 2 = 1 n i=1 x i 2

15 Basic EM Used to find the maximum-likelihood estimate when features are missing Limitations of the observation process calculation can be simplified First, we assume the observed data X are generated by some distribution Second, we assume, that a complete set of data Z=X+Y exists

16 Basic EM Then we can create a density function : p z = p x, y = p y x, p x With this density function, the complete-data likelihood, p(x,y Θ), can be formed Then the Algorithm calculates the expected value of the complete-data log-likelihood Q, i 1 =E [log p X,Y X, i 1 ] = y Y log p X, y f y X, i 1 dy

17 Basic EM The evaluation of the expectation is called the E- step The M-step is to maximize the expectation, we computed in the first step : i =argmax Q, i 1 These steps are repeated as necessary each iteration is guaranteed to increase the likelihood

18 Basic EM - example We have got 4 Points in 2D Coordinates : { 0 2, 1 0, 2 2,? 4 } We assume, the model is a Gaussian : 1 =

19 Basic EM - example The first estimation, we get with the assumption of an Gaussian model : =

20 Basic EM - example 3 = k=1 Now we calculate the expected value : = 3 [ k=1 Q, 0 =E [ln p x g, x b ; 0 ; D g ] ln p x k ln p x 4 ] p x 41 0 ; x 42 =4 dx 41 [ln p x k ] ln p x 41 4 p x p x ' dx 41 ' =K dx 41

21 Basic EM - example K is constant and can be brought out of the integral : 3 Q ; 0 = k=1 [ln p x k ] 1 K ln p x exp [ x ] dx 41

22 Basic EM - example At the end we get (E step completed) [ln p x k ] Q ; 0 = k= Then, we can calculate the parameters, which maximize Q(..) 1 = ln 2 1 2

23 Basic EM - example

24 Gaussian Mixture Models We assume the following probilistic model: The parameters M are and i=1 i =1 p x = With a lot of calculations we get the function M i=1 i p i x i = 1,..., M, 1,..., M M N Q, g = l=1 i=1 M N log l p l x i, g l=1 i=1 log p l x i l p l x i, g

25 Gaussian Mixture Models After some more pages of calculations you get a new estimation of the parameters : N l new = 1 N i=1 N new i=1 l = p l x i, g new i=1 l = These values can be used to repeat the iteration as often as needed N N i=1 x i p l x i, g p l x i, g p l x i, g x i l new x i l new T N i=1 p l x i, g

26 Gaussian Mixture Models - example Java Demo Applet