Expectation-Maximization for Estimating Parameters for a Mixture of Poissons

Transcription

1 Expectation-Maximization for Estimating Parameters for a Mixture of Poissons Brandon Maone Department of Computer Science University of Hesini February 18, 2014 Abstract This document derives, in excrutiating detai at some points, the EM update rues for a simpe mixture of Poisson distributions. This document is argey based on [1] and [2]. 1 Introduction This wor assumes a dataset is distributed according to a mixture of Poisson distributions. The goa is to earn the mixing distributions and Poisson parameters. Broady, this can be considered the density estimation probem. 2 Density Estimation The density estimation probem is a common probem in machine earning. Broady, the probem is formuated as: given a set of N observations, D, in some space, and a famiy Θ of probabiity density functions, find θ Θ which most iey generated the data. In this wor, we wi assume that Θ is a famiy of mixture modes, so the ieihood of its members are defined as: L(D : θ) p g (D : λ ), (1) where θ {p 1... p, λ 1... λ } is a set of parameters, K is the number of mixture components, p are the mixing probabiities (P (z )), g is the density function of the th component, and λ are the set of parameters for g. The p s define a probabiity distribution, so we aso have that 0 p 1 and p 1. If we further assume that each component has the same density function (but different parameters) and that the observations are exchangeabe, then we can 1

2 write: L(D : θ) N p g(d : λ ). (2) Typicay, we wi wor in og-space because it is more amenabe to anaysis and computation. Consequenty, we define the og-ieihood of θ as: (D : θ) og N p g(d ; λ ) og p g(d : λ ). (3) The density function of the Poisson distribution is: g(x : λ) λx exp { λ} (4) x! With definitions, we can define our version of the density estimation probem as: The goa is to find θ. θ arg max (D : θ) (5) θ Θ 3 Notation In order to express ideas succincty, we use the foowing notation. q(, ) p g(d : λ ) (6) This is the joint probabiity of seecting component (p ) and seecting an observation from that component (g(d : λ )). This maes expicit the assumption that seecting a component is independent of seecting an observation from a component. We now define the conditiona probabiity of seecting component, given the observation D. First, reca one definition of conditiona probabiity: P (A B) P (A, B) P (B) (7) Additionay, reca that we can find prior probabiities for B by taing the joint probabiity of a variabes, and then summing (for discrete variabes) or integrating (for continuous variabes) the others out. Supposing A is the ony other variabe and that it is discrete, then: P (B) A P (A, B) (8)

3 With these basic ideas of probabiity in mind, then, we can write the conditiona probabiity of seecting component given the observation D as: p( ) q(, ) K m q(m, ) (9) We ca these the membership probabiities because they give the probabiity that D is a member of component. One other usefu observation is that p( ) 1. That is to say, some component generated each observation. 4 Jensen s Inequaity A potentia difficuty in evauating the og-ieihood (Equation 3) is that it contains the ogarithm of a sum. Jensen s inequaity shows that: og π α π og α, (10) when the π s define a probabiity distribution (i.e., 0 π 1 and π 1). With this inequaity, we can define a probabiity distribution π and rearrange terms to see that for any vaues of c : og c og c π π og π c π π og c π. (11) 5 The EM Agorithm With these toos, we can now derive the expection-maximization agorithm for finding the parameters of the Poisson mixture mode. The EM agorithm begins with a (bad) set of estimates for the p s and λ s and then aternates between two steps. In the first E step, we use our parameter estimates to construct a bound b on using Jensen s inequaity. Then, in the M step, we find parameter estimates which maximize the bound. Concretey, during the E step, we compute the expectation that each observation came from each component. That is, we use the existing estimates to cacuate the new membership probabiities: p( ) p g(d : λ ) K m p g(d : λ ) (12) As mentioned in Section 3, and expicit because of the normaization in Equation 12, the p( )s sum to 1. Therefore, we can use Jensen s inequaity to bound the ieihood: (D : θ) og q(, ) q(, ) p( ) og b(θ) (13) p( )

4 by pugging in p( ) for π and q(, ) for c. We can then expand the ogarithm, distribute p( ), and spit the sum to see that: b(θ) p( ) og q(, ) p( ) og p( ). (14) The p( ) vaues are fixed, so we can maximize the bound by focusing ony on the first term of the summation. We ca this function Q: Q(θ) p( ) og q(, ) (15) We cacuate the new parameters by differentiating Q with respect to each parameter, setting the derivative equa to 0, and soving for the parameters. We now derive these resuts in quite a bit of detai. 5.1 Updating the Poisson parameters λ λ p(m ) og q(m, ) (16) m The derivative of a sum is the sum of derivatives, and p( ) is a constant here. Aso, the derivative of every term which does not invove is 0, so we can ignore those. λ Repace q with its definition, Equation 6. p( ) λ og q(, ) (17) λ p( ) λ og p g(d : λ ) (18) Repace g with its definition (the Poisson density function). λ p( ) λ D og p λ D! exp { λ } (19) Expand the ogarithm. λ p( ) λ og p + og λ D og D! + og exp { λ } (20) Simpify the ogarithm of exponentia, and the exponentiation in the ogarithm. λ p( ) λ og p + D og λ og D! λ (21)

5 Evauate the derivative. λ p( )( D λ 1) (22) Set the derivative to 0. We now simpy have some arithmetic to do. 0 Distribute p( ) and expand the sum. p( )( D λ 1) (23) 0 n p( )( D λ ) p( ) (24) For convenience, we wi define Z(). N Z() λ is constant in the sum, so pu it out. Z() 1 λ Mutipy both sides by λ, and divide both sides by Z(). p( ). Add it to both sides. p( )( D λ ) (25) N p( )D (26) λ N Thus, we have our new estimate of λ. p( )D Z() (27) 5.2 Updating the mixing probabiities There derivation for updating the p s is simiar to that of λ, but one ey difference is the constraint that the p s are a probabiity distribution. To address this constraint, we use a Lagrange mutipier δ to constrain the p vaues. p p p(m ) og q(m, ) + δ( m p 1) (28) The derivative of sums is the sum of derivatives. The derivative of every term which does not invove is 0. p Evauate the derivative. p( ) og q(, ) + δ(p 1) (29) p p p p( ) p + δ (30)

6 Set the derivative equa to 0 and sove for p. 0 p( ) p + δ (31) First, we need to find the vaue of δ. By summing over, we see that δ is N. N p( ) p (32) Since 1 p is constant, we can pu it out of the sum and move it to the other size. Then, we divide both sizes by N. p N p( ) N (33) Reca that we defined Z(). N p( ), so rewrite the equation using Z(). Thus, we have our new estimate of p. p Z() N (34) References [1] J. A. Bimes. A gente tutoria of the em agorithm and its appication to parameter estimation for gaussian mixture and hidden marov modes. Technica Report TR , Internationa Computer Science Institute, [2] C. Tomasi. Estimating Gaussian mixture densities with EM - a tutoria.