Tutorial on Maximum Likelyhood Estimation: Parametric Density Estimation

Transcription

1 Tutorial on Maximum Likelyhoo Estimation: Parametric Density Estimation Suhir B Kylasa 03/13/ Motivation Suppose one wishes to etermine just how biase an unfair coin is. Call the probability of tossing a HEAD is p. The goal then is to etermine p. Also suppose the coin is tosse 80 times: i.e., the sample might be something likex 1 H, x 2 T,, x 8 0 T, an the count of number of HEADS, H is observe. The probability of tossing TAILS is 1 p. Suppose the outcome is 49 HEADS an 31 TAILS, an suppose the coin was taken from a box containing three coins: one which gives HEADS with probability p 1/3, one which gives HEADS with probability p 1/2 an another which gives HEADS with probability p 2/3. The coins have lost their labels, so which one it was is unknown. Clearly the probability mass function for this experiment is binomial istribution with sample size equal to 80, number of successes equal to 49 but ifferent values of p. We have the following probability mass functions for each of the above mentione cases: P r(h 49 p 1/3) P r(h 49 p 1/2) P r(h 49 p 2/3) ( ) 80 (1/3) 49 (1 1/3) , (1) 49 ( ) 80 (1/2) 49 (1 1/2) , (2) 49 ( ) 80 (2/3) 49 (1 2/3) (3) 49 Base on the above equations, we can conclue that the coin with p 2/3 was more likely to be picke up for the observations which we were given to begin with.

2 2 Definition The generic situation is that we observe a n-imensional ranom vector X with probability ensity (or mass) function f(x; θ). It is assume that θ is a fixe, unknown constant belonging to the set Θ R n For x R n, the likelihoo function of θ is efine as L(θ/x) f(x/θ). (4) x is regare as fixe, an θ is regare as the variable for L. The log-likelihoo function is efine as l(θ/x) logl(θ/x). The maximum likelihoo estimate (or MLE) is the value ˆθ θ(x) ˆ Θ maximizing L(θ/x), provie it exists: 3 What is Likelihoo function? L(ˆθ/(x)) arg max L(θ/x) (5) θ If the probability of an event X epenent on moel parameters p is written as then we talk about the likelihoo P (X p) L(p X) that is the likelihoo of the parameters given the ata. For most sensible moels, we will fin that certain ata are more probable than other ata. The aim of maximum likelihoo estimation is to fin the parameter value(s) that makes the observe ata most likely. This is because the likelihoo of the parameters given the ata is efine to be equal to the probability of the ata given the parameters If we were in the business of making preictions base on a set of soli assumptions, then we woul be intereste in probabilities - the probability of certain outcomes occurring or not occurring. However, in the case of ata analysis, we have alreay observe all the ata: once they have been observe they are fixe, there is no probabilistic part to them anymore (the wor ata comes from the Latin wor meaning given ). We are much more intereste in the likelihoo of the moel parameters that unerly the fixe ata.

3 The following is the relation between the likelihoo an the probability spaces: Probability Likelihoo Knowing paramtres preiction of outcomes Observation of ata estimation of parameters 4 Metho Maximum likelihoo (ML) estimates nee not exist nor be unique. In this section, we show how to compute ML estimates when they exist an are unique. For computational convenience, the ML estimate is obtaine by maximizing the log-likelihoo function, log L(θ/x). This is because the two functions log L(θ/x) an L(θ/x) are monotonically relate to each other so the same ML estimate is obtaine by maximizing either one. Assume that the log-likelihoo function is ifferentiable, if θ MLE exists, it must satisfy the following partial ifferential equation known as the likelihoo equation: (logl(θ/x)) 0 (6) θ at θ θ MLE. This is because maximum or minimum of a continuously ifferentiable function implies that its first erivatives vanishes at such points. The likelihoo equation represents a necessary conition for the existence of an MLE estimate. An aitional conition must also be satisfie to ensure that log L(θ/x) is a maximum an not minimum, since the first erivative cannot reveal this. To be a maximum, the shape of the log-likelihoo function shoul be convex in the neighborhoo of θ MLE. This can be checke by calculating the secon erivatives of the log-likelihoos an showing whether they are all negative at θ θ MLE. 5 Properties 2 (logl(θ/x)) < 0 (7) θ2 Some general properties (avantages an isavantages) of the Maximum Likelihoo Estimate are as follows:

4 1. For large ata samples (large ) the likelihoo function L approaches a Gaussian istribution 2. Maximum Likelihoo estimates are usually consistent. For large the estimates converge to the true value of the parameters which are estimate. 3. Maximum Likelihoo Estimates are usually unbiase. For all sample sizes the parameter of interest is calculate correctly. 4. Maximum Likelihoo Estimate is efficient: (the estimates have the smallest variance). 5. Maximum Likelihoo Estimate is sufficient: (it uses all the information in the observations). 6. The solution from the Maximum Likelihoo Estimate is unique. On the other han, we must know the correct probability istribution for the problem at han. 6 umerical examples using Maximum Likelihoo Estimation In the following section, we iscuss the applications of MLE proceure in estimating unknown parameters of various common ensity istributions. 6.1 Estimating prior probability using MLE Consier a two-class classification problem, with classes (ω 1, ω 2 ) an let Prob(ω 1 ) p an Prob(ω 2 ) 1 p (here p is the unknown parameter). By using MLE, we can estimate p as follows: Let the sample be D (x 1, x 2,..., x ). Let ω ij be the class of the feature vector x j ( is the sample size an 1 is the number of feature vectors belonging to class ω 1 ). Also assume that samples x 1, x 2..., x are inepenent events. Then we have the following equations. P rob(d/p) By Inepenence of the feature vectors P rob(ω ij /p) p 1 (1 p) 1

5 Please note that P rob(d/p) is infinitely ifferentiable function of p, so the local maxima lies where its erivative is zero. ( p 1 (1 p) ) 1 p 0 1 p 1 1 (1 p) 1 ( 1 ) (1 p) 1 1 p 0 Solving the above equation for p, we get the following: p 1 (1 p) 1 1 p 1 (1 p) (8) 1 (1 p) ( 1 ) p (9) So p is either 0 or 1 by eq. 8 an p is ( 1 /) by eq. 9. Hence this proves that taking the frequencies for probabilities of the feature vectors is optimum an using MLE we showe that p is maximize. Hence the class probabilities are optimum (the likelihoo function is maximize using MLE). 6.2 Estimating µ of a Gaussian istribution when Σ is known In this section, we estimate the value of µ (µ MLE ), when the covariance matrix (Σ) in known, for gaussian istribution in n-imensional feature space. ρ(d/µ) log likelihoo function is logρ(d/µ) ρ(x j /µ) log(ρ(x j /µ)) [ ] 1 log exp (x j µ) T Σ 1 (xj µ) (2π) n 2 Σ [ ] 1 log 1 [ (xj µ) T Σ 1 (x (2π) n 2 Σ 1 j µ) ] (10) 2 2 This function is infinitely ifferentiable function of unknown parameters (µ) s. To fin the maxima, we set the erivative of this eq. 10 to 0.

6 µ 1 µ 2... µ n 1 Differentiating the log likelihoo functions yiels the following results: n 1 (11) But µ log(d/ µ) 1 2 µ i (x j µ) T Σ 1 (x j µ) [ µ (xj µ) T Σ 1 (x j µ) ] µ 1 (x j µ) T Σ 1 (x j µ) µ 2 (x j µ) T Σ 1 (x j µ)... µ (x j µ) T Σ 1 (x j µ) n 1 [ ] [ µ (x j µ) T Σ 1 (x j µ) + (x j µ) T Σ 1 [ ] µ (x j µ) T Σ 1 (x j µ) 2e T i Σ 1 (x j µ) ] µ (x j µ) where e T i [0,0,0..., 1,... ] an 1 is locate at position i in the array 2e µ log(d/ µ) 1 T 1 Σ 1 (x j µ) 2e T 2 Σ 1 (x j µ) e T Σ 1 (x j µ) e T 1 e T 2... Σ 1 (x j µ) e T n 1 n 1 otice that e T 1 e T 2... e T n 1 is the ientity matrix.

7 So the above equation reuces to. µ log(d/ µ) Σ 1 (x j µ) By solving the equation. 12 for µ. Σ 1 (x j µ) (12) Σ 1 Σ 1 (x j µ) 0 (x j µ) 0 (x j µ) 0 (x j ) (µ) 0 µ 1 ( (x j )) µ MLE (13) Hence, we prove that using MLE the sample mean is the maximum likelihoo estimate of any given sample. 6.3 Estimating µ an σ 2 for 1-D gaussian istribution using MLE In this subsection, we estimate the µ an σ 2 for one-imensional gaussian ata. Here θ (θ 1, θ 2 ) are (µ, σ 2 ), which are unknown parameters. We estimate these parameter using the proceure iscusse in the section 4.

8 The log-likelihoo function for this case is given by the following equation: [ ] logρ(x k /µ, σ 2 1 ) log exp( x k µ ) 2πσ 2 2σ log(2πσ2 ) 1 2σ (x 2 k µ) 2 logρ(d/µ, σ 2 ) logρ(x k /µ, σ 2 ) (14) k1 Since (x 1, x 2,..., x ) are I.I.D s, the ensity function can be written in prouct form as follows: ρ(d/µ, σ 2 ) ρ((x 1, x 2,..., x )/µ, σ 2 ) ρ(x 1 /µ, σ 2 )ρ(x 2 /µ, σ 2 )... ρ(x /µ, σ 2 ) logρ(d/µ, σ 2 ) log ρ(x k /µ, σ 2 ) k1 k1 [ 1 2 log(1πσ2 ) 1 ] 2σ (x 2 k µ) 2 (15) Differentiating an setting the eq. 15 to 0, yiels the following equations: µ,σ 2ρ(D/µ, σ 2 ) [ logρ(d/µ, ] µ σ2 ) logρ(d/µ, σ 2 ) σ [ ( ] µ 2 log(2πσ2 ) 1 2σ 2 k1 (x k µ)) ( σ 2 2 log(2πσ2 ) 1 2σ 2 k1 (x k µ)) [ 1 σ 2 k1 (x ] k µ) + 1 2σ 2 2σ 4 k1 (x k µ)

9 Solving the eq. 16 for µ an σ 2 yiels the following: 2σ σ 4 ( ) 1 (x σ 2 k µ) 0 k1 (x k µ) 0 k1 ˆµ µ 1 (x k ) (16) k1 (x k µ) 2 0 k1 (x k µ) 2 σ 2 k1 ˆσ 2 σ 2 k1 (x k ˆµ) 2 which are the mean an stanar eviation of the empirical ata. In general for -imensional Gaussian ata, when X (µ, Σ) where X R n, with µ an Σ are unknown parameters, then MLE for µ an Σ are as follows: ˆσ 2 ˆµ 1 ( x k ) k1 k1 ( x k µ)( x k µ) T 6.4 How far oes the ˆµ eviate from the true µ when MLE is use. As iscusse in section 6.3, we know that E[ˆµ] 1 µ ( x k ) k1 ow we compute the expecte value of ( ˆµ µ ) 2 as follows:

10 Substituting E[ˆµ] µ we have E[( ˆµ µ ) 2 ] E[(ˆµ µ)(ˆµ µ)] E[ˆµµ µˆµ ˆµµ + µµ] E[ˆµˆµ] 2E[µˆµ] + E[µµ] E[ˆµˆµ] µµ E( 1 X k 1 2 k1 X j ) µµ E[X j X k ] µµ j,k1 Treating X j as ranom variables in the above equations, we have But, we know that [ E[( ˆµ µ ) 2 ] 1 E(X 2 j )E(X k ) + j,k1 ] E(X j X k ) µµ j,k1 1 2 [ ( 1)µµ + E[X 2 ] ] µµ E[( ˆµ µ ) 2 ] 1 E[X2 ] 1 µµ E[ X µ 2 ] E[(X µ)(x µ)] σ 2 E[X X] µ 2 σ 2 E[X X] σ 2 µ 2 Therefore, we have the following result: E[( ˆµ µ ) 2 ] 1 (σ2 µ 2 ) 1 (µµ) 1 σ2 So the expecte value of ( ˆµ µ ) 2 is proportional to the true stanar eviation.

11 6.5 Estimate for Σ is biase, when MLE is use. In the following section, we will show that the estimate for covariance is biase (when µ is unknown) when MLE is use to estimate its value an equals to true covariance when µ is known. E[ˆΣ] E[ 1 (X k µ)(x k µ) T ] k1 E[ 1 (X k µ)(x k µ) T ] k1 1 E[X k Xk T X K µ T µx T + µµ T ] k1 1 E[X KX T K] µµ T E[XX T µµ T ] E[XX T 2µµ T µµ T ] E[(X µ)(x µ) T ] Σ If the µ is known, then it turns out that Estimate value of ˆΣ is equal to true Σ. We now show the erivation that in case where µ is not known, then estimate value of ˆΣ is not equal to true Σ E[ˆΣ] 1 E[ (X k ˆµ)(X k ˆµ) T ] k1 E[X k Xk T X K ˆµ T ˆµX T + ˆµˆµ T ] k1 ( E[Xk Xk T ] E[X K ˆµ T ] E[ˆµX T ] E[ˆµˆµ ]) k1 ( E[X k Xk T ] E[( 1 k1 E[ 1 X k Xl T ] + E[ 1 l1 ) X l )X k ] l1 l1 X l 1 Xm] T m1

12 Splitting the summation above into two parts, l k an l k, we have the following: E[ˆΣ] 1 {E[XX T ] 1 Σ l1,l ke(x l )E(Xk T ) 1 Σ l1,lke(x k Xk T ) k1 1 Σ l1,l ke(x k )E(X T l ) 1 Σ l1,lke(x k X T k ) ΣK l,m1,l me(x l )E(Xm) T ΣK l,m1,lme(x l )E(Xl T )} 1 {E[XX T ] 1 Σ l1,l kµµ T 1 Σ l1,lke(x k Xk T ) k1 1 Σ l1,l kµµ T 1 Σ l1,lke(x k X T k ) ΣK l,m1,l mµµ T ΣK l,m1,lme(x l )E(X T l )} E[ˆΣ] Binomial Distribution [(1 1 E(XXT ) + ( 1 1)µµT ] (1 1 ) [ E(XX T ) µµ T ] ( 1 )E(XXT µµ T )) ( 1 )E[(X µ)(x µ)t ] Σ This section iscusses the estimation of p in a typical binomial istribution. Assuming that parameter p, is unknown in a binomial istribution give by the equation below:

13 ( ) n P rob(r/p) p r (i p) n r r ( ) n logp rob(r/p) log rlog(p) + (n r)log(1 p) r logp rob(d/p) log P rob(r i /p) i1 log[p rob(r i /p)] i1 ( ) n [log + r i log? + (n r i )log(1 p)] i1 To fin the maximum, we set the erivative of log-likelihoo function to be 0: r i i1 p log(d/p) 0 [ r i p + n r i 1 p ( 1)] 0 1 p (1 p) r i i1 r i p i1 p p (n r i ) i1 (n r i ) i1 r i /n Hence p, which is the unknown parameter is equal to the average of number of successes in the sample space. i1