IEOR 165 Lecture 13 Maximum Likelihood Estimation

Transcription

1 IEOR 165 Lecture 13 Maximum Likelihood Estimation 1 Motivating Problem Suppose we are working for a grocery store, and we have decided to model service time of an individual using the express lane (for 10 items or less) with an exponential distribution. An exponential service time is a common assumption in basic queuing theory models. Recall that: A random variable X with exponential distribution is denoted by X E(λ), where λ > 0 is the rate; in our context, the rate is better understood to be the service rate. The exponential distribution has pdf { λexp( λu), if u 0, f X (u) = 0 otherwise. In our motivating problem, suppose the data X i for i = 1,...,n are assumed to be iid measurements of the service time. The first idea we might have to estimate the service rate is to observe that the mean of an exponential distribution is 1/λ. And so we could estimate the service rate using the equation ( 1 ˆλ = n n ) 1. X i i=1 The intuition is that the sample average converges to the expectation as we get more data. And since the service rate is assumed to be λ > 0 in the definition of an exponential random variable, it turns out that the inverse of the sample average will converge to the service rate λ. The potential weakness of this approach is that though it will eventually give us the correct answer, we are not using information we know about the underlying distribution of the data; it may be possible to get an accurate estimate of the service rate using less data with a more sophisticated approach. Recall the likelihood function for a parametric distribution P θ with pdf f θ (u) is defined as L(X,θ) = n f θ (X i ). One alternative idea is to choose the value of θ that maximizes this likelihood. And so in order to be able to solve these problems, we have to first discuss optimization problems. 1 i=1

2 2 Nonlinear Programming Consider the following optimization problem (P): min f(x) s.t. x R n g i (x) 0, i = 1,...,m h i (x) = 0, i = 1,...,k where f(x),g i (x),h i (x) are continuously differentiable functions. We call the function f(x) the objective, and we define the feasible set X as X = {x R n : g i (x) 0 i = 1,...,m h i (x) = 0 i = 1,...,k}. Note that this formulation also incorporates maximization problems such as max{f(x) x X} through rewriting the problem as min{ f(x) x X}. Let B(x,r) = {y R n : x y r} be a ball centered at point x with radius r. A point x is a local minimizer for (P) if there exists ρ > 0 such that f(x ) f(x) for all x X B(x,ρ). A point x is a global minimizer for (P) if f(x ) f(x) for all x X. Note that a global minimizer is also a local minimizer, but the opposite may not be true. 2.1 Unconstrained Optimization First consider the special case where f : R R. When (P) does not have any constraints, we know from calculus (specifically Fermat s theorem) that the global minimum must occur at points where either (i) the slope is zero f (x) = 0, (ii) at x =, or (iii) at x =. More rigorously, the theorem states that if f (x) 0 for x R, then this x is not a local minimum. This result extends naturally to the multivariate case where f : R n R, by insteading looking for points where the gradient is zero f(x) = 0. This result is useful because it gives one possible approach to solve (P) in the case where there are no constraints: We can find all points where the gradient is zero, evaluate the function in the limits as x tends to x =, or x =, and then select the minimum amongst these points. One natural question to ask is how can we extend this approach to the general case (P)? 2.2 Karush-Kuhn-Tucker (KKT) Conditions If x X is a local minimizer for (P), then under some technical conditions there exists λ i for i = 1,...,m and µ i for i = 1,...,k such that the following equations, known as the 2

3 Karush-Kuhn-Tucker (KKT) conditions, hold: f(x )+ m i=1 λ i g i (x )+ k i=1 µ i h i (x ) = 0 λ i 0, i = 0,...,m λ i g i (x ) = 0, i = 1,...,m As in the unconstrained case, this is a necessary (but not sufficient) condition for optimality. 2.3 Example Consider the following optimization problem min x s.t. x R x 0. The KKT conditions are that the local minima x satisfy 1 λ 1 = 0 λ 1 0 λ 1 x = 0 Also,allpossiblecombinationsof(λ 1,x)forwhichtheKKTconditionsholdcanbecomputed. From the condition: 1 λ 1 = 0, we have that λ 1 = 1. Combining this with the condition: λ 1 x = 0 gives that x = 0. This is the only possible solution of the KKT conditions, and so the local minimizer must also be the global minimizer. Summarizing, the global minimizer is x = 0. 3 Maximum Likelihood Estimation The idea of maximum likelihood estimation (MLE) is to generate an estimate of some unknown parameters by solving the following optimization problem max{l(x,θ) θ Θ} = max{ n i=1 f θ(x i ) θ Θ}. Here, the constraint θ Θ is meant to be short-hand notation for the inequality and equality constraints discussed earlier. 3.1 Example: Service Rate of Exponential Distribution To get a better understanding of how this approach works, consider the case of using MLE to estimate the service rate of an exponential. The likelihood is L(X,λ) = n ( i=1 λexp( λxi ) ) = λ n exp( λ i=1 X i). 3

4 And we would like to solve the problem max{λ n exp( λ i=1 X i) λ > 0}. This formulation does not exactly fit the framework described above, and the reason is that strict inequalities are somewhat problematic for optimization problems. The reason is that a minimum may not exist. And so we will consider the following optimization max{λ n exp( λ i=1 X i) λ 0}, and we will hope that λ = 0 is not the solution because this would indicate a problem. In principle, we can solve this problem by finding all points satisfying the KKT conditions. However, sometimes ad hoc approaches can be easier. For instance, suppose we find the points at which the gradient is zero: nλ n 1 exp( λ i=1 X i)+ λ n exp( λ i=1 X i) n i=1 X i = 0. Now assuming λ > 0, we can simplify this to n+ λ n i=1 X i = 0 λ = ( 1 n i=1 X i) 1. Since the random variables satisfy X i > 0 when drawn from an exponential distribution, this estimate is strictly positive. The objective at λ = 0 is zero, and the objective as λ 0 is also zero. Consequently, the above solution must be the global maximum since the function is strictly concave (i.e., concave down) in λ. Observe that in this case the MLE estimate is equivalent to the inverse of the sample average. However, in other cases the MLE estimate can differ from the estimates we previously considered. 3.2 Transformation of the Objective Function In some cases, it can be easier to solve a transformed version of the optimization problem. For instance, two transformations that are useful in practice are: 1. instead of maximizing the likelihood, we could equivalently minimize the negative of the likelihood; 2. instead of maximizing the likelihood, we could equivalently maximize the likelihood composed with a strictly increasing function. In fact, one very useful transformation is the minimize the negative log-likelihood. The utility of such transformations is most clearly elucidated by an example. 4

5 3.3 Example: Mean and Variance of Gaussian Suppose X i N(µ,σ 2 ) are iid measurements from a Gaussian with unknown mean and variance. Then the MLE estimate is given by the solution to max{(2πσ 2 ) n/2 exp ( i=1 (X i µ) 2 /(2σ 2 ) ) σ 2 > 0}. If we instead minimize the negative log-likelihood, then the problem we would like to solve is min{(n/2)log(2πσ 2 )+ i=1 (X i µ) 2 /(2σ 2 ) σ 2 > 0}. If we consider the objective function h(µ,σ 2 ) = (n/2)log(2πσ 2 )+ i=1 (X i µ) 2 /(2σ 2 ) (that is we are treating the σ 2 as a single variable), then setting its gradient equal to zero gives [ ] [ hµ = i=1 2(X ] i µ)/(2σ 2 ) h σ 2 n/(2σ 2 ) i=1 (X i µ) 2 /(2(σ 2 ) 2 = 0, ) where h µ and h σ 2 denote the partial derivatives of h(µ,σ 2 ) with respect to µ and σ 2, respectively. To solve this, we first begin with h µ = i=1 2(X i µ)/(2σ 2 ) = 0 i=1 (X i µ) = 0 ˆµ = 1 n i=1 X i, where we have made use of the fact that σ 2 > 0. Next, note that h σ 2 = n/(2σ 2 ) i=1 (X i ˆµ) 2 /(2(σ 2 ) 2 ) = 0 nσ 2 = i=1 (X i ˆµ) 2 ˆσ 2 = 1 n i=1 (X i ˆµ) 2, where we multiplied by (σ 2 ) 2 on the second step. The estimate of the mean ˆµ = 1 n n i=1 X i is simply the sample average; however, the estimate of the variance ˆσ 2 = 1 n n i=1 (X i ˆµ) 2 is the biased estimator of the variance (cf. the unbiased estimate used for hypothesis testing s 2 = 1 n n 1 i=1 (X i ˆµ) 2 ). So in this case, the MLE estimates differ from the estimates we have previously used for hypothesis testing. 5