Lecture Stat Maximum Likelihood Estimation

Size: px

Start display at page:

Download "Lecture Stat Maximum Likelihood Estimation"

Aleesha Pope
5 years ago
Views:

1 Lecture Stat Maximum Likelihood Estimatio A.D. Jauary 2008 A.D. () Jauary / 63

2 Maximum Likelihood Estimatio Ivariace Cosistecy E ciecy Nuisace Parameters A.D. () Jauary / 63

3 Parametric Iferece Let f (xj θ) deote the joit pdf or pmf of the sample X = (X 1,..., X ) parametrized by θ 2 Θ. The give that X = x is observed, the fuctio L ( θj x) = f (xj θ) is the likelihood fuctio. The most commo estimate is the Maximum Likelihood Estimate (MLE) give by bθ = arg max L ( θj x). θ2θ A.D. () Jauary / 63

4 Example: Gaussia distributio f (x i j θ) = The we have with θ = log L ( θj x) = = 1 p exp (x i µ) 2 2πσ 2 2σ 2 µ, σ 2 log f (x i j θ) 2 log 2πσ2 1 2σ 2 (x i µ) 2. By takig the derivatives ad settig them to zero log L ( θj x) µ log L ( θj x) σ 2 = = 1 σ 2 (x i µ) = 0,! 2σ (σ 2 ) 2 (x i µ) 2 = 0.. A.D. () Jauary / 63

5 By solvig these equatios, we obtai cσ 2 = 1 bµ = 1 x i, (x i bµ) 2. Note that bµ is a ubiased estimate but c σ 2 is biased. A.D. () Jauary / 63

6 Example: Laplace Distributio (Double Expoetial) The we have f (x i j θ) = 1 2 exp ( jx i θj). log L ( θj x) = log 2 jx i θj. By takig the derivative, we obtai d log L ( θj x) dθ = sg (x i θ) hece bθ = med fx 1,..., x g for = 2p + 1. A.D. () Jauary / 63

7 Example (Uiform Distributio): Cosider X i U (0, θ), i.e. We have L ( θj x) = f (x i j θ) = 1/θ if 0 x < θ, 0 otherwise. (1/θ) f (x i j θ) = if θ x () 0 if θ < x (). It follows that bθ = x () where x (1) < x (2) < < x (). A.D. () Jauary / 63

8 Example (Liear Regressio): Let fx i, y i g be a set of data where x i = x1 i, x 2 i,..., x p i T is a set of explaatory variables ad yi 2 R is the respose. We assume y i = x T i β + ɛ i, ɛ i N 0, σ 2 thus We have for θ = log L (θ) = f (y i j x i, β) = = = β,σ 2 1 p exp y i x T i β! 2 2πσ 2 2σ 2 log f (y i j x i, β) 2 log 2πσ2 1 2σ 2 y i 2 x T i β 2 log 2πσ2 1 2σ 2 (y Xβ)T (y Xβ) where y = (y 1,..., y ) T ad X = (x 1,..., x ) T. A.D. () Jauary / 63

9 By takig the derivatives ad settig them to zero log L ( θj x) β = log L ( θj x) σ 2 = 1 2σ 2 2X T β + 2X T Xβ = 0, 2σ (σ 2 ) 2 (y Xβ)T (y Xβ) = 0. Thus we obtai 1 bβ = X X T X T y, cσ 2 = 1 y Xβ b T y Xβ b. A.D. () Jauary / 63

10 Example (Time Series): Cosider the followig autoregressio X 0 = x 0, X = αx 1 + σv where V i.i.d. N (0, 1) where θ = α, σ 2. We have L ( θj x) = f (xj θ) = Thus we have! 1 p exp (x i αx i 1 ) 2 2πσ 2 2σ 2 log L ( θj x) = cst log σ2 2 (x i αx i 1 ) 2 2σ 2 A.D. () Jauary / 63

11 It follows that Thus we have 2 log L ( θj x) σ 2 = 2 log L ( θj x) α bα = x i x 2 1 x i i 1 σ 2 + (x i αx i 1 ) 2 σ 4, = 2 σ 2 x i 1 (x i αx i 1 ) 2., σ c2 = (x i bαx i 1 ) 2. A.D. () Jauary / 63

12 Ivariace Cosider η = g (θ). We itroduce the iduced likelihood fuctio L L ( ηj x) = sup L ( θj x). fθ:g (θ)=ηg Ivariace property: If bθ is the MLE of θ the for ay fuctio η = g (θ) the g b θ is the MLE of η. Proof : The MLE of η is de ed by De e bη = arg sup η sup fθ:g (θ)=ηg L ( θj x) g 1 (η) = fθ : g (θ) = ηg. The clearly bθ 2 g 1 (bη) ad caot be i ay other preimage so bη = g b θ. A.D. () Jauary / 63

13 Cosistecy De itio. A sequece of estimators bθ = bθ (X 1,..., X ) is cosistet for the parameter θ if, for every ε > 0 ad every θ 2 Θ lim P θ bθ θ < ε = 1 (equivaletly lim P θ bθ θ ε = 0).!! i.i.d. Example: Cosider X i N (θ, 1) ad bθ = 1 X i the bθ N (θ, 1/) ad P θ bθ θ < ε = Z ε p ε p 1 p 2π exp u 2 2 du! 1. It is possible to avoid this calculatios ad use istead Chebychev s iequality b 2 E P θ bθ θ ε = P θ bθ θ 2 θ θ θ ε 2 where E θ b θ 2 2 θ = var θ b θ + E θ b θ θ. A.D. () Jauary / 63 ε 2

14 Example of icosistet MLE (Fisher) (X i, Y i ) N µi µ i The likelihood fuctio is give by L (θ) = We obtai 1 (2πσ 2 ) exp l (θ) = cste log σ 2 " 1 xi + y i 2σ We have bµ i = x i + y i 2 σ 2 0, 0 σ 2. 1 h 2σ 2 (x i µ i ) 2 + (y i µ i ) 2i! µ i , c σ 2 = (x i y i ) 2 4 # (x i y i ) 2.! σ2 2. A.D. () Jauary / 63

15 Cosistecy of the MLE Kullback-Leibler Distace: For ay desity f, g Z f (x) D (f, g) = f (x) log dx g (x) We have Ideed Z D (f, g) = D (f, g) 0 ad D (f, f ) = 0. f (x) log Z g (x) dx f (x) g (x) f (x) f (x) 1 dx = 0 D (f, g) is a very useful distace ad appears i may di eret cotexts. A.D. () Jauary / 63

16 Alterative measures of similarity Helliger distace D (f, g) = Z q f (x) q g (x) 2 dx Geeralized iformatio D (f, g) = 1 λ Z f (x) λ 1! f (x) dx g (x) L1-orm / Total variatio Z D (f, g) = jf (x) g (x)j dx L2-orm / Total variatio Z D (f, g) = (f (x) g (x)) 2 dx A.D. () Jauary / 63

17 Example Suppose we have f (x) = N x; ξ, τ 2 ad g (x) = N x; µ, σ 2. We have h E f (X µ) 2i h = E f (X ξ) (X ξ) (ξ µ) + (ξ µ) 2i So it follows that ad = τ 2 + (ξ µ) 2 E f [log g (X )] = E f " = # 1 2 log 2πσ2 (X µ) 2 2σ log 2πσ2 τ 2 + (ξ µ) 2 2σ 2 E f [log f (X )] = 1 2 log 2πτ A.D. () Jauary / 63

18 It follows that D (f, g) = Z f (x) f (x) log dx g (x) = E f [log f (X )] E f [log g (X )] ( = 1 ) σ 2 log 2 τ 2 + τ2 + (ξ µ) 2 σ 2 1 It ca be easily checked that D (f, f ) = 0 (less easy to show D (f, g) 0). A.D. () Jauary / 63

19 Example Assume we have f (x) = 1 2 exp ( jxj) ad g (x) = N x; µ, σ2. We obtai E f [log f (X )] = log 2 = log = log 2 1, Z 0 Z jxj exp ( x exp ( E f [log g (X )] = 1 2 log 2πσ2 1 4σ 2 It follows that jxj) dx jxj) dx 4 + 2µ 2 D (f, g) = 1 2 log 2πσ σ µ 2 log 2 1. A.D. () Jauary / 63

20 Assume the pdfs f (xj θ) have commo support for all θ ad f (xj θ) 6= f xj θ 0 for θ 6= θ 0 ; i.e. S θ = fx : f (xj θ) > 0g is idepedet of θ. Deote M (θ) = 1 log f (X i j θ) f (X i j θ ) As the MLE bθ maximises L ( θj x), it also maximizes M (θ). Assume X i i.i.d. f (xj θ ). Note that by the law of large umbers M (θ) coverges to f (X j θ) E θ log f (X j θ ) = Z f (xj θ) f (xj θ ) log f (xj θ ) dx = D (f ( j θ ), f ( j θ)) := M (θ). Hece, M (θ) D (f ( j θ ), f ( j θ)) which is maximized for θ so we expect that its maximizer will coverge towards θ. A.D. () Jauary / 63

21 Example Assume f (x) = g (x) = N (x; 0, 1). We approximate E f [log g (X )] = 1 2 log (2π) 1 2 = through E b f [log g (X )] = 1 2 log (2π) 1 2 Xi 2 Numerical examples , , 000 E f [log g (X )] Mea Variace Stadard deviatio Mea, variace ad stadard deviatio by ruig 1,000 Mote Carlo trials A.D. () Jauary / 63

22 Theorem. Suppose ad that, for every ε > 0, sup jm (θ) M (θ)j! P 0 θ2θ sup M (θ) < M (θ ) θ:jθ θ jε the bθ P! θ A.D. () Jauary / 63

23 Proof. Sice bθ maximizes M (θ), we have M b θ M (θ ). Thus, M (θ ) M b θ = M (θ ) M b θ + M (θ ) M (θ ) M b θ M b θ + M (θ ) M (θ ) 2sup jm (θ) θ P! 0. M (θ)j Thus it implies that for ay δ > 0, we have Pr M b θ < M (θ ) δ! 0. Now for ay ε > 0, there exists δ > 0 such that jθ θ j ε implies M (θ) < M (θ ) δ. Hece, Pr bθ θ > ε Pr M b θ < M (θ ) δ! 0. A.D. () Jauary / 63

24 Asymptotic Normality Assumig we have bθ P! θ, what ca we say about p b θ log f ( x jθ) θ Lemma. Let s (xj θ) := be the score fuctio, the we have for ay θ E θ [s (X j θ)] = 0. Proof. We have Z log f (xj θ) f (xj θ) dx θ Z f ( x jθ) Z θ f (xj θ) = f (xj θ) f (xj θ) dx = dx θ = Z f (xj θ) dx = 0. θ {z } =1 θ? A.D. () Jauary / 63

25 Lemma. We also have h var θ [s (X j θ)] = E θ s (X j θ) 2i = 2 log f (X j θ) E θ θ 2 := I (θ) Proof. This follows from Z log f (xj θ) f (xj θ) dx = 0 θ thus by takig the derivative oce more with respect to θ 0 = Z log f (xj θ) f (xj θ) dx θ θ Z = 2 Z log f (xj θ) log f (xj θ) θ 2 f (xj θ) + θ f (xj θ) dx θ A.D. () Jauary / 63

26 Heuristic Derivatio. We have for l (θ) := log L ( θj x) 0 = l 0 bθ l 0 (θ ) + b θ θ l 00 (θ ) ) b θ θ = l 0 (θ ) l 00 (θ ) That is p b θ θ = p1 l 0 (θ ) 1 l 00 (θ ). Now remember that l 0 (θ ) = s (X i j θ ) where E θ [s (X i j θ )] = 0 ad var θ [s (X i j θ )] = I (θ ) so the CLT tells us that 1 p l 0 (θ ) D! N (0, I (θ )) A.D. () Jauary / 63

27 Now the law of large umber yields 1 l 00 (θ ) P! I (θ ) so by Slutsky s theorem p b D! 1 θ θ N 0,, p q I (θ ) b D! θ θ N (0, 1) I (θ ) Note that you have already see this expressio whe establishig the Cramer-Rao boud. It is importat to remember that depedig o θ the parameter ca be more or less easy to estimate. A.D. () Jauary / 63

28 Similarly, we ca prove that r p I b θ b D! θ θ N (0, 1). We ca also prove that p g 0 bθ r I b θ g b θ g (θ ) D! N (0, 1). This allows us to derive some co dece itervals. A.D. () Jauary / 63

29 Makig the proof more rigourous We have l 0 bθ = l 0 (θ ) + b θ θ l 00 (θ ) b θ θ 2 l 000 (θ, ) where θ, lies betwee bθ ad θ so that p b θ θ = 1 p l 0 (θ ) 2 1 l 00 1 (θ ) b 2 θ θ l 000 (θ, ). To proof the result, we eed to check that 2 1 b θ θ l 000 (θ, )! P P 0. As bθ! θ, we just eed to prove that 2 1 l 000 (θ, ) is bouded (i probability). So we eed a additioal coditio of the form say for ay θ with E θ [C (X )] <. 3 log f (xj θ) θ 3 C (x) A.D. () Jauary / 63

30 Multiparameter Case The extesio to the multiparameter case θ = (θ 1,..., θ d ) is straightforward p b θ θ D! N (0, J (θ )) where J (θ ) = I (θ ) 1 where We de e rg := p g b θ [I (θ )] k,l = 2 log f (xj θ) E θ. θ k θ l g θ 1,..., g θ d T the if rg (θ ) 6= 0 D! g (θ ) N 0, rg (θ ) J (θ ) rg (θ ) T A.D. () Jauary / 63

31 Example: If X i i.i.d. N log f (xj θ) = cst s (xj θ) = I (θ) = 1 σ log σ (x µ) σ 2 µ, σ 2 with θ = (µ, σ) the + (x µ)2 σ 3 The MLE of µ is give by (x µ) 2 2σ 2! h i 1 2(x µ) E σ 2 θ σ h i 3 2(x µ) E θ E 1 σ 3 θ + σ 2 bµ = 1, 3(x µ)2 σ 4 X i ) var [bµ] = σ2 1 A = 1 σ σ 2 ad the MLE is ideed e ciet (it reaches Cramer-Rao lower boud). A.D. () Jauary / 63

32 Assume we observe a vector X = (X 1,..., X k ) where X j 2 f0, 1g, k j=1 X j = 1 with f (xj p 1,..., p k 1 ) =! k 1 p x j j j=1 where p j > 0 ad p k := 1 k 1 j=1 p j < 1. We have θ = (p 1,..., p k 1 ). We have log f (xj θ) = x j p j p j 2 log f (xj θ) pj 2 = 2 log f (xj θ) p j p l = x j p 2 j x k pk 2 1 x k p k, x k pk 2, j 6= l < k.! xk k 1 p j j=1 A.D. () Jauary / 63

33 Recall that X j has a Beroulli distributio with mea p j so 2 p1 1 + pk 1 pk 1 pk 1 pk 1 p2 1 + pk 1. I (θ) = 6 4. pk 1... pk p k Oe ca check that 2 p 1 (1 p 1 ) p 1 p 2 p 1 p k 1 I (θ) 1 p 1 p 2 p 2 (1 p 2 ) p 2 p k 1 = p 1 p k 1 p 2 p k 1 p k 1 (1 p k 1 ) A.D. () Jauary / 63

34 Now assume we observe X 1, X 2,..., X the log L ( θj x) = k t j log p j = j=1 k 1 t j log p j + t k log 1 j=1! k 1 p j j=1 where t j = x i j. So we have log L ( θj x) p j = t j p j t k p k for j = 1,..., k 1 ) bp j = t j. Clearly t j is Biomial(, p j ) with variace p j (1 e ciet. p j ) so bp j is A.D. () Jauary / 63

35 Nuisace Parameters Assume θ = (θ 1,..., θ d ) is the parameter vector but oly the scalar θ 1 is of iterest whereas (θ 2,..., θ d ) are uisace parameters. We wat to assess how the asymptotic precisio with which we estimate θ 1 is i ueced by the presece of uisace parameters; i.e. if bθ is a e ciet estimate for θ, the how does bθ 1 as a estimator of θ 1 compare to a e ciet estimatio of θ 1, say eθ 1, which would assume that all the uisace parameters are kow. h i h i Ituitively, we should have var e θ 1 var b θ 1 ; i.e. igorace caot brig you ay advatage. A.D. () Jauary / 63

36 Asymptotic variace of p b θ parameter is deoted α i,j. Asymptotic variace of p e θ 1 θ is I 1 (θ ) whose (i, j) θ,1 is I 1 (θ,1 ) = 1/γ 1,1. Theorem. We have α 1,1 1/γ 1,1, with equality if ad oly if α 1,2 = = α 1,d = 0. A.D. () Jauary / 63

37 Partitio I (θ) as follows I (θ) = γ1,1 ρ ρ T Σ. Now we use the fact that I 1 (θ) = 1 1 ρ T Σ 1 τ Σ 1 ρ Σ 1 ρρ T Σ 1 + τσ 1 where τ = γ 1,1 ρ T Σ 1 ρ. As I (θ) is de ite positive the Σ 1 is de ite positive ad α 1,1 = 1 τ 1/γ 1,1 with equality i ρ = 0. To show that τ > 0 we use the fact that I (θ) is p.d. ad that τ = v T 1 I (θ) v where v = ρ T Σ 1. A.D. () Jauary / 63

38 Beyod Maximum Likelihood: Method of Momets MLE estimates ca be di cult to compute, the method of momets is a simple alterative. The obtaied estimators are typically ot optimal but ca be used as startig values for more sophisticated methods. For 1 j d, de e the j th momet of f (xj θ) where θ = (θ 1,..., θ d ) α j (θ) = E θ X j Z = x j f (xj θ) dx ad, give X = (X 1,..., X ), the j th sample as bα j = 1 X j i. The idea of the method of momets method is to match the theoretical momets to the sample momets; that is we de ed bθ as the value of θ such that α j b θ = bα j for j = 1,..., d. A.D. () Jauary / 63

39 Example: Let X i i.i.d. N µ, σ 2 with θ = µ, σ 2 the Thus we obtai Note that c σ 2 is ot ubiased. α 1 (θ) = µ, α 2 (θ) = σ 2 + µ 2, bα 1 = 1 Xi 1, bα 2 = 1 Xi 2 bµ = bα 1 ad c σ 2 = bα 2 (bα 1 ) 2. A.D. () Jauary / 63

40 Assume X i i.i.d. U (θ 1, θ 2 ) where < θ 1 < θ 2 < + the α 1 (θ) = θ 1 + θ 2 2 Now we solve ad obtai θ 1 = 2bα 1 θ 2,, α 2 (θ) = θ2 1 + θ θ 1 θ bα 2 = (2bα 1 θ 2 ) 2 + θ (2bα 1 θ 2 ) θ 2, (θ 2 bα 1 ) 2 = 3 Sice θ 2 > E (X ) the r r bθ 2 = bα bα 2 bα 2 1, bθ 1 = bα 1 3 bα 2 bα 2 1. Note that b θ 1, bθ 2 is NOT a fuctio of the su ciet statistics X (1), X (). bα 2 bα 2 1 A.D. () Jauary / 63

41 Assume X i i.i.d. Bi (p, k) with parameters k 2 N ad p 2 (0, 1). We have Thus we obtai α 1 (θ) = kp, α 2 (θ) = kp (1 p) + k 2 p 2. bp = bα 1 + bα 2 1 bα 2 /bα 1, bk = bα 2 1/ bα 1 + bα 2 1 bα 2. The estimator bp 2 (0, 1) but bk is geerally ot a iteger. A.D. () Jauary / 63

42 Statistical Properties of the Estimate Let bα = (bα 1,..., bα d ), we have bα = h b θ ad if the iverse fuctio g = h 1 exists, the bθ = g (bα). If g is cotiuous at α = (α 1,..., α d ) the bθ is a cosistet estimate of θ as bα j! α j. Moreover if g is di eretiable at α ad E θ X 2d < the p b D! θ θ N 0, rg (α) T V α rg (α) where V α [i, j] = α i+j α i α j. A.D. () Jauary / 63

43 The result follows from p (bα α) D! N (0, V α ) as We have E θ [bα j, ] = α j, cov [bα i, bα j, ] = α i+j bθ θ = g (bα ) g (α ) α i α j so usig the delta method p b D! θ θ N 0, rg (α) T V α rg (α) A.D. () Jauary / 63

44 We ca also establish the 1 order asymptotic bias of the estimate as g (bα ) = g (α) + rg (α) T (bα α) (bα α) T r 2 g (α) (bα α) + o 1 where p (bα α) D! Z Σ with Z Σ N (0, Σ) so (bα α) T r 2 g (α) (bα α) D! Z T Σ r 2 g (α) Z Σ as recall that X D! X implies ϕ (X ) D! ϕ (X ). Thus we have E [g (bα )] = g (α) + 1 h i 1 2 E ZΣ T r 2 g (α) Z Σ + o = g (α) + tr r2 g (α) Σ 1 + o. 2 A.D. () Jauary / 63

45 Beyod Maximum Likelihood: Pseudo-Likelihood Assume X = (X 1,..., X q ) f (xj θ). Give observatios X i f (xj θ), the MLE requires maximizig L ( θj x). However i some problems, it might be di cult to specify f (xj θ) ad we may be oly able to specify say f (x k, x l j θ) for 1 k < l q Based o this iformatio ad observatios, we could de e the pseudo-log-likelihood fuctios l 1 ( θj x) = l 2 ( θj x) = l 1 θj x i = l 2 θj x i = q s=1 q s=1 log f (x s j θ), q t=s+1 log f (x s, x t j θ) + αl 1 ( θj x). These pseudo-likelihood fuctios are simpler that the full likelihood. A.D. () Jauary / 63

46 Uder regularity coditios very similar to the oes for the MLE, solvig l 0 k ( θj x) = 0 for k = 1, 2 will provide ubiased estimates. To derive the asymptotic variace, we use 0 = lk 0 b θ lk 0 (θ ) + b θ ) p b θ θ = 1 p l 0 k (θ ) 1 l 00 k (θ ) θ l 00 k (θ ) where 1 l k 00 (θ )! P E θ [lk 00] ad p 1 l 0 (θ )! D N 0, E θ l 02 k, thus p b D! θ θ N 0, E θ lk 00 2 Eθ l 02 k. We have the estimates E θ l 00 1 k l 00 k ( θj x i ), E θ l 02 1 k lk 02 ( θj x i ). A.D. () Jauary / 63

47 Example. Assume that X = (X 1,..., X q ) N (0, Σ) where [Σ] (i, j) = 1 if i = j ad ρ otherwise. We are iterested i estimatig θ = ρ. There is o iformatio about ρ i l 1 ( θj x) so we use l 2 ( θj x) for α = 0. For observatios X 1,..., X, we have l 2 ( θj x) = q (q 1) 4 (q 1) (1 ϱ) 2 (1 ρ 2 ) log 1 ρ 2 q 1 + ρ 2 (1 ρ 2 ) SS W SS B q where SS W = q s=1 X i s!! q 2 Xt i, SS B = X i 2 t. t=1 A.D. () Jauary / 63

48 After simple but tiedous calculatios, we obtai for the asymptotic variace 2 1 ρ 2 c (q, ρ) q (q 1) (1 + ρ 2 ) 2 where c (q, ρ) = 1 ρ ρ 2 + qρ 3ρ 3 + 8ρ 2 3ρ + 2 +q 2 ρ 2 1 ρ 2 whereas for MLE we have 2 (1 + (q 1) ρ) 2 (1 ρ) 2 q (q 1) 1 + (q 1) ρ 2. The ratio is 1 for q = 2 as expected ad also 1 if ρ = 0 or 1. For ay other values, there is a loss of e ciecy for l 2 ( θj x) which icreases as q!. A.D. () Jauary / 63

49 Cosider the followig time series X 0 N 0, σ 2 1 α 2, X = αx 1 + σv where V i.i.d. N (0, 1) where θ = σ 2. We ca show that we have for ay i = 0, 1,..., 1 f (x i j θ) = p exp 1 α 2! xi 2 2πσ 2 2σ 2 ad we cosider 2l 1 ( θj x) = 2 log f (x i j θ) = cste log σ 2 1 α 2 This pseudo-likelihood ca easily be maximized 2σ 2 xi 2 cσ 2 = 1 α2 xi 2. If oe is iterested i estimatig α, it would be ecessary to itroduce f (x i, x i+1 j θ). A.D. () Jauary / 63

50 Pseudo-likelihood is widely used for Markov radom elds sice its itroductio by Besag (1975). I the Gaussia cotext, we have X = (X 1,..., X ) where d is extremely large ad Gaussia ad the model is speci ed by E θ [X i j x i ] = λ H ij x j, var θ [X i j x i ] = κ. Computig the likelihood for θ = (λ, κ) ca be too computatoally itesive so the pseudo-likelihood is de ed through thus el ( θj x) = bλ = xt Hx x T H 2 x, κ = d 1 log f (x i j θ, x i ) x T x x T Hx! 2 x T H 2 x I this cotext, it ca be show that the estimate is cosistet ad has a reasoable asymptotic variace. A.D. () Jauary / 63

51 Summary of Pseudo Likelihood I may applicatios, the log-likelihood l (θ; y 1: ) is very complex to compute. Istead we maximize a surrogate fuctio l S (θ; y 1: ). If possible, we pick this fuctio such that if θ is the true parameter the E θ (l S (θ; Y 1: )) is maximized for θ = θ ad solvig is easy. rl S b θ ; Y 1: = 0 A.D. () Jauary / 63

52 Uder regularity assumptios, we have p b θ θ ) N 0, G 1 (θ ) where with G 1 (θ) = H 1 (θ) J (θ) H T (θ) J (θ) = V frl S (θ; Y 1: )g, H (θ) = E r 2 l S (θ; Y 1: ). Whe l S (θ; Y 1: ) = l (θ; Y 1: ) ad the model is correctly speci ed the G (θ) is the Fisher iformatio matrix. Whe l S (θ; Y 1: ) 6= l (θ; Y 1: ), we typically lose i terms of e ciecy. A.D. () Jauary / 63

53 Applicatio to Geeral State-Space Models Cosider the followig geeral state-space model. Let fx k g k1 be a Markov process de ed by X 1 µ θ ad X k j (X k 1 = x k 1 ) f θ ( j x k 1 ). The we have that for ay > 0 p θ (x 1: ) = p θ (x 1 ) = µ θ (x 1 ) k=2 k=2 p θ (x k j x 1:k 1 ) f θ (x k j x k 1 ). A.D. () Jauary / 63

54 We are iterested i estimatig θ from the data but we do ot have access to fx k g k1. We oly have access to a process fy k g k1 such that, coditioal upo fx k g k1, the observatios are statistically idepedet ad That is we have for ay > 0 p θ (y 1: j x 1: ) = Y k j (X k = x k ) g θ ( j x k ). p θ (y k j x k ) = g θ (y k j x k ). k=1 k=1 A.D. () Jauary / 63

55 Examples Liear Gaussia model. Cosider say for jαj < 1 X 1 N σ 2 i.i.d. 0, 1 α 2, X k = αx k 1 + σv k where V k N (0, 1), Y k = β + X k + τw k where W k i.i.d. N (0, 1). I this case we have say θ = β, σ 2, α, τ 2 ad f θ (x k j x k 1 ) = N x k ; αx k 1, σ 2, g θ (y k j x k ) = N y k ; β + x k, τ 2. A.D. () Jauary / 63

56 Stochastic Volatility model. Cosider say for jαj < 1 X 1 N σ 2 i.i.d. 0, 1 α 2, X k = αx k 1 + σv k where V k N (0, 1), Y k = β exp (X k /2) W k where W k i.i.d. N (0, 1). I this case we have say θ = β, σ 2, α ad f θ (x k j x k 1 ) = N x k ; αx k 1, σ 2, g θ (y k j x k ) = N y k ; 0, β 2 exp (x k ). A.D. () Jauary / 63

57 I this case, the likelihood of the observatios y 1: is give by Z p θ (y 1: ) = p θ (x 1:, y 1: ) dx 1: Z = p θ (y 1: j x 1: ) p θ (x 1: ) dx 1:!! Z = g θ (y k j x k ) µ θ (x 1 ) f θ (x k j x k 1 ) dx 1:. k=1 k=2 If the model is liear Gaussia or ite state-space, we ca compute the likelihood i closed-form but the maximizatio is ot trivial. Otherwise, we caot. A.D. () Jauary / 63

58 Pairwise likelihood for state-space models We cosider the followig pseudo-likelihood for m 1 L S (θ; y 1: ) = 1 mifi +m,g p θ (y i, y j ) j=i+1 where Z p θ (y i, y j ) = g θ (y i j x i ) g θ (y j j x j ) p θ (x i, x j ) dx i dx j. As a alterative, if = pm, we could maximize L S (θ; y 1: ) = p p θ y (i 1)m+1:im. A.D. () Jauary / 63

59 For the two models discussed earlier, it is possible to compute exactly p θ (x i, x j ) as p θ (x i, x j ) = p θ (x i ) p θ (x j j x i ) where σ p θ (x i ) = N 2 x i ; 0, 1 α 2! p θ (x j j x i ) = N x j ; α j i x i, σ 2 j i 1 α 2k. k=0 I a geeral case, we could approximate p θ (y i, y j ) through Mote Carlo bp θ (y i, y j ) = 1 N N g θ y i j Xi l g θ y j j Xj l l=1 where Xi l, X j l p θ (x i, x j ). A.D. () Jauary / 63

60 Uder regularity assumptios, we have Z l S (θ; y 1: ) p θ (y 1: ) dy 1: which is maximum i θ so maximizig this pseudo-likelihood method makes sese. To prove it, ote that l S (θ; y 1: ) = 1 mifi +m,g log p θ (y i, y j ) j=i+1 ad = Z Z log p θ (y i, y j ).p θ (y 1: ) dy 1: log p θ (y i, y j ).p θ (y i, y j ) dy i dy j which is maximum i θ = θ. A.D. () Jauary / 63

61 Applicatio Cosider X 1 N σ 2 i.i.d. 0, 1 α 2, X k = αx k 1 + σv k where V k N (0, 1), Y k = β + X k + τw k where W k i.i.d. N (0, 1). where θ = β, σ 2, α, τ 2. I this case, we ca directly establish ot oly p θ (x i, x j ) but p θ (y i, y j )!! Yi β τ 2 + σ2 α j i σ2 N, 1 α 2 1 α 2 Y j β α j i σ2 τ 2 + σ2 1 α 2 1 α 2 For m = 2,..., 20 we compare the performace of bθ MLE ad bθ MPL where the likelihood ad pseudo-likelihood are maximized usig a simple gradiet algorithm (EM could be used). 1,000 time series of legth = 500 with β = 0.1, τ = 1.0, α = 0.95 ad σ = 0.55 are simulated. A.D. () Jauary / 63

62 true bθ (2) MPL bθ (6) MPL bθ (12) MPL bθ (20) MPL bθ ML β (0.488) (0.489) (0.4908) (0.492) (0.481) τ (0.066) (0.048) (0.054) (0.068) (0.046) α (0.033) (0.020) (0.022) (0.024) (0.020) σ (0.160) (0.064) (0.072) (0.087) (0.061) A.D. () Jauary / 63

63 Now you should... be able to compute MLE estimate for rather complex models, be able to compute the asymptotic variace of the MLE estimate, be able to derive the expressio of the asymptotic variace of simple estimates di eret from MLE. A.D. () Jauary / 63

EECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1

EECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1 EECS564 Estimatio, Filterig, ad Detectio Hwk 2 Sols. Witer 25 4. Let Z be a sigle observatio havig desity fuctio where. p (z) = (2z + ), z (a) Assumig that is a oradom parameter, fid ad plot the maximum