SOLUTION FOR HOMEWORK 6, STAT 6331

Transcription

1 SOLUTION FOR HOMEWORK 6, STAT 633. Exerc.7.. It is given that X,...,X n is a sample from N(θ, σ ), and the Bayesian approach is used with Θ N(µ, τ ). The parameters σ, µ and τ are given. (a) Find the joinf pdf of X and Θ. We know that X N(θ, σ /n), thus f X,Θ (x, θ) (πσ /n) / e (x θ) /[σ /n] (πτ ) / e (θ µ) /[τ ] [4π σ τ /n] / exp( x xθ + θ σ /n θ µθ + µ τ ) () [4π σ τ /n] / exp( τ x θ[τ x + µσ /n] + (τ + σ /n)θ + µ σ /n (σ /n)τ ). () What can we say about this distribution? It is bivariate normal! Remember that Y (Y, Y ) T is bivariate normal with mean (ν, ν ) T and covariance matrix iff Σ d ρd d ρd d d, d > 0, d > 0, ρ, f Y,Y (y, y ) (π) / (det(σ)) / exp{ (/)(y ν) T Σ (y ν)} (3) ( ρ ) / πd d exp{ ( ρ ) [(y ν ) ρ (y ν )(y ν ) + (y ν ) ]}. (4) d d d Let us now directly match () and (4). Because these two densities are the same for all x and θ, we can equate coefficients (first for x and then for θ ) σ /n ( ρ )d τ σ /n ( ρ )d Now note that d Var(Θ) τ, and using this we get the simplified system σ /n ( ρ )d + τ σ /n ( ρ )τ d which yields { d n σ + τ ρ τ n σ +τ. Now I note that ν E(Θ) µ. Keeping this in mind, I compare () and (4) in terms of coefficients for x (note that there is no term x in ()), 0 ν ( ρ )d ρν ( ρ )d d.

2 This yields ν ρ µd d µ. We conclude that () is the bivariate normal with ν µ, ν µ, d n σ + τ, d τ, ρ τ n σ + τ. (5) (ii) Can we figure out (5) faster/simpler? Of course! We know that ν µ and d τ. Then using X N(θ, σ ), we can write, and Now, our next step is to calculate ν E( X) E{E( X Θ)} E(Θ) µ, d Var( X) E( X) (E X) E( X ) µ. E( X) E{E{( X) Θ}} E{[Var( X Θ) + (E{ X Θ}) } E{[n σ + Θ ]} n σ + E(Θ ) n σ + [Var(Θ) + (EΘ) ] n σ + τ + µ. (6) This allows us to conclude that d E( X ) µ n σ + τ. Finally, because E{( X µ)(θ µ)} E( XΘ) µ, we get ρ E( XΘ) µ d d E{E( XΘ Θ)} µ d d E(Θ ) µ d d I use (6) to continue We gor the same parameters! τ + µ µ τ(n σ + τ ) / τ (n σ + τ ) /. (b) We have shown this in Section (a), but let us do this directly to improve our calculus skills. Write f X(x) f X,Θ (x, θ)dθ [I use ()] π(τ σ /n) / [ exp{ τ x (σ /n)τ } (7) exp{ (τ + σ /n)[θ θ(τ x + µσ /n)(τ + σ /n) + (τ x + µσ /n) /(τ + σ /n) ] } (σ /n)τ (8) exp{ (τ x + µσ /n) (τ + σ /n) µ σ /n } ] dθ. (9) (σ /n)τ

3 Now see how I am finishing. The integral with respect to θ, according to (8), is a constant. Now, I expect to have f X(x) (πd ) / exp{ (x ν) /(d )} (πd ) / exp{ x d + xν d ν d }. Thus, let us look at terms with e cx and e cx. From (7) and (9) we get that exp{ x d } τ 4 (τ + σ /n) exp{ τ x } exp{ (σ /n)τ (τ + σ /n) }. This yields that d τ + σ /n (what was expected to see). Further, exp{ ν d } µ(σ /n)(τ + σ /n) µ exp{τ } exp{ (σ /n)τ τ + σ /n }. This implies the wished µ ν. (c) The posterior is π(θ x, σ, µ, τ ) f Θ, X(θ, x). f X( x) We did this in class so I just remind you the idea. You wish to see The only θ what we see in () are π(θ x, σ, µ, τ ) (πd ) / (θ ν) exp{ }. (0) d π(θ x, σ, µ, τ ) exp{ (τ + σ /n)θ θ[τ x + µσ /n] σ /n)τ }. () Thus we conclude vie a comparison of (0) and () that Solving the system yields Finally, note that ν E(Θ X) ˆθ Bayes. exp{ θ } exp{ τ +σ /n)θ } d (σ /n)τ exp{ θν } exp{ θ[τ x+µσ /n] }. d (σ /n)τ d (σ.n)τ τ +σ /n ν x τ τ +σ /n + µ σ /n τ +σ /n.. Exerc First of all, let is discuss the following. If Sn : (n n ) (X l X), 3 l x

4 then why is (n )S n/σ is Chi-squared with (n ) degrees of freedom? Recall that if ξ,...,ξ m are iid standard normal then χ m D m ξj j is the (centered) chi-squared RV with m degrees of freedom. A direct calculation shows that (n )S n (n )S n + [(n )/n](x n X n ), () where X n (n ) n l X l. Now we use the induction. For n we have (I see this directly since (/)(X X ) N(0, )) S (/)(X X ) χ. Next, if (m )Sm D χ m, we should prove that ms D m+ χ m. To verify this we use () and write, msm (m )Sm + (m/(m + ))(X m+ X m ). On the right side, the first term is χ m by the induction s assumption, and the second is independent of S m because X m+ is obviously independent and X m is independent by Basu s Theorem. Plus we can check directly that (m/(m+))(x m+ X) D χ. We conclude that ms m+ has chi-squared distribution with m degress of freedom. Remark: If you note that n l Xl n X + n l (X l X) where n / X N(0, σ ) and it is independent of n l (X l X), then this is another way to look (establish) the property. Now let us look at the problem in the text. Let t s. Then I use the chi-squared density F Sn σ (t σ ) Γ((n )/) (n )/[(n )σ t] (n )/ e (n )t/σ [(n )/σ]. Here I used the fact that if A RV Z has the pdf f Z (z) then (do you remember how to verify it?) f bz (y) b f Z (y/b). Now I use the given prior pfd π(σ ) and get (as usual in Bayesian analysis, we are interested in factors depending only on σ ) π(σ t) [(/σ ) (n )/ e (n )t/(σ) σ ][σ α e /(βσ ] (/σ ) (n )/+α+ exp{ σ [(n )t/ + (/β)]}. As you see, this is again the inverted gamma pdf, that is, π(σ t) is IG(a, b) with a (n )/ + α, b [(n )t/ + (/β)]. Because for the IG distribution the mean is /[(a )b], we get ˆσ Bayes E(σ t) n t + β n + α 4 n S n + β n + α.

5 3. Exers Let X,..., X n be a sample from Poisson(λ) and λ Gamma(α, β). (a). What is the posterior distribution of Λ? Write π(λ x) π(λ)f X λ(x λ) f X (x) λ β e λ/β e nλ λ n l x l λ (β+ n l xl) λ n exp{ β/(nβ + ) } Gamma(α + β x l, nβ + ). (b) Write (using the previous result and a formula for the mean of a Gamma RV, β ˆλ Bayes E(Λ X) (α + X l ) l nβ + + nβ + (αβ). Note that we again see the familiar structure in the Bayes estimate as an average because the prior mean and the MLE. For the variance we similarly get β Var(ˆλ Bayes X) (α + X l ) l (nβ + ). l 4. Exerc A sequence of RVs X,...,X n is observed. It is known that X i θ i N(θ i, σ ) and θ i N(µ, τ ). We begin with part (b) because we will use it in part (a). (b) Assertion: If X i θ i are independent with the pdf f(x θ i ), and Θ i are iid according to the pdf π(θ τ) then X,..., X n are iid. Let us prove this assertion. The joint pdf of (X, Θ) is Then f(x, θ τ) f(x l, θ l τ) π(θ l τ)f(x l θ). l l f(x τ)... π(θ l τ)f(x l θ l )dθ l l... { π(θ τ)f(x θ, τ)dx } π(θ l τ)f(x l θ l )dθ l. l Note that the integral in the curly brackets is f(x τ), and then repeating taking integrals (note that in the ofirst line there are n integrals) and taking into account that all obtaoined pdfs are the same we obtain that This shows that marginally all X s are iid. f(x τ) f(x l τ). l 5

6 (a) Using the result of part (b) we can consider f X µ,σ,τ (x) f(x θ, σ )π(θ µ, τ )dθ [πστ] e (x θ) /(σ ) e (θ µ) /(τ ) dθ. Now we do some calculations for the product of the two exponential functions in the last integral. Write e (x θ) /(σ ) e (θ µ) /(τ ) xτ (θ [ + µσ ]) σ exp{ +τ σ +τ (x µ) σ τ (σ + τ ) }. σ +τ Only the first term in the exponent involves θ. Let us denote it as T(θ). Then it is the kernel of a normal pdf so Combining the results we conclude that e T(θ) dθ (π) / στ (σ + τ ) /. f X µ,σ,τ (x) (πστ) (π) / στ (x µ) exp{ (σ + τ ) / (σ + τ ) }. Simplify it a bit and you can clearly see that this is the pdf of N(µ, σ + τ ). (As we knew before, sure.) 5. Exerc. 7.7 (a) First, let us write down the log-likelihood, l(β, τ (x,y)) [ βτ l + y l ln(β) + y l ln(τ l ) ln(y l!) τ l + x l ln(τ l ) ln(x l!)]. l Take partial derivatives with respect to the parameters and then equate them to zero. This allows us to find extremes. The equations are and τ l + β n y l 0, l l β + τl y l + τl x l 0, l,,..., n. We got a system of n + equations with n + parameters which we can solve. Write, β nl y l nl τ l, τ l x l + y l, l,,..., n. β + Taking the sum of τ l yields n l τ l (β+) n l (x l +y l ), and we can simplify the relation for β in the following way, β (β + ) n l y l nl (x l + y l ) 6

7 which in its turn yields β nl y l nl x l. Now you should check that the extreme point is the maximum. Here either graphic, or the following analysis can be used. Note that for any vector τ an extrema in β is the maximum (the second derivative is β n l y l, it is negative with the exception of the event with the exponentially vanishing probability when n l y l 0 which is a special case to consider! What do you think the MLE should be in this case?). The same is true for any τ l. We conclude that (β, τl, l,,..., n) are the MLE. (b) Let ˆβ n and ˆτ l are the limit points. Then they satisfy nl y l ˆβ ˆτ + n, ˆτ ˆτ + y l x l ˆβ +, ˆτ i x i + y i, i,...,n. ˆβ + Solve this system of equations and you get (7..6). Well, what is the simplest way to do that? You may begin with the second equation and get ˆτ y /ˆβ, substitute it into the first equation and get ˆβ n l y l n. Then you sum over i in the third equation, substitute ˆβ and l x l get n i ˆτ i n i x i, and then plug into the first equation; this yields the first equation in (7..6). The two others have being already obtained. (c) We already did this in part (b). 6. Exerc Note that if U f(u), W g(w) and Z Bernoulli(p), then X D ZU + ( Z)W. This is how you generate a RV with the mixture distribution! This remark can help you to understand the considered setting of missed data where observations of Z are missed. (a) Write for z {0, }, f X,Z (x,z p) f Z (z p)f X Z (x z, p) [pf(x l )] z l [( p)g(x l )] z l. l To see why this expression is correct, note that f X,Z (x, p) pf(x) and f X,Z (x, 0 p) ( p)g(x). (Note that there is a misprint in the text on page 36, first line). (b) Consider f Z X (z x, p) f Z,X(z, x p) f X (x p) [pf(x)]z [( p)g(x)] z z0 [pf(x)] z [( p)g(x)] [pf(x)]z [( p)g(x)] z, z ( p)g(x) + pf(x) which is the pmf of Bernoulli(pf(x)/[pf(x) + ( p)g(x)]). (c) Using results of part (b) we can write, E{ln(L(p X,Z)) p ˆp,X x} E{ [Z l ln(pf(x l ))+( Z l ) ln(( p)g(x+l))] p ˆp,X x} l 7

8 l [ ˆpf(x l ) ˆpf(x l ) + ( ˆp)g(x l ) ln(pf(x l)) + ( ˆp)g(x l ) ˆpf(x l ) + ( ˆp)g(x l ) ln(( p)g(x l)) ]. The last expression, as a function in p, has a maximum (check this!) and the p is the solution of the equation p n l ˆpf(x l ) ( p) ˆpf(x l ) + ( ˆp)g(x l ) l ( ˆp)g(x l ) ˆpf(x l ) + ( ˆp)g(x l ) 0. Solve it and get ˆp (r+) n ˆp (r) n l f(x l ) ˆp (r) f(x l ) + ( ˆp (r) )g(x l ). 7. Exerc. 7.3 The part (b) is well explained just apply Jensen s inequality. For part (a), the first assertion is plain just plug ˆθ (r) in (7..9). Then, using (7..9) we can write, log L(θ y) E{log(L(θ y,x)) θ (r),y y} E{log k(x θ,y) θ (r),y y}. Now, we choose θ (r+) which maximizes the expected complete-data log-likelihood, that is At the same time, thanks to the part (b), θ (r+) : argmax θ Ω E{log(L(θ Y,X)) θ (r),y}. E{log k(x θ (r),y) (θ (r),y} E{log k(x θ (r+),y) (θ (r),y}. This yields log(l(θ (r+) y)) log(l(θ (r) y)). Remark: Please pay attention to the Jensen inequality it is very useful in solving many problems. 8